Distribution Autotesting

From The Linux Foundation
Revision as of 23:10, 11 October 2008 by Licquia (Talk | contribs)

Jump to: navigation, search

To assist in automated quality assurance for distributions, the LSB supports running most of the distribution test suite in an automated fashion, either at the command line or via cron. The basis of that support is the LSB DTK Manager, which supports a command-line tool for running the tests. You can read more about that tool at the Nightly Run HOWTO.

We have also created a small wrapper around DTK Manager which does several things:

  • Prevents multiple runs, so the test run can be run out of cron without fear.
  • Saves additional result output for inclusion with the official results.
  • Provides an easy interface for emailing summary reports.
  • Uploads the data to the Linux Foundation (or, optionally, to your own site) so the LSB workgroup can use your results in its own planning and development of the LSB.

Here are simple instructions for preparing automatic test runs.

Set Up The Distribution

You first have to set up a system to test. This may be a native system, or a virtual machine using virtualization software. The system should be LSB-compliant; make sure, if your distribution needs special treatment to install LSB support, that the support is installed.

It's important to note that several of the LSB tests are not secure, and thus the tests should never be run on a system doing anything important. Ideally, native boxes running the tests will run nothing else, and will be shut off when not being tested. Virtual machines should not be kept running when not doing a test run.

One particular part of the LSB which must be supported beforehand is the installation of LSB-compliant RPM packages. Most distributions either support RPM directly, or via the "alien" package conversion tool. If your distribution has no way of installing RPMs, please contact the LSB workgroup; we would be happy to assist you, if we can, in supporting this vital infrastructure.

Install Prerequisites

In addition to an LSB-compliant system, the automated tests will need the following:

  • The "expect" tool, which is distributed with nearly all Linux distributions, or can be found on the expect web site.
  • The "rsync" tool, which is also distributed with nearly all Linux distributions, or can be found on the rsync web site.
  • The "ssh" tool, which is also distributed with nearly all Linux distributions, or can be found on the ssh web site.

Install DTK Manager

There are a number of ways to get the LSB DTK Manager. See the Downloads page for details on installing a stable version of DTK Manager, or the LSB Wiki for information on snapshots, betas, and other unreleased versions.

Manual Setup

At this point, you can run the automated test framework yourself:

/opt/lsb/test/manager/autotest-ext/lsb-tef.pl --help

Options exist for choosing which LSB version to test against, whether the tests should be automatically updated from the Linux Foundation archive, whether results should be uploaded to a server or emailed, etc.

Automated Setup

We provide a wrapper script which can