Distribution Autotesting

From The Linux Foundation
Jump to: navigation, search

To assist in automated quality assurance for distributions, the LSB supports running most of the distribution test suite in an automated fashion, either at the command line or via cron. The basis of that support is the LSB DTK Manager, which supports a command-line tool for running the tests. You can read more about that tool at the Nightly Run HOWTO.

We have also created a small wrapper around DTK Manager which does several things:

  • Prevents multiple runs, so the test run can be run out of cron without fear.
  • Saves additional result output for inclusion with the official results.
  • Provides an easy interface for emailing summary reports.
  • Uploads the data to the Linux Foundation (or, optionally, to your own site) so the LSB workgroup can see your results in its own testing.

Here are simple instructions for preparing automatic test runs.

Set Up The Distribution

You first have to set up a system to test. This may be a native system, or a virtual machine using virtualization software. The system should be LSB-compliant; make sure, if your distribution needs special treatment to install LSB support, that the support is installed.

It's important to note that several of the LSB tests are not secure, and thus the tests should never be run on a system doing anything important. Ideally, native boxes running the tests will run nothing else, and will be shut off when not being tested. Virtual machines should not be kept running when not doing a test run.

One particular part of the LSB which must be supported beforehand is the installation of LSB-compliant RPM packages. Most distributions either support RPM directly, or via the "alien" package conversion tool. If your distribution has no way of installing RPMs, please contact the LSB workgroup; we would be happy to assist you, if we can, in supporting this vital infrastructure.

Install Prerequisites

In addition to an LSB-compliant system, the automated tests will need the following:

  • The "expect" tool, which is distributed with nearly all Linux distributions, or can be found on the expect web site.
  • The "rsync" tool, which is also distributed with nearly all Linux distributions, or can be found on the rsync web site.
  • The "ssh" tool, which is also distributed with nearly all Linux distributions, or can be found on the ssh web site.

Install DTK Manager

There are a number of ways to get the LSB DTK Manager. See the Downloads page for details on installing a stable version of DTK Manager, or the LSB Wiki for information on snapshots, betas, and other unreleased versions.

Manual Setup

At this point, you can run the automated test framework yourself:

/opt/lsb/test/manager/autotest-ext/lsb-tef.pl --help

Options exist for choosing which LSB version to test against, whether the tests should be automatically updated from the Linux Foundation archive, whether results should be uploaded to a server or emailed, etc.

Automated Setup

At present, our wrapper script is only available via the Linux Foundation's version control repository. If you are familiar with Bazaar, you can check it out from http://bzr.linuxfoundation.org/unofficial/auto-dist-test. If not, go to the web front-end, select the "inventory" link, and then the "targz" link to download a tarball of the latest version.

You can create a package to install by going into the "package" subdirectory and running "make rpm_package", or you may copy the script in bin to an appropriate location. Edit the script (in /opt/lsb/test/auto-dist/bin, if installed via a package), and change LSB_VER to the version of the LSB you want to test. If you want to have results mailed to you, set the email address in EMAIL_ADDR.

The script will attempt to upload results to the Linux Foundation. This process uses ssh to a special account on our server, and requires that you set a ssh key for your "root" user. Use the following command to create a key:

ssh-keygen -t rsa

Then email the file in ~root/.ssh/id_rsa.pub to lsbtest@linuxfoundation.org. One of our admins will get back to you. Unfortunately, we can't always accomodate everyone who want to send us test results; priority will be given to official developers from major distributions.

Our test data is summarized and reported at our results page.