Within the LSB we maintain 65K+ tests and the number of tests is growing. These tests are run by the LSB working group on a regular basis on leading edge community distributions and problems are reported back to the distributions. This process is resource consuming and is somewhat related to the modularization question, see A new approach to LSB Distribution Testing for more details.
While we have test groups within the LSB test suite today, these groups are loosely connected to a "hidden" LSB module structure. With the LSB 5.0 release the LSB will be formally modularized, see LSB Modularization Design for details, which will allow us to tie the LSB testing infrastructure to these formally specified modules.
Each module and sub-module in the LSB will have a corresponding test suite such that a tester can run:
This command will execute the all the tests associated with the LSB-Base submodule. Executing
would run all the tests associated with the LSB-Base submodule, the LSB-Security submodule, and any module level tests if they exist.
From this it follows that the architecture of the tests follows the LSB module architecture.
Existing tests need to be associated with the module and sub-module names of the LSB module architecture. This association is dictated by the location of the interface within the LSB module architecture. For example POSIX interfaces are part of the LSB_Base sub-module and thus any tests testing these interfaces would be in the lsb-core test group.
The existing distribution checker implementation already has the capabilities to execute a named group of tests. Therefore, it is expected that changes to the distribution checker code for modularization purposes are relatively minor. The key changes in the testing infrastructure will be to recognize that not all tests have to be present on the machine being tested. This allows testing with a reduced set of data.
The modularization of the test infrastructure will allow us to gradually integrate the LSB tests into distribution QA efforts. This will slowly help us to shift bug detection to the distributions themselves. Getting any distribution to agree to integrate 65K+ tests at once is an undertaking with an unfavorable outcome and we must find a way to gradually move the testing upstream.