Linux Foundation Wiki

project collaboration site

User Tools

Site Tools


civilinfrastructureplatform:ciptesting

Testing

Testing has been identified as one of the key activities for the CIP group due to the nature of the Civil (Social) infrastructure industry and the challenges that maintaining a Linux base system for over 15 years brings.

To ease the collaboration among Members, CIP has set up a project in Gitlab called testing where you can follow what is being done or even better, participate in our efforts.

Testing at CIP: actions

The CIP testing project has 5 actions/milestones so far:

  1. Board at desk - single developer: set up that allows a developer to test the CIP kernel on the Beaglebone Black (CIP selected hardware platform) connected locally to the user's development machine using kernelci tools.
  2. CIP kernel testing: test the CIP kernel on regular basis and share the results with other CIP community Members.
  3. From kernel testing to system testing: once the testing environment has been defined and is working for the kernel, explore how to extend it to the entire CIP platform.
  4. Define kernel testing as a service within CIP: define the testing environment within CIP assuming that in some cases, Members would share the tests, test results and laboratory and in others they will not.

For more details, please check the CIP Testing FAQ

Action 1: Board At Desk - Single Developer (B@D)

kernelci.org is one of the most sucessful Open Source testing projects. Its distributed nature improve the possibilities of collaborating among different developers and corporations. Distributed boards farms across the world are working together to test every merge of an upstream Linux kernel tree, detecting bugs and regressions in a timely manner and reporting them back to kernel developers.

Board At Desk - Single Developer, given name for action 1, has as goal to create and publish a VM that contains all the kernelci tools so a single developer can test the CIP kernel (or any other Linux kernel) on a board connected to the user's development machine. This board will initially be the selected hardware platforms for the CIP project which at the moment is Beaglebone Black. Any other board would work too after some basic configuration. Board At Desk - Single Dev. Schema

The following links are of interest:

  • Download Board at desk - single dev.
  • B@D feature page describes the main characteristics of the tools.
  • The wiki pages that will guide you through the process of using B@D are:
    1. Instructions to deploy and configure B@D, connect the board and try the reference tests.
      1. Deploy B@D through Vagrant or by downloading the B@D standalone VM box directly.
      2. Once deployed, configure B@D.
      3. Connect the board (Beaglebone Black in this case) to the host machine, where B@D is deployed.
      4. Check the reference test cases and try them as examples.
    2. If you experience any problem following the above instructions, please check the Known Issues and Workarounds wiki page before reporting any issue.
  • Contribute to the CIP testing project

Action 2: CIP kernel testing

Once the VM with the needed tools has been finished, it is time to focus on CIP use case. The goals of this Action 2 are the following:

  1. Test CIP kernel in a transparent way so the test environment, the tests themselves, the results and the logs got from each test are reproducible by anyone at any given time as well as traceable. .
  2. Ensure that testing the CIP kernel can be performed by any CIP and community Member at any time.
  3. Tests, test results, logs and other metadata are shared and public through Open Source licenses.
  4. Increase the CIP kernel test coverage.

1.- Test CIP kernel in a transparent way

The most effective and scalable way to achieve this goal is by creating a testing service in a similar way that the kernel does through KernelCI which is creating a public testing service. Even more effective is if the tools used by the public service can be downloaded and easily configured by any contributor in order to replicate the tests and test results in his own environment, like openSUSE does with openQA.

As a first step on the journey towards transparency, CIP will create an environment in which the tools, the configuration files, the tests, the kernel to be tested and the outcome of the tests are public under GPL based licenses so anyone can reproduce the outcome of the tests performed, as well as the environment. All those inputs will also be signed so they can be verified at any stage of the testing process and life time of the CIP kernel.

2.- Testing performed by any developer at any time

Action 1 provides a test environment that any developer can use locally. In order to reach this goal, all the canonical inputs and configurations must be trusted and available to anyone. The same principle applies to the outcome of the tests so inspection can occur to evaluate potential mismatches between results.

3.- Share inputs, outputs and configurations

All inputs, configurations and outputs, including metadata and environment variables will be shared through publicly accessible repositories or mailing lists. Based on the current LAVA and KernelCI licenses, AGPLv3, CIP will adopt the spirit of the tooling licenses and use GPL based licenses.

4.- Increase CIP kernel test coverage

The main goal for CIP testing project when it comes to the kernel is to detect potential regressions and security issues in those backported patches applied to our current CIP kernel. An outcome of this action should be to select the best tests available and add a few more basic ones. At some point, early in the process, no patch can be merged without associated tests.

Action 3: Define kernel testing as a service within CIP

Once the CIP kernel is being tested on regular basis, it will be time to improve the tooling and extend it to new technologies specially on the deployment side, to reach more potential users.

Action 4: From kernel testing to system testing

CIP is not just about the kernel. The goal is to create a base system for the industry. In order to achieve the general goal, this initiative will require more of a journey than the one taken for the kernel, but for a base/core system.

Action 5: Define kernel testing as a service within CIP

Once CIP develops a testing culture, it will be time for scaling up the testing activities by turning the project into a service for Members and CIP friends, in collaboration with the kernel community.

Back to the parent page

civilinfrastructureplatform/ciptesting.txt · Last modified: 2017/05/22 12:38 by rajmarshall