User Tools

Site Tools


civilinfrastructureplatform:ciptesting

This is an old revision of the document!


Testing

Testing has been identified as one of the key activities for the CIP group due to the nature of the Civil (Social) infrastructure industry and the challenges that maintaining a Linux base system for over 15 years brings.

To ease the collaboration among Members, CIP has set up a project in Gitlab called testing where you can follow what is being done or even better, participate in our efforts.

Testing at CIP: actions

The CIP testing project has 5 actions/milestones so far:

  1. Board at desk - single developer: set up that allows a developer to test the CIP kernel on the Beaglebone Black (CIP selected hardware platform) connected locally to the user's development machine using kernelci tools.
  2. CIP kernel testing: test the CIP kernel on regular basis and share the results with other CIP community Members.
  3. From kernel testing to system testing: once the testing environment has been defined and is working for the kernel, explore how to extend it to the entire CIP platform.
  4. Define kernel testing as a service within CIP: define the testing environment within CIP assuming that in some cases, Members would share the tests, test results and laboratory and in others they will not.

For more details, please check the CIP Testing FAQ. If you are interesting in following up how the project is progressing, please check the CIP Testing Project Management page.

Action 1: Board At Desk - Single Developer (B@D)

kernelci.org is one of the most successful Open Source testing projects. Its distributed nature improve the possibilities of collaborating among different developers and corporations. Distributed boards farms across the world are working together to test every merge of an upstream Linux kernel tree, detecting bugs and regressions in a timely manner and reporting them back to kernel developers.

Board At Desk - Single Developer, given name for action 1, has as goal to create and publish a VM that contains all the kernelci tools so a single developer can test the CIP kernel (or any other Linux kernel) on a board connected to the user's development machine. This board will initially be the selected hardware platforms for the CIP project which at the moment is Beaglebone Black. Any other board would work too after some basic configuration. Board At Desk - Single Dev. Schema

The following links are of interest:

Action 2: CIP kernel testing

Now that the VM with the needed tools has been finished, it is time to focus on CIP use case. The goals of this Action 2 are the following:

  1. Test CIP kernel in a transparent way so the test environment, the tests themselves, the results and the logs got from each test are reproducible by anyone at any given time as well as traceable. .
  2. Ensure that testing the CIP kernel can be performed by any CIP and community Member at any time.
  3. Tests, test results, logs and other metadata are shared and public through Open Source licenses.
  4. Increase the CIP kernel test coverage.

1.- Test CIP kernel in a transparent way

The most effective and scalable way to achieve this goal is by creating a testing service in a similar way that the kernel does through KernelCI which is creating a public testing service. Even more effective is if the tools used by the public service can be downloaded and easily configured by any contributor in order to replicate the tests and test results in his own environment, like openSUSE does with openQA.

As a first step on the journey towards transparency, CIP will create an environment in which the tools, the configuration files, the tests, the kernel to be tested and the outcome of the tests are public under Open Source licenses so anyone can reproduce the outcome of the tests performed, as well as the environment. All those inputs will also be signed so they can be verified at any stage of the testing process and life time of the CIP kernel.

2.- Testing performed by any developer at any time

Action 1 provides a test environment that any developer can use locally. In order to reach this goal, all the canonical inputs and configurations must be trusted and available to anyone. The same principle applies to the outcome of the tests so inspection can occur to evaluate potential mismatches between results.

3.- Share inputs, outputs and configurations

All inputs, configurations and outputs, including metadata and environment variables will be shared through publicly accessible repositories or mailing lists.

4.- Increase CIP kernel test coverage

The main goal for CIP testing project when it comes to the kernel is to detect potential regressions and security issues in those backported patches applied to our current CIP kernel. An outcome of this action should be to select the best tests available and add a few more basic ones. At some point, early in the process, no patch can be merged without associated tests.

Action 3: Extend the number of technologies used to deploy B@D

In order to reach the maximum number of developers, CIP testing team plans to enhance the deployment adding additional virtualisation technologies like KVM. Using containers will also be evaluated.

Action 4: From kernel testing to system testing

CIP is not just about the kernel. The goal is to create a base system for the industry. In order to achieve the general goal, this initiative will require more of a journey than the one taken for the kernel, but for a base/core system.

Action 5: Define kernel testing as a service within CIP

Once CIP develops a testing culture, it will be time for scaling up the testing activities by turning the project into a service for Members and CIP friends, in collaboration with the kernel community.

Back to the parent page

civilinfrastructureplatform/ciptesting.1507564933.txt.gz · Last modified: 2017/10/09 16:02 by Agustin Benito Bethencourt