Testing has been identified as one of the key activities for the CIP group due to the nature of the Civil (Social) infrastructure industry and the challenges that maintaining a Linux base system for over 15 years brings.
To ease the collaboration among Members, CIP has set up a project in Gitlab called testing where you can follow what is being done or even better, participate in our efforts.
The CIP testing project has 5 actions/milestones so far:
kernelci.org is one of the most successful Open Source testing projects. Its distributed nature improve the possibilities of collaborating among different developers and corporations. Distributed boards farms across the world are working together to test every merge of an upstream Linux kernel tree, detecting bugs and regressions in a timely manner and reporting them back to kernel developers.
Board At Desk - Single Developer, given name for action 1, has as goal to create and publish a VM that contains all the kernelci tools so a single developer can test the CIP kernel (or any other Linux kernel) on a board connected to the user's development machine. This board will initially be the selected hardware platforms for the CIP project which at the moment is Beaglebone Black. Any other board would work too after some basic configuration.
The following links are of interest:
Now that the VM with the needed tools has been finished, it is time to focus on CIP use case. The goals of this Action 2 are the following:
The most effective and scalable way to achieve this goal is by creating a testing service in a similar way that the kernel does through KernelCI which is creating a public testing service. Even more effective is if the tools used by the public service can be downloaded and easily configured by any contributor in order to replicate the tests and test results in his own environment, like openSUSE does with openQA.
As a first step on the journey towards transparency, CIP will create an environment in which the tools, the configuration files, the tests, the kernel to be tested and the outcome of the tests are public under Open Source licenses so anyone can reproduce the outcome of the tests performed, as well as the environment. All those inputs will also be signed so they can be verified at any stage of the testing process and life time of the CIP kernel.
Action 1 provides a test environment that any developer can use locally. In order to reach this goal, all the canonical inputs and configurations must be trusted and available to anyone. The same principle applies to the outcome of the tests so inspection can occur to evaluate potential mismatches between results.
All inputs, configurations and outputs, including metadata and environment variables will be shared through publicly accessible repositories or mailing lists.
The main goal for CIP testing project when it comes to the kernel is to detect potential regressions and security issues in those backported patches applied to our current CIP kernel. An outcome of this action should be to select the best tests available and add a few more basic ones. At some point, early in the process, no patch can be merged without associated tests.
In order to reach the maximum number of developers, CIP testing team plans to enhance the deployment adding additional virtualisation technologies like KVM. Using containers will also be evaluated.
CIP is not just about the kernel. The goal is to create a base system for the industry. In order to achieve the general goal, this initiative will require more of a journey than the one taken for the kernel, but for a base/core system.
Once CIP develops a testing culture, it will be time for scaling up the testing activities by turning the project into a service for Members and CIP friends, in collaboration with the kernel community.