====== Testing ======

Testing has been identified as one of the key activities for the CIP group due to the nature of the Civil (Social) infrastructure industry and the challenges that maintaining a Linux base system for over 15 years brings. 

To ease the collaboration among Members, CIP has set up a project in Gitlab called [[https://gitlab.com/cip-project/testing|testing]] where you can follow what is being done or even better, participate in our efforts. 

===== Testing at CIP: actions =====

The CIP testing project has [[https://gitlab.com/cip-project/cip-testing/testing/milestones|5 actions/milestones]] so far:
   - [[https://gitlab.com/cip-project/cip-testing/testing/milestones/1|Board at desk - single developer]]: set up that allows a developer to test the CIP kernel on the Beaglebone Black (CIP selected hardware platform) connected locally to the user's development machine using kernelci tools.
   - [[https://gitlab.com/cip-project/cip-testing/testing/milestones/2|CIP kernel testing]]: test the CIP kernel on regular basis and share the results with other CIP community Members.
   - [[https://gitlab.com/cip-project/cip-testing/testing/milestones/3|Extend the number of technologies used to deploy Board at desk - Single dev]] to reach more potential users.
   - [[https://gitlab.com/cip-project/cip-testing/testing/milestones/4|From kernel testing to system testing]]: once the testing environment has been defined and is working for the kernel, explore how to extend it to the entire CIP platform.
   - Define kernel testing as a service within CIP: define the testing environment within CIP assuming that in some cases, Members would share the tests, test results and laboratory and in others they will not.

For more details, please check the [[:civilinfrastructureplatform/ciptestingfaq|CIP Testing FAQ]]. If you are interesting in following up how the project is progressing, please check the [[:civilinfrastructureplatform/ciptesting/management|CIP Testing Project Management page]].

==== Action 1: Board At Desk - Single Developer (B@D) ====

[[http://www.kernelci.org|kernelci.org]] is one of the most successful Open Source testing projects. Its distributed nature improve the possibilities of collaborating among different developers and corporations. Distributed boards farms across the world are working together to test every merge of an upstream Linux kernel tree, detecting bugs and regressions in a timely manner and reporting them back to kernel developers. 

Board At Desk - Single Developer, given name for action 1, has as goal to create and publish a VM that contains all the kernelci tools so a single developer can test the CIP kernel (or any other Linux kernel) on a board connected to the user's development machine. This board will initially be the selected hardware platforms for the CIP project which at the moment is Beaglebone Black. Any other board would work too after some basic configuration. {{ :civilinfrastructureplatform:board_at_desk_single_dev_block_diagram_low_quality.jpg?​500|Board At Desk - Single Dev. Schema}}

The following links are of interest:
   * [[cipdownload|Download]] Board at desk - single dev.
   * B@D [[ciptestingboardatdesksingledevfeaturepage|feature page]] describes the main characteristics of the tools.
   * The wiki pages that will guide you through the process of using B@D are:
       - Instructions to deploy and configure B@D, connect the board and try the reference tests.
         - Deploy B@D through [[ciptestingboardatdeskdingledevdeployment#b-d-deployment-method-2-building-vm-from-scratch-using-vagrant-15|Vagrant]] or by downloading the [[ciptestingboardatdeskdingledevdeployment#b-d-standalone-virtual-machine-box|B@D standalone VM box]] directly.
         - Once deployed, [[[[ciptestingboardatdesksingledevsetup|configure B@D]].
         - Connect the board ([[beagleboneblackboard|Beaglebone Black]] or [[renesasboard|Renesas IWG20M]]) to the host machine, where B@D is deployed.
         - [[cipsystembuildhowto|Build the kernel (system)]] to be tested and required artefacts.
         - Check the [[civilinfrastructureplatform/ciptestingreferencetestcases|reference test cases]] and try them as examples.
      - If you experience any problem following the above instructions, please check the [[ciptestingknownissues|Known Issues and Workarounds]] wiki page before reporting any issue.
   * Do you want to know more about the release process followed with the CIP Testing project of tools and code? You should take a look at the following pages then:
      * Check the CIP Testing project [[:civilinfrastructureplatform/ciptesting/management|management wiki page]] to get a description of the process followed and links to other related content.
      * Interested in previous releases of B@D? Please check the [[/civilinfrastructureplatform/ciptesting/releases|CIP Testing Project Releases]] wiki page.
      * All the marketing related material created for every release is published on the CIP Testing Project [[/civilinfrastructureplatform/ciptesting/batdmarketing|marketing page]].
 
=== Contribute to the CIP testing project ===

If you want to contribute to the CIP Testing project, please consider reading the following information:
      * Check the code at [[https://gitlab.com/cip-project/cip-testing/board-at-desk-single-dev/tree/master|Gitlab.com]]
      * Please read the [[ciptestingfaq|CIP Testing FAQ]] to learn more about B@D.
         * The delivery process is described in the [[/civilinfrastructureplatform/ciptesting/management|CIP testing management]] wiki page.
      * Join the technical mailing list ([[https://lists.cip-project.org/mailman/listinfo/cip-dev|cip-dev]]) to follow this effort. 
      * Report a [[https://gitlab.com/cip-project/cip-testing/testing/boards/322796?&label_name[]=bug|bug]].
      * Any contributions for the CIP Testing project need to have licences that are compatible with the existing licences (APGL-3.0), source files must have a licence header and any repositories must have a LICENSE file and a LICENSE.spdx file - see the current repositories for style and see also [[https://commonsconservancy.org/faq/licenseinfo/|the commons conservancy]] guidance.

==== Action 2: CIP kernel testing ====

Now that the VM with the needed tools has been finished, it is time to focus on CIP use case. The goals of this Action 2 are the following:

   - Test CIP kernel in a transparent way so the test environment, the tests themselves, the results and the logs got from each test are reproducible by anyone at any given time as well as traceable. .
   - Ensure that testing the CIP kernel can be performed by any CIP and community Member at any time.
   - Tests, test results, logs and other metadata are shared and public through Open Source licenses.
   - Increase the CIP kernel test coverage.

=== 1.- Test CIP kernel in a transparent way ===

The most effective and scalable way to achieve this goal is by creating a testing service in a similar way that the kernel does through KernelCI which is creating a public testing service. Even more effective is if the tools used by the public service can be downloaded and easily configured by any contributor in order to replicate the tests and test results in his own environment, like openSUSE does with openQA. 

As a first step on the journey towards transparency, CIP will create an environment in which the tools, the configuration files, the tests, the kernel to be tested and the outcome of the tests are public under Open Source licenses so anyone can reproduce the outcome of the tests performed, as well as the environment. All those inputs will also be signed so they can be verified at any stage of the testing process and life time of the CIP kernel.

=== 2.- Testing performed by any developer at any time ===

Action 1 provides a test environment that any developer can use locally. In order to reach this goal, all the canonical inputs and configurations must be trusted and available to anyone. The same principle applies to the outcome of the tests so inspection can occur to evaluate potential mismatches between results. 

=== 3.- Share inputs, outputs and configurations ===

All inputs, configurations and outputs, including metadata and environment variables will be shared through publicly accessible repositories or mailing lists.

=== 4.-  Increase CIP kernel test coverage ===

The main goal for CIP testing project when it comes to the kernel is to detect potential regressions and security issues in those backported patches applied to our current CIP kernel. An outcome of this action should be to select the best tests available and add a few more basic ones. At some point, early in the process, no patch can be merged without associated tests.

==== Action 3: Extend the number of technologies used to deploy B@D ====

In order to reach the maximum number of developers, CIP testing team plans to enhance the deployment adding additional virtualisation technologies like KVM. Using containers will also be evaluated.

==== Action 4: From kernel testing to system testing ====

CIP is not just about the kernel. The goal is to create a base system for the industry. In order to achieve the general goal, this initiative will require more of a journey than the one taken for the kernel, but for a base/core system. 

==== Action 5: Define kernel/system testing as a semi-distributed service within CIP ====

Once CIP develops a testing culture, it will be time for scaling up the testing activities by turning the project into a service for Members and CIP friends, in collaboration with the kernel community.

[[start|Back to the parent page]]