Differences

This shows you the differences between two versions of the page.

--- civilinfrastructureplatform:ciptesting:centalisedtesting [2019/03/18 16:10]
patersonc Started to add more content.
+++ civilinfrastructureplatform:ciptesting:centalisedtesting [2023/12/19 08:55] (current)
swvanbuuren Add SQUAD watch job workflow
@@ Line 2: / Line 2: @@
 The first goal in CIP testing was the creation of the [[civilinfrastructureplatform:ciptesting#action_1board_at_desk_-_single_developer_b_d|B@D]] (Board at Desk) virtual machine. This allows developers to easily run automated testing on a local Beaglebone Black or Renesas RZ/G1M iwg20m platform.
-The next goal for CIP is to centralise testing, so that developers can run tests without having local access to a platform. This will become more and more useful for CIP as the list of [[civilinfrastructureplatform:ciptesting:cipreferencehardware|reference platforms]] grows. It also means that tests can be triggered and run automatically 24/7.
+The second goal for CIP testing was to centralise testing so that developers can run tests without having local access to a platform. This is becoming more and more useful for CIP as the list of [[civilinfrastructureplatform:ciptesting:cipreferencehardware|reference platforms]] grows. It also means that tests can be triggered and run automatically 24/7.
+The third goal for CIP testing was to set up Continuous Integration (CI) testing to automatically test CIP software on CIP hardware. See the [[civilinfrastructureplatform:ciptesting:centalisedtesting:cioverview|Continuous Integration testing overview]] page for more details.
+The fourth goal for CIP testing is to integrate with the [[https://kernelci.org/|KernelCI project]]. This work is underway and the latest results can be viewed at [[https://kernelci.ciplatform.org/|kernelci.ciplatform.org]].
 ===== CIP Testing Architecture =====
 The block diagram below provides an overview of CIP's centralised test infrastructure.
-{{ :civilinfrastructureplatform:ciptesting:20190318_cip_testing_overview.png |}}
+{{ :civilinfrastructureplatform:ciptesting:20190807_cip_testing_overview.png |}}
 Explanations for each block are below.
@@ Line 13: / Line 16: @@
 This schedules submitted test jobs to run on available hardware targets.
-CIP have set up their own instance of LAVA (Linaro Automated Validation Architecture). LAVA is a continuous integration system for deploying operating systems onto physical and virtual hardware for running tests. See the [[https://www.lavasoftware.org/|LAVA website]] for more details.
+CIP has set up their own instance of LAVA (Linaro Automated Validation Architecture). LAVA is a continuous integration system for deploying operating systems onto physical and virtual hardware for running tests. See the [[https://www.lavasoftware.org/|LAVA website]] for more details.
-CIP are using a version of LAVA maintained by the [[https://kernelci.org/|KernelCI project]] which [[https://github.com/kernelci/lava-docker/|containerises]] support in a Docker image.
+CIP is using a version of LAVA maintained by the [[https://kernelci.org/|KernelCI project]] which [[https://github.com/kernelci/lava-docker/|containerises]] support in a Docker image.
 === Links ===
-  * CIP LAVA master website: https://lava.ciplatform.org/
+  * CIP LAVA master website: https://lava.ciplatform.org
+  * CIP LAVA documentation: [[civilinfrastructureplatform:ciptesting:centalisedtesting:ciplava|CIP LAVA Wiki]]
   * cip-lava-docker source code: https://gitlab.com/cip-project/cip-testing/lava-docker
+  * How to submit a LAVA test job: [[civilinfrastructureplatform:ciptesting:centalisedtesting:submittinglavajobs|Submitting a LAVA test job]]
+==== Test scheduler workflow ====
+The following schematic shows the workflow, that describes how test images are built and tested with LAVA.
+{{:civilinfrastructureplatform:ciptesting:cip-testing-6.1-workflow.png?800 |}}
+=== Links ===
+  * CIP Testing group: https://gitlab.com/cip-project/cip-testing
+    * [[https://gitlab.com/cip-project/cip-testing/linux-cip-ci|linux-cip-ci]]
+    * [[https://gitlab.com/cip-project/cip-testing/linux-cip-pipelines|linux-cip-pipelines]]
+  * CIP Kernel group: https://gitlab.com/cip-project/cip-kernel
+    * [[https://gitlab.com/cip-project/cip-kernel/linux-cip|linux-cip]]
 ==== Target Control ====
 A LAVA worker (or lab) manages target (virtual/physical hardware) control. There can be one or many different LAVA workers.
-Currently CIP has one LAVA worker (lab-cip-renesas) based in the Renesas UK office. Targets attached to the worker can be viewed on the [[https://lava.ciplatform.org/scheduler/worker/lab-cip-renesas|master website]].
+Currently CIP has three LAVA workers. A summary of all of the attached targets can be viewed on the [[https://lava.ciplatform.org/scheduler/alldevices|LAVA master]].
 CIP encourage as many LAVA workers to be set up as possible and have provided a [[https://gitlab.com/cip-project/cip-testing/lava-docker/blob/cip-lava-docker/documentation/setup_cip_lava_lab.md|guide]] on how to do this.
 === Links ===
-  * lab-cip-renesas: https://lava.ciplatform.org/scheduler/worker/lab-cip-renesas
+  * lab-cip-cybertrust: [[https://lava.ciplatform.org/scheduler/worker/lab-cip-cybertrust]]
+  * lab-cip-mentor: [[https://lava.ciplatform.org/scheduler/worker/lab-cip-mentor]]
+  * lab-cip-renesas: [[https://lava.ciplatform.org/scheduler/worker/lab-cip-renesas]]
   * CIP worker setup guide: https://gitlab.com/cip-project/cip-testing/lava-docker/blob/cip-lava-docker/documentation/setup_cip_lava_lab.md
@@ Line 41: / Line 62: @@
 ==== Test Results ====
-KernelCI / Mailing list
+There needs to be an easy way to view test results, otherwise there is little point in running tests.
+There are three ways to view test results from tests run in the CIP testing infrastructure.
+== LAVA ==
+LAVA hosts it's own [[https://lava.ciplatform.org/results/|results page]] which details what test cases have been run, test results, and links to the relevant logs.
+{{ :civilinfrastructureplatform:ciptesting:lava_results.png |}}
+== KernelCI ==
+KernelCI is a widely used project that aims to unify all upstream Linux kernel testing efforts in order to provide a single place where to store, view, compare and track these results.
+CIP plans to submit our test results to kernelci.org so that they can be viewed by all.
+== Mailing list ==
+The CIP project hosts a [[https://lists.cip-project.org/mailman/listinfo/cip-testing-results|mailing list]] where test results can be reported.
+LAVA can also email any email address specified in the job description.
+== SQUAD ==
+For detailed test reporting CIP uses the Software Quality Dashboard (SQUAD). Build success and test results are reported to the [[https://squad.ciplatform.org/|CIP SQUAD]].
+The following diagram shows how build results are reported to SQUAD and test jobs are submitted to LAVA and monitored by SQUAD.
+{{:civilinfrastructureplatform:ciptesting:cip-testing-with-squad.png?400|}}
+=== Links ===
+  * CIP LAVA master results page: https://lava.ciplatform.org/results
+  * CIP KernelCI instance: https://kernelci.ciplatform.org
+  * CIP test results mailing list: https://lists.cip-project.org/mailman/listinfo/cip-testing-results
+  * CIP SQUAD Reporting: https://squad.ciplatform.org/
 ==== Artifact Storage ====
-AWS S3 Bucket
+A number of different artifacts are required for testing. These may include Kernel, device tree and filesystem binaries, test applications and test data.
+These artifacts need to be stored somewhere where all LAVA workers can access.
+CIP uses Amazon Web Services Simple Cloud Storage Service‎ (AWS S3).
+=== Links ===
+  * CIP artifact storage: http://download.cip-project.org
 ==== CIP Tests ====
-GitLab
+Tests created and shared by CIP are stored in a public GitLab repository for all to use.
+In addition to individual test cases this repository also provides sample LAVA test job definition yaml files that will work with the CIP LAVA infrastructure.
+=== Links ===
+* CIP Kernel tests repository: https://gitlab.com/cip-project/cip-testing/cip-kernel-tests
 ==== Build Server ====
-GitLab
+A key part of any CI/CD setup is the ability to build the software. Builds trigger automatically when a change is made to the software. The new software is then uploaded to the artifact storage and then tested on real hardware in the LAVA infrastructure.
 ==== CIP Source Code ====
-GitLab
+CIP hosts a number of software projects including the SLTS Linux Kernels and CIP-Core - our reference filesystem.
+All CIP software is publicly available on CIP's GitLab site.
+=== Links ===
+  * CIP GitLab home: https://gitlab.com/cip-project
+  * CIP SLTS Linux Kernels: https://gitlab.com/cip-project/cip-kernel/linux-cip
+  * CIP Core: https://gitlab.com/cip-project/cip-core
+  * CIP Testing: https://gitlab.com/cip-project/cip-testing
 ==== CIP Contributions ====
-Mailing List / GitLab pull requests
+Anyone can submit code to any of the CIP projects by either submitting a pull request the relevant GitLab repository or by submitting patches to the cip-dev mailing list.
+Patches that are submitted to the cip-dev mailing list are tracked using patchwork.
+=== Links ===
+  * CIP GitLab home: https://gitlab.com/cip-project
+  * CIP development mailing list: https://lists.cip-project.org/mailman/listinfo/cip-dev
+  * CIP Patchwork: https://patchwork.kernel.org/project/cip-dev/list
 ==== External Tests ====
-Various
+CIP doesn't plan to create duplicates of tests that are already published by other projects. Part of our regular testing is to run through a large number of external test cases. For example, ones provided by the LTP project, or by Linaro.
+=== Links ===
+  * Linux Test Project: http://linux-test-project.github.io
+  * KSelfTest: https://www.kernel.org/doc/Documentation/kselftest.txt
+  * Linaro test definitions: https://github.com/Linaro/test-definitions
 ==== External Source Code ====
-Various
+In many cases CIP needs to reuse code from other projects. For example there is a benefit to build and test the LTS Linux Kernel as this is what the CIP SLTS Kernel is based on. If issues are reported and fixed in LTS pre-releases it will directly improve the stability of the CIP Kernel.
+=== Links ===
+  * Mainline Linux Kernel: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+  * Stable Kernel repository: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
+  * Stable Kernel release candidate repository: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/
+  * Real Time Kernel repository: https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git

Wiki

User Tools

Site Tools

Differences

Page Tools