User Tools

Site Tools


civilinfrastructureplatform:ciptesting:centalisedtesting:cioverview

This is an old revision of the document!


Continuous Integration Testing Overview

The CIP CI infrastructure monitors the CIP Kernel branches and upstream stable release candidates for changes. Once a change is detected a build process is kicked off using GitLab CI/CD that provides binaries for a number of different Kernel configurations. These binaries are then tested on real hardware (where appropriate) using LAVA.

Tests

Boot Test (Healthcheck)

This checks that the provided Kernel will boot on the target device.

Spectre/Meltdown Checker

Linux Test Project (LTP)

LTP consists of a collection of tools for testing the Linux Kernel. More information can be found on the LTP website.

We currently support the following LTP tests:

  • ltp-cve-tests
  • ltp-dio-tests
  • ltp-fs-tests
  • ltp-ipc-tests
  • ltp-math-tests
  • ltp-sched-tests
  • ltp-syscalls-tests
  • ltp-timers-tests

The tests can take a long time to run, so currently they are only run during CIP Kernel release candidate testing.

Real-Time Tests

Some tests are designed specifically for Real-Time Linux enabled Kernels.

Cyclictest

Cyclicdeadline

Hackbench

Cyclictest+Hackbench

This test measures event latency in the Linux Kernel, with hackbench running in the background to stress the system. This provides a more realistic use case for cyclictest.

When Tests are Run

Kernel Normal Testing Release Candidate Testing
v5.10.y-cip boot
smc
boot
smc
ltp
v5.10.y boot
smc
N/A
v4.19.y-cip boot
smc
boot
smc
ltp
v4.19.y-cip-rt boot
smc
cyclictest+hackbench
boot
smc
ltp
cyclictest+hackbench
v4.19.y boot
smc
N/A
v4.4.y-cip boot
smc
boot
smc
ltp
v4.4.y-cip-rt boot
smc
cyclictest+hackbench
boot
smc
ltp
cyclictest+hackbench
v4.4.y boot
smc
N/A

CIP Kernel Configurations

The Kernel configurations that CIP is using are stored in the cip-kernel-config repository.

The linux-cip-pipelines repository stores the GitLab CI configuration files that specify what Kernel configurations are built and tested, including the specific devices used and test cases run.

The below table maps which of these configurations is automatically built and tested by the CIP CI infrastructure.

Kernel Arch Configuration Built Tested Target Devices Device Tree
v5.10.y-cip arm cip_bbb_defconfig Y N - -
hitachi_cyclonev_defconfig Y N - -
hitachi_omap_defconfig Y N - -
moxa_mxc_defconfig Y N - -
renesas_shmobile_defconfig Y Y r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
siemens_de0-nano-soc_defconfig Y Y de0-nano-soc socfpga_cyclone5_de0_nano_soc.dtb
siemens_imx6.config Y N - -
arm64 ctj_zynqmp_defconfig Y Y zynqmp-zcu102 zynqmp-zcu102-rev1.0.dtb
moxa_eds_defconfig Y N - -
renesas_defconfig Y Y r8a774a1-hihope-rzg2m-ex r8a774a1-hihope-rzg2m-ex.dtb
x86 cip_qemu_defconfig Y Y qemu N/A
plathome_obsvx2.config Y N - -
siemens_iot2000.config Y N - -
siemens_ipc227e_defconfig Y Y x86-simatic-ipc227e N/A
siemens_server_defconfig Y N - -
toshiba_atom_baytrail_cip.config Y N - -
v5.10.y arm cip_bbb_defconfig Y N - -
hitachi_cyclonev_defconfig Y N - -
hitachi_omap_defconfig Y N - -
moxa_mxc_defconfig Y N - -
renesas_shmobile_defconfig Y Y r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
siemens_de0-nano-soc_defconfig Y Y de0-nano-soc socfpga_cyclone5_de0_sockit.dtb
siemens_imx6.config Y N - -
arm64 moxa_eds_defconfig Y N - -
renesas_defconfig Y N - -
x86 cip_qemu_defconfig Y Y qemu N/A
plathome_obsvx2.config Y N - -
siemens_iot2000.config Y N - -
siemens_ipc227e_defconfig Y Y x86 N/A
siemens_server_defconfig Y N - -
toshiba_atom_baytrail_cip.config Y N - -
v4.19.y-cip arm cip_bbb_defconfig Y N - -
hitachi_cyclonev_defconfig Y N - -
hitachi_omap_defconfig Y N - -
moxa_mxc_defconfig Y N - -
renesas_shmobile_defconfig Y Y r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
siemens_de0-nano-soc_defconfig Y Y de0-nano-soc socfpga_cyclone5_de0_nano_soc.dtb
siemens_imx6.config Y N - -
arm64 ctj_zynqmp_defconfig Y Y zynqmp-zcu102 zynqmp-zcu102-rev1.0.dtb
moxa_eds_defconfig Y N - -
renesas_defconfig Y Y r8a774a1-hihope-rzg2m-ex r8a774a1-hihope-rzg2m-ex.dtb
x86 cip_qemu_defconfig Y Y qemu N/A
plathome_obsvx2.config Y N - -
siemens_iot2000.config Y N - -
siemens_ipc227e_defconfig Y Y x86-simatic-ipc227e N/A
siemens_server_defconfig Y N - -
toshiba_atom_baytrail_cip.config Y N - -
v4.19.y-cip-rt arm cip_bbb_defconfig Y N - -
hitachi_cyclonev_defconfig Y N - -
hitachi_omap_defconfig Y N - -
moxa_mxc_defconfig Y N - -
renesas_shmobile_defconfig Y Y r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
renesas_shmobile-rt_defconfig Y Y1 r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
siemens_de0-nano-soc_defconfig Y Y de0-nano-soc socfpga_cyclone5_de0_nano_soc.dtb
siemens_imx6.config Y N - -
socfpga_defconfig Y N - -
arm64 ctj_zynqmp_defconfig Y Y zynqmp-zcu102 zynqmp-zcu102-rev1.0.dtb
moxa_eds_defconfig Y N - -
renesas_defconfig Y Y r8a774a1-hihope-rzg2m-ex r8a774a1-hihope-rzg2m-ex.dtb
renesas-rt_defconfig Y Y1 r8a774a1-hihope-rzg2m-ex r8a774a1-hihope-rzg2m-ex.dtb
x86 cip_qemu_defconfig Y Y qemu N/A
plathome_obsvx2.config Y N - -
siemens_i386-rt.config Y N - -
siemens_iot2000.config Y N - -
siemens_ipc227e_defconfig Y Y x86-simatic-ipc227e N/A
siemens_server_defconfig Y N - -
toshiba_atom_baytrail_cip.config Y N - -
v4.19.y arm cip_bbb_defconfig Y N - -
hitachi_cyclonev_defconfig Y N - -
hitachi_omap_defconfig Y N - -
moxa_mxc_defconfig Y N - -
renesas_shmobile_defconfig Y Y r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
siemens_de0-nano-soc_defconfig Y Y de0-nano-soc socfpga_cyclone5_de0_sockit.dtb
siemens_imx6.config Y N - -
arm64 ctj_zynqmp_defconfig Y Y zynqmp-zcu102 zynqmp-zcu102-rev1.0.dtb
moxa_eds_defconfig Y N - -
renesas_defconfig Y N - -
x86 cip_qemu_defconfig Y Y qemu N/A
plathome_obsvx2.config Y N - -
siemens_iot2000.config Y N - -
siemens_ipc227e_defconfig Y Y x86-simatic-ipc227e N/A
siemens_server_defconfig Y N - -
toshiba_atom_baytrail_cip.config Y N - -
v4.4.y-cip arm cip_bbb_defconfig Y N - -
hitachi_omap_defconfig Y N - -
moxa_mxc_defconfig Y N - -
renesas_shmobile_defconfig Y Y r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
siemens_am335x-axm2_defconfig Y N - -
siemens_am335x-draco_defconfig Y N - -
siemens_am335x-dxr2_defconfig Y N - -
siemens_am335x-etamin_defconfig Y N - -
siemens_am57xx-pxm3.config Y N - -
siemens_dcu2.config Y N - -
siemens_imx6_defconfig Y N - -
toshiba_tegra_defconfig Y N - -
toshiba_zynq_defconfig Y N - -
x86 cip_qemu_defconfig Y Y qemu N/A
plathome_obsvx1.config Y N - -
siemens_iot2000.config Y N - -
siemens_server_defconfig Y N - -
toshiba_defconfig Y N - -
v4.4.y-cip-rt arm cip_bbb_defconfig Y N - -
hitachi_omap_defconfig Y N - -
moxa_mxc_defconfig Y N - -
renesas_shmobile_defconfig Y Y r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
renesas_shmobile-rt_defconfig Y Y1 r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
siemens_am335x-axm2_defconfig Y N - -
siemens_am335x-draco_defconfig Y N - -
siemens_am335x-dxr2_defconfig Y N - -
siemens_am335x-etamin_defconfig Y N - -
siemens_am57xx-pxm3.config Y N - -
siemens_dcu2.config Y N - -
siemens_imx6_defconfig Y N - -
toshiba_tegra_defconfig Y N - -
toshiba_zynq_defconfig Y N - -
x86 cip_qemu_defconfig Y Y qemu N/A
plathome_obsvx1.config Y N - -
siemens_i386-rt.config Y N - -
siemens_iot2000.config Y N - -
siemens_server_defconfig Y N - -
toshiba_defconfig Y N - -
v4.4.y arm cip_bbb_defconfig Y N - -
hitachi_omap_defconfig Y N - -
moxa_mxc_defconfig Y N - -
renesas_shmobile_defconfig Y N - -
siemens_am335x-axm2_defconfig Y N - -
siemens_am335x-draco_defconfig Y N - -
siemens_am335x-dxr2_defconfig Y N - -
siemens_am335x-etamin_defconfig Y N - -
siemens_am57xx-pxm3.config Y N - -
siemens_dcu2.config Y N - -
siemens_imx6_defconfig Y N - -
toshiba_tegra_defconfig Y N - -
toshiba_zynq_defconfig Y N - -
x86 cip_qemu_defconfig Y Y qemu N/A
plathome_obsvx1.config Y N - -
siemens_iot2000.config Y N - -
siemens_server_defconfig Y N - -
toshiba_defconfig Y N - -

1 Tested with Real Time specific tests

Extra Kernel Configurations

CIP also builds and tests a few non-CIP related Kernel configurations that are present in the Kernel.

The below table maps which of these configurations is automatically built and tested by the CIP CI infrastructure.

Kernel Arch Configuration Built Tested Target Devices Device Tree
v5.10.y arm shmobile_defconfig Y Y r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
arm multi_v7_defconfig Y Y r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
arm64 defconfig Y Y r8a774a1-hihope-rzg2m-ex r8a774a1-hihope-rzg2m-ex.dtb
x86 defconfig Y N - -
v4.19.y arm shmobile_defconfig Y Y r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
arm multi_v7_defconfig Y Y r8a7743-iwg20d-q7 r8a7743-iwg20d-q7-dbcm-ca.dtb
arm64 defconfig Y N - -
x86 defconfig Y N - -
v4.4.y arm shmobile_defconfig Y N - -
arm multi_v7_defconfig Y N - -
x86 defconfig Y N - -
civilinfrastructureplatform/ciptesting/centalisedtesting/cioverview.1626948003.txt.gz · Last modified: 2021/07/22 10:00 by patersonc