This page presents some of the concepts to keep in mind when setting up a system for measuring latencies and when selecting Cyclictest options. The purpose of this information is to facilitate measuring latencies correctly. However, measuring latencies and evaluating a system's performance remains a difficult task that requires an in-depth knowledge of the platform, its intended final purpose (application), and its latency requirements.

When learning how to design a test, it is useful to have a look at examples of results that show the effect of different loads and parameters on the latencies measured by Cyclictest. It is also important to understand how Cyclictest works before reading the rest of the content on this page.

Cyclictest measures the latencies that occur in the one particular situation for which it is configured. In other words, running Cyclictest once will not identify all the possible latencies in all possible situations on a system. If Cyclictest is being used to debug a latency, the options and system load should be chosen with the goal of observing that latency. However, if Cyclictest is being used to globally evaluate a system's performance, then many different runs of Cyclictest with different parameters and loads are required.

To get the most accurate latency measurements, Cyclictest should be executed while the evaluated system is running a load as similar as possible to the RT application for which the system is intended. During a typical test Cyclictest's overhead is relatively negligible, so it does not really apply a load to the system. A few ways to load the system are suggested below.

The best way to measure the latencies that an RT application would experience on a specific system is to run the actual RT application on the platform along with Cyclictest and any other non-RT application that would normally be running as well. This method has its disadvantages. For example, if the applications cause system latencies very infrequently then it could take a very long time for Cyclictest to notice them.

Another solution is to simulate the load generated by the final application. This can be partially done using a tool like Hackbench (available in the rt-tests suite). Ideally, the simulation should be as close as possible to the final application in all respects including CPU use, memory use, I/O, number of tasks, network use, etc. Examples of how to generate different loads for testing real-time systems can be found here. However, even if the application is simulated correctly it is possible that an artificial load would not trigger system latencies that the final application would trigger. In this case, the Cyclictest results might not reflect the real maximum latencies that the application would experience. Accurately simulating the load that an application will apply on a system tends to be very complex and difficult, so running a simulated load that stresses the system more than the application is predicted to stress it is often done instead.

In the end, it can be valuable to run tests using both techniques. If some tests are run using an artificial load and some tests are run using the real applications as the load, then there is a better chance of detecting all the possible latencies that the RT application could experience.

Cyclictest measures the difference between the expected execution time of a group of periodic measuring tasks and their actual execution time. So Cyclictest will detect a latency if and only if the latency prevents a measuring task from executing on time. This principle should guide the selection of the Cyclictest options, otherwise relevant latencies could be missed. For example, if the execution period of the measuring task is very long and the latencies always occur and are resolved between the times when the measuring task is supposed to execute, then these latencies will never be noticed by Cyclictest because they never delay the execution of the measuring task.

Here is a list of options that should always be considered when measuring latencies with Cyclictest:

• Number and affinity of measuring threads (–threads, –affinity)
• Thread wake-up interval (–interval, –distance)
• Thread real-time priority (–priority)
• Test duration
• Prevent Cyclictest pages from being paged out of memory (–mlockall)

When running Cyclictest on a symmetric multi-processing (SMP) system, specifying the –smp option can be useful as it takes care of some of the options mentioned above without having to specify them all individually.

When designing a test, it is important to minimize the effect that running Cyclictest could have on the latencies that are being observed in a system. For example, when measuring the latencies on a specific CPU, make sure that the main Cyclictest thread is not running on that CPU, otherwise the additional context switches will influence the latencies that are measured. A technique for doing this is presented on the Cyclictest FAQ.