This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
realtime:documentation:howto:debugging:cyclictest:test-design:start [2018/07/23 09:34] ebugden Mention smp option |
realtime:documentation:howto:debugging:cyclictest:test-design:start [2018/08/21 10:48] ebugden Move from temporary location to main tools page |
||
---|---|---|---|
Line 1: | Line 1: | ||
====== Cyclictest - Test Design ====== | ====== Cyclictest - Test Design ====== | ||
- | Measuring latencies correctly can be complex and time consuming. To make the process less confusing, this page explains some of the concepts to keep in mind when setting up a system for measuring latencies and when selecting Cyclictest options. However, measuring latencies and evaluating a system's performance remains a difficult task that requires an in-depth knowledge of the platform, its intended final purpose (application), and its latency requirements. | + | This page presents some of the concepts to keep in mind when setting up a system for measuring latencies and when selecting Cyclictest options. The purpose of this information is to facilitate measuring latencies correctly. However, measuring latencies and evaluating a system's performance remains a difficult task that requires an in-depth knowledge of the platform, its intended final purpose (application), and its latency requirements. |
+ | When learning how to design a test, it is useful to have a look at [[realtime:documentation:howto:tools:cyclictest:result-examples|examples of results]] that show the effect of different loads and parameters on the latencies measured by Cyclictest. It is also important to understand [[realtime:documentation:howto:tools:cyclictest:start#technical-explanation|how Cyclictest works]] before reading the rest of the content on this page. | ||
+ | |||
+ | ---- | ||
+ | |||
+ | Cyclictest measures the latencies that occur in the one particular situation for which it is configured. In other words, running Cyclictest once will not identify all the possible latencies in all possible situations on a system. If Cyclictest is being used to debug a latency, the options and system load should be chosen with the goal of observing that latency. However, if Cyclictest is being used to globally evaluate a system's performance, then many different runs of Cyclictest with different parameters and loads are required. | ||
===== System Load ===== | ===== System Load ===== | ||
To get the most accurate latency measurements, Cyclictest should be executed while the evaluated system is running a load as similar as possible to the RT application for which the system is intended. During a typical test Cyclictest's overhead is relatively negligible, so it does not really apply a load to the system. A few ways to load the system are suggested below. | To get the most accurate latency measurements, Cyclictest should be executed while the evaluated system is running a load as similar as possible to the RT application for which the system is intended. During a typical test Cyclictest's overhead is relatively negligible, so it does not really apply a load to the system. A few ways to load the system are suggested below. | ||
- | The best way to measure the latencies that an RT application would experience on a specific system is to run the actual RT application on the platform along with Cyclictest. This method has its disadvantages. For example, if the RT application causes system latencies very infrequently then it could take a very long time for Cyclictest to notice them. | + | The best way to measure the latencies that an RT application would experience on a specific system is to run the actual RT application on the platform along with Cyclictest and any other non-RT application that would normally be running as well. This method has its disadvantages. For example, if the applications cause system latencies very infrequently then it could take a very long time for Cyclictest to notice them. |
- | Another solution is to simulate the load generated by the final RT application. This can be done using a tool like Hackbench (available in the [[realtime:documentation:howto:tools:rt-tests|rt-tests]] suite). Ideally, the simulation should be as close as possible to the final application in all respects including CPU, memory use, I/O, number of tasks, network use, etc. However, even if the application is simulated correctly it is possible that an artificial load would not trigger system latencies that the final RT application would trigger. In this case, the Cyclictest results might not reflect the real maximum latencies that the application would experience. Accurately simulating the load that an application will apply on a system tends be very complex and difficult, so running a simulated load that stresses the system more than the application is predicted to stress it is often done instead. | + | Another solution is to simulate the load generated by the final application. This can be done using a tool like Hackbench (available in the [[realtime:documentation:howto:tools:rt-tests|rt-tests]] suite). Ideally, the simulation should be as close as possible to the final application in all respects including CPU use, memory use, I/O, number of tasks, network use, etc. Examples of how to generate different loads for testing real-time systems can be found [[realtime:documentation:howto:tools:worstcaselatency|here]]. However, even if the application is simulated correctly it is possible that an artificial load would not trigger system latencies that the final application would trigger. In this case, the Cyclictest results might not reflect the real maximum latencies that the application would experience. Accurately simulating the load that an application will apply on a system tends to be very complex and difficult, so running a simulated load that stresses the system more than the application is predicted to stress it is often done instead. |
- | In the end, it can be valuable to run tests using both techniques. If some tests are run using an artificial load and some tests are run using the RT application as the load, then there is a better chance of detecting all the possible latencies that the application could experience. | + | In the end, it can be valuable to run tests using both techniques. If some tests are run using an artificial load and some tests are run using the real applications as the load, then there is a better chance of detecting all the possible latencies that the RT application could experience. |
===== Option Selection ===== | ===== Option Selection ===== | ||
Line 17: | Line 22: | ||
Cyclictest measures the difference between the expected execution time of a group of periodic measuring tasks and their actual execution time. So Cyclictest will detect a latency if and only if the latency prevents a measuring task from executing on time. This principle should guide the selection of the Cyclictest options, otherwise relevant latencies could be missed. For example, if the execution period of the measuring task is very long and the latencies always occur and are resolved between the times when the measuring task is supposed to execute, then these latencies will never be noticed by Cyclictest because they never delay the execution of the measuring task. | Cyclictest measures the difference between the expected execution time of a group of periodic measuring tasks and their actual execution time. So Cyclictest will detect a latency if and only if the latency prevents a measuring task from executing on time. This principle should guide the selection of the Cyclictest options, otherwise relevant latencies could be missed. For example, if the execution period of the measuring task is very long and the latencies always occur and are resolved between the times when the measuring task is supposed to execute, then these latencies will never be noticed by Cyclictest because they never delay the execution of the measuring task. | ||
- | Here is a list of options that should always be specified when measuring latencies with Cyclictest: | + | Here is a list of options that should always be considered when measuring latencies with Cyclictest: |
- | * [[realtime:documentation:howto:debugging:cyclictest:test-design:threads|Number of measuring threads]] (--threads) | + | * [[realtime:documentation:howto:debugging:cyclictest:test-design:threads|Number and affinity of measuring threads]] (--threads, --affinity) |
- | * CPUs executing the measuring threads (--affinity) | + | |
* [[realtime:documentation:howto:debugging:cyclictest:test-design:interval|Thread wake-up interval]] (--interval, --distance) | * [[realtime:documentation:howto:debugging:cyclictest:test-design:interval|Thread wake-up interval]] (--interval, --distance) | ||
* [[realtime:documentation:howto:debugging:cyclictest:test-design:priority|Thread real-time priority]] (--priority) | * [[realtime:documentation:howto:debugging:cyclictest:test-design:priority|Thread real-time priority]] (--priority) | ||
* [[realtime:documentation:howto:debugging:cyclictest:test-design:duration|Test duration]] | * [[realtime:documentation:howto:debugging:cyclictest:test-design:duration|Test duration]] | ||
* Prevent Cyclictest pages from being paged out of memory (--mlockall) | * Prevent Cyclictest pages from being paged out of memory (--mlockall) | ||
+ | * Use clock_nanosleep for high-resolution thread sleeping with a faster expiry path than POSIX interval timers (--nanosleep) | ||
+ | |||
+ | Using the --mlockall option, which prevents Cyclictest pages from being paged out, reduces the Cyclictest overhead by avoiding having to reload pages from the disk. | ||
+ | |||
+ | The POSIX interval timers are signal based and their expiry path is slow because it goes through the hrtimer softirq. On the other hand nanosleep has a much less expensive expiry path because it is just a wakeup. | ||
When running Cyclictest on a symmetic multi-processing (SMP) system, specifying the --smp option can be useful as it takes care of some of the options mentioned above without having to specify them all individually. | When running Cyclictest on a symmetic multi-processing (SMP) system, specifying the --smp option can be useful as it takes care of some of the options mentioned above without having to specify them all individually. | ||
- | ===== Interactions ===== | + | ===== Observer Effect ===== |
- | When measuring the latencies in a system, it is important to think about interactions and how running Cyclictest could potentially affect the latencies that are being observed. | + | When designing a test, it is important to minimize the effect that running Cyclictest could have on the latencies that are being observed in a system. For example, when measuring the latencies on a specific CPU, make sure that the main Cyclictest thread is not running on that CPU, otherwise the additional context switches will influence the latencies that are measured. A technique for doing this is presented on the [[realtime:documentation:howto:tools:cyclictest:faq#how-can-the-influence-of-cyclictest-be-minimized-when-evaluating-latencies-on-an-isolated-set-of-cpus|Cyclictest FAQ]]. |