Differences

This shows you the differences between two versions of the page.

--- realtime:documentation:howto:debugging:cyclictest:debugging [2018/07/18 13:33]
ebugden Clarify explanations and add examples
+++ — (current)
@@ Line 1: / Line 1: @@
-====== Cyclictest - Latency debugging with tracing ======
-Tracing is useful when trying to determine the cause of a latency because it can reveal relevant details about program execution. There are several options in Cyclictest that can be used to produce traces via Ftrace. These traces can then be consulted via tracefs.
-===== Break Trace Option =====
-A trace can be collected through Cyclictest by using the break trace option together with one of the tracing options. The break trace option (--breaktrace) will stop tracing and end Cyclictest if it detects a latency that is longer than a specified limit. This option is intended to be used with one of the Cyclictest options that enables a specific Ftrace tracer (e.g. --irqsoff).
-===== Selecting Instrumentation =====
-Cyclictest offers a number options which use different Ftrace tracers and instrumentation. These options are described in the Cyclictest help text (--help) and on the man page. The most important things to keep in mind when choosing instrumentation are the trace detail and the resulting overhead. A trace with more detail will require a higher overhead. In general, it is effective to corner the source of the latency over several iterations of tracing and analysis while gradually increasing the trace detail. So start with a small amount of instrumentation, which produces a simple trace. One good option would be to start by eliminating the most obvious possible sources of latency by tracing when interrupts or preemption are disabled.
-It can sometimes be tempting to immediately jump to enabling function tracing. A full function trace is highly detailed because it includes all of the function calls that were made in the kernel. Function tracing can be very useful in the debugging process, but starting with such a highly detailed trace is not recommended. Analyzing a full function trace can be time consuming and confusing, especially if the latency's cause is complex. To make matters worse, function tracing has a significant overhead so it has a higher chance of affecting the behavior of the system. Because of this high overhead, is some cases it is possible that the targeted latency will not occur when function tracing is enabled.
-===== Latency Tracing =====
-There are a couple steps that can help correctly produce a trace that includes the relevant latency. The following tips are relevant if the goal is to produce a trace that includes the system's maximum latency. In general, it is a good idea to start by resolving the longest latencies in a system and to work down towards solving the shorter latencies.
-Before tracing, make sure that the correct options for measuring the have been chosen and that Cyclictest detects the latency without having any tracing enabled. Adding tracing will not help if the latency is never detected in the first place. Some suggestions for choosing the right Cyclictest options for a particular situation can be found on the Cyclictest [[realtime:documentation:howto:debugging:cyclictest:test-design|Test Design]] page.
-==== Measuring the Tracing Overhead ====
-When trying to produce a trace that includes a specific latency it is important to always adjust the break trace limit based on the observed tracing overhead. When tracing is enabled, the normal operation of the system will take longer and so the system's latencies will increase in length. For example, a system with an average latency of 1 us could have an average latency of 10 us with tracing enabled. If a maximum latency of around 130 us was being observed without tracing, the maximum latency with tracing could be around 200 us.
-The easiest way to measure the overhead for a specific set of tracing instrumentation is to run Cyclictest with the desired tracing options and with a break trace limit that is much longer than the previously observed latency. Under these conditions, the maximum latency measured with Cyclictest will include the tracing overhead.
-Here is an example. On a certain system, the following Cyclictest command is run:
-<code bash>
-cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0
-</code>
-and after a couple hours a maximum latency of 130 us is detected. After a few iterations of analysis using smaller amounts of instrumentation, the latency is to be traced using function tracing (--ftrace). To measure the same maximum latency including the function tracing overhead, the following Cyclictest command is run:
-<code bash>
-cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0 --breaktrace=2000 --ftrace
-</code>
-and after a couple of hours a latency of 190 us is observed. This new maximum latency will be used to determine the appropriate break trace limit for tracing the latency.
-Since the intent of this step is just to observe the overhead, the break trace limit should be much longer than the length of the expected latencies and the tracing should never actually break. If the tracing does break, run Cyclictest again with a higher break trace limit. The magnitude of the increase of the maximum latency due to tracing depends on what generates the latency, so it is important to be critical and to make sure that the high latency has actually occurred and the suspected spike is not just an elevated baseline due to the tracing. If the latency spikes at a similar time or in a similar situation as before the tracing was enabled, it is a good sign that it is the same latency as before.
-==== Producing a Trace ====
-After measuring the overhead, Cyclictest can be run again with a break trace limit that is slightly below the maximum latency observed with the tracing overhead. Continuing the example from the previous section, if the maximum latency with the overhead from function tracing is 190 us then the following command should be used to produce the trace:
-<code bash>
-cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0 --breaktrace=180 --ftrace
-</code>
-using a break trace limit of 180 us. This new limit will cause Cyclictest to stop when the maximum latency occurs and will produce the relevant trace. This trace can then be viewed via tracefs in the same way as other Ftrace traces.

Wiki

User Tools

Site Tools

Differences

Page Tools