The rtla – Real-time Linux Analysis toolset analyzes real-time properties of the Linux kernel.

The tools has several modes. The default top mode displays periodic summary from a tracer. The hist mode displays histogram at the end of operation. Automatic mode (--auto) stops when the threshold is reached.

The tool offers several commands:

• rtla timerlat measures IRQ and kernel thread timer latency with timerlat tracer. Modes: top, hist.

• rtla osnoise provides information about operating system noise measured with osnoise tracer. It dispatches kernel threads per CPU to measure various sources of interference during execution, such as preemption, softirq, and IRQs. Modes: top, hist.

• rtla hwnoise detects and quantifies hardware-related noise with osnoise tracer. By disabling interrupts with OSNOISE_IRQ_DISABLE and scheduling of threads, only non-maskable interrupts and hardware-related noise are analyzed.

Each of these commands has various options for configuration, such as setting tracer periods, runtimes, thresholds, scheduling parameters, trace events and triggers, and more. These commands offer different ways to analyze and understand the real-time properties and noise sources in the Linux operating system.

rtla commands include several command line options that can be used to tune various parameters of the system affecting its real-time properties.

• –dma-latency <n> - this holds /dev/cpu_dma_latency at the specified value during the measurement. Note that unlike cyclictest, the option is not enabled by default in rtla.
• –deepest-idle-state <n> - limits the deepest allowed idle state via cpuidle interface, only on CPUs where measurement is running, unlike –dma-latency, which affects all CPUs.

Setting the deepest idle state instead of DMA latency helps with power consumption, but might lead to increased latencies on some systems.

All rtla tools include latency source auto-analysis, enabled with -a <us> option, which takes the duration of a latency in microseconds for rtla to stop tracing at. Auto-analysis uses tracepoints to capture thread and IRQ interference to the real-time workload, which are collected, processed, and finally formatted and printed at the end of the measurement.

If auto-analysis is not enough to determine the cause of the latency, additional tracepoints may be enabled using the -e <event_name> option. Arguments to enable common events for automatic and trace options:

-e sched:sched_switch -e sched:sched_wakeup -e sched:sched_migrate_task -e irq -e irq_vectors -e timer -e workqueue

To see all enabled events, run:

grep 1 /sys/kernel/debug/tracing/instances/ /sys/kernel/debug/tracing/events/ -R --include enable

To be able to see captured events after the end of the rtla run, trace output has to be enabled using the –trace/-t option, unless -a is used, which enables -t automatically.

rtla supports integrating external tools into latency measurement by the means of actions. Actions can run a command or send a signal to a process whenever tracing is stopped on latency threshold set with -T/-i/-a (for timerlat; -s/-S/-a for osnoise), or at the end of tracing. A special action resumes tracing after stop, if requested.

External tooling might be used in cases where rtla's tracefs-based diagnostics are insufficient. See the rtla manpages for how to use actions.

RT tools and utilities
Linux scheduling latency debug and analysis with RTLA
🎥 rtla timerlat: Debugging Real-time Linux Scheduling Latency, video
user mode source
trace_osnoise.c: trace_sample_threshold() - kernel mode source
RTLA uses libtracefs