hwlat is an Ftrace tracer designed to detect latencies caused by a system's firmware or hardware. The tracer was originally intended for detecting SMIs, but it can detect other low level latencies as well.

This article is brief because it is dedicated to a specific Ftrace tracer. More complete information about how to use Ftrace can be found here. More details about this tracer can be found in the kernel documentation for Ftrace and in the kernel documentation for the Hardware Latency Detector.

Before any Ftrace tracers can be used, the tracefs file system must be mounted.

If the kernel is already configured correctly and the hwlat tracer is available, hwlat will be listed in the available_tracers file in tracefs (after mounting tracefs). Otherwise, the hwlat tracer can be included in the kernel by enabling the CONFIG_HWLAT_TRACER option.

If the tracer is available, hwlat can be used via tracefs in the same way as any other Ftrace tracer.

Before consulting the trace results for analysis, make sure that only one instance of hwlat is running. The program top can be used for this. If more than one instance is running, restart the system, and trace the latencies again with only one instance of hwlat.

Because of a bug, an additional instance of hwlat is started every time 1 is written to tracing_on regardless of whether or not tracing with hwlat is already enabled. When tracing is disabled, only one instance of hwlat will be stopped and the only way to stop the additional instances is to restart the system. Running multiple instances of hwlat can quickly cripple a system so it is best to avoid this.

The kernel documentation for Ftrace and for the hardware latency detector contain details about how to interpret the resulting traces.