This library provides instrumentation for detecting different kinds of lockups.
The library has two tools, a critical section detector and a CPU heartbeat checker.
The critical section detector detects when the kernel has remained in a critical section for “too long”. Critical sections are marked using LOCKUP_TIMED_BEGIN() and LOCKUP_TIMED_END(). The code executing during an SMC call is an example of a critical section. A function that temporarily disables interrupts would be another example.
Currently, detection is performed in two ways.
First, each time a CPU leaves a critical section (calls LOCKUP_TIMED_END()), it will observe the time spent in the critical section, and then update kcounters which track the number of long running critical sections we have seen. Additionally, it will track the “worst case” critical section time for that CPU.
Second, If a CPU calls LOCKUP_TIMED_BEGIN(), but never calls LOCKUP_TIMED_END(), and the CPU has spent more than the configured threshold amount of time in the critical section, a lockup will be reported as a KERNEL_OOPS by another one of the CPUs when it is performing a heartbeat check. If the amount of time spent by the locked-up CPU in the critical section exceeds the fatal threshold, the kernel will generate a crashlog and reboot, indicating a reboot reason of SOFTWARE_WATCHDOG in the crashlog as it does.
The k lockup status command can be used to check if a CPU is currently in a critical section, and to print the current worst case critical section times for each CPU.
Choosing appropriate thresholds for the critical section detector is important. The values should be small enough to detect performance impacting lockups, but large enough to avoid false alarms, especially when running on virtualized hardware. Setting both of these values to 0 will completely disable critical section lockup detection.
See also kernel.lockup-detector.critical-section-threshold-ms and kernel.lockup-detector.critical-section-fatal-threshold-ms.
The heartbeat checker is used to detect when a CPU has stopped responding to interrupts.
When enabled, all CPUs will run a periodic timer callback that emits a “heartbeat” by updating a timestamp in its per CPU structure. Afterwards, they will check each of the other CPUs' last heartbeat and if the last heartbeat is older than the configured threshold, a (rate limited) KERNEL_OOPS will be emitted. If the time since last heartbeat exceeds the fatal threshold, a crashlog will be generated and the kernel will reboot, indicating a reboot reason of SOFTWARE_WATCHDOG in the crashlog as it does.
kernel.lockup-detector.heartbeat-period-ms controls how frequently the CPUs emit heartbeats and perform checks.
kernel.lockup-detector.heartbeat-age-threshold-ms controls how long a CPU can go without emitting a heartbeat before it is considered to be locked up and a KERNEL_OOPS is generated.
kernel.lockup-detector.heartbeat-age-fatal-threshold-ms controls how long a CPU can go without emitting a heartbeat before a SOFTWARE_WATCHDOG reboot is triggered.
A future version of this library may add instrumentation for detecting when a CPU has been “executing kernel code for too long”, or has been in a hypervisor or Secure Monitor call for too long.