This document describes thread signaling, a Zircon kernel mechanism used to implement thread suspend and kill operations. Thread signaling is not related to object signaling.
The target audience is kernel developers and anyone interested in understanding how suspend and kill operations work in the kernel.
Suspend and kill are operations that can be performed on threads. Both of these operations are asynchronous in that the caller must wait for the operation to complete. Inside the kernel, these operations are implemented as instance methods on the Thread struct:
Thread::Suspend - Request a thread to suspend its execution until it is resumed via Thread::Resume. Suspend is used to implement debuggers. Once suspended, a thread's register state can be read/written prior to resuming it. This operation is exposed to user mode via zx_task_suspend().
Thread::Kill - Request a thread to terminate itself. This operation is not directly exposed to user mode. That is, attempting to zx_task_kill() a thread is an error. However, this operation is indirectly exposed via process destruction, both voluntary and involuntary.
Notice that both of these operations are described as requests. The caller is requesting that the target suspend or, in the case of kill, terminate its execution. The caller has no ability to forcibly suspend or terminate the target. While the target cannot refuse the request, it can delay action until the appropriate time and place. This is a key element of the design.
To understand why these operations are requests, consider the alternative of forcibly killing or suspending a thread. If a thread is forcibly killed while holding a resource (like a mutex) then it won‘t get the chance to free the resource before it’s destroyed. You could end up with memory leaks, permanently locked locks, corrupted data structures, all sorts of bad stuff.
By modeling kill and suspend as requests that can only be performed by the target thread, we provide a way for the target to free its resources and perform any necessary cleanup before it stops executing, temporarily (in the case of suspend) or permanently (in the case of kill).
Before we cover how kill and suspend requests are issued, let's talk about the safety of thread termination.
There is one place where it's always safe for a thread to suspend or terminate its execution, the “edge” of the kernel, just before returning from the kernel back to user mode. Before returning to user mode, the thread unwinds its callstack, executing the destructors of any RAII objects. By the time it has reached the edge and is about to return to user mode, there will be nothing left on the kernel stack. It is here that a thread may safely suspend or terminate its execution.
Concretely, there are two safe points at which a thread may suspend or terminate. They are just before returning to user mode from a syscall and just before returning to user mode from an exception/fault/interrupt handler (exception handler, for short).
Note, exception handlers are not just invoked when executing in user mode. They can also be invoked when executing in kernel mode. When returning back to kernel mode it is not safe to suspend or terminate because the outer kernel mode context may still be holding a resource. In other words, an exception handler is only a safe point when it is triggered from a user mode context.
So we know that kill and suspend are merely requests and that it's up to the target thread to decide when and how to fulfill the request. We also know that the only safe places for a thread to suspend or terminate itself are at the edges of the kernel, just before returning to user mode. How do thread signals fit into all this?
Thread signals are the mechanism by which suspend and kill are requested. Each Thread object has a field containing the set of asserted signals. There's a bit for suspend, THREAD_SIGNAL_SUSPEND, and a bit for kill, THREAD_SIGNAL_KILL.
Requesting a thread to suspend or terminate is achieved by setting the appropriate bit on the target Thread object and then, depending on the target‘s state, poking it in some way to ensure it reaches a safe point in a timely fashion. The exact type of poke depends on the target thread’s state: sleep/blocked, suspended, or running. Note, there are two flavors of sleeping/blocked, interruptible and uninterruptible. We'll focus on interruptible and ignore uninterruptible.
If the target thread is sleeping or blocked then by definition it‘s not running, but it’s in the kernel. Since only a running thread can check its signals we must wake or unblock it. When a thread is unblocked or woken, it's given a zx_status_t. Usually the value is ZX_OK or ZX_ERR_TIMED_OUT. However when waking a thread early like this we use a special zx_status_t value, ZX_ERR_INTERNAL_INTR_KILLED in the case of a kill operation and ZX_ERR_INTERNAL_INTR_RETRY in the case of a suspend operation.
When a thread is woken/unblocked, it will see the zx_status_t result and begin backing out of the kernel, unwinding its stack. In general, any kernel function returning one of the two special values will cause its caller to immediately return, propagating that value.
Eventually, when the stack has unwound, the thread will be at the edge, a safe point. It is here, just before returning to user mode, that the thread checks its signals once more and acts on them by calling arch_iframe_process_pending_signals() or x86_syscall_process_pending_signals().
Just like the sleeping/blocked case, the thread must resume execution in order for it to be killed. In the case of kill, the thread will be unblocked with ZX_ERR_INTERNAL_INTR_KILLED and unwind until just before returning to user mode where it acts on the signal.
The target thread could be running user code or kernel code. If it‘s running user code, then we’ll need to force it to enter the kernel where it can check the signals field of its Thread struct. If it‘s running kernel code, then we’ll have to trust that it will check for pending signals in a reasonable time frame.
The sender can't know if the target is in kernel mode or user mode so it behaves the same in either case. The sender sends an Inter-processor Interrupt (IPI) to the CPU on which the target is currently running. Part of the interrupt handlers job is to check for and optionally process pending signals.
If the handler was invoked in a user context, that is, the CPU was in user mode at the time of the interrupt, then it's a safe point to suspend/terminate and the handler will call arch_iframe_process_pending_signals().
If, however, the handler was invoked in a kernel context, then the handler will do nothing because it can‘t know the state of the thread at the point it was interrupted. It’s not safe to suspend/terminate here. Instead, the handler will return to the kernel context from which it was invoked and rely on this outer context to eventually notice the signal and reach a safe point.
You may be wondering if the IPI is really necessary. There are two cases where it's critical. The first is when the target thread is running in user mode and simply not entering the kernel on its own. On a lightly loaded system with no other interrupt traffic, a thread may not enter the kernel for extended periods of time, or ever in the case of an infinite loop. We need the IPI in this case to ensure the target thread observes and processes any pending signals in a timely manner. The second is when the target thread is performing a long running operation in the kernel, but not checking for pending signals. These are rare, but do exist. The best example would be the execution of a guest OS via zx_vcpu_enter(). The interrupt would cause a VMEXIT back to the host kernel where it can check for pending signals and unwind.
Let‘s walk through an example to see how this all works. Imagine thread A is suspending thread B, as B is performing a zx_port_wait(). Depending on exactly when the operation is performed, we can end up in one of several different scenarios. We’ll examine each scenario briefly.
Thread A issues the suspend just before thread B begins its zx_port_wait() syscall. Thread B is still in user mode and is running. Thread A sets thread B‘s THREAD_SIGNAL_SUSPEND bit and issues an IPI to thread B’s current CPU. Thread B‘s CPU takes the interrupt and calls the interrupt handler. Just before returning back to user mode, thread B checks its pending signals. Seeing that THREAD_SIGNAL_SUSPEND is set, it suspends itself. Here’s a sketch of thread B's callstack:
suspend_self() interrupt_handler() ---- interrupt ---- user code
Later on, after being resumed, thread B will return back to user mode as if nothing happened.
Thread A issues the suspend after thread B has entered the kernel to perform a zx_port_wait() syscall. Thread B is executing kernel code and hasn't yet blocked. Just like Scenario 1, thread A issues an IPI, which causes thread B to check for pending signals:
interrupt_handler() ---- interrupt ---- PortDispatcher::Dequeue() sys_port_wait() syscall_dispatch() ---- syscall ---- vdso zx_port_wait() user code
However, this time the interrupt handler sees that it was invoked in kernel context rather than user context so it does not suspend itself. Instead it returns back to the kernel context in which it was invoked. Thread B reaches the core of the zx_port_wait() operation, the point at which it will block if there are no packets available. Thread B sees there are no packets available and prepares to block:
WaitQueue::BlockEtcPreamble() WaitQueue::BlockEtc() PortDispatcher::Dequeue() sys_port_wait() syscall_dispatch() ---- syscall ---- vdso zx_port_wait() user code
Just before blocking, it checks for pending signals and sees that it has been asked to suspend. Instead of blocking it returns ZX_ERR_INTERNAL_INTR_RETRY and the callstack unwinds to the edge, just prior to returning to user mode:
WaitQueue::BlockEtcPreabmle() ZX_ERR_INTERNAL_INTR_RETRY WaitQueue::BlockEtc() | PortDispatcher::Dequeue() | sys_port_wait() | syscall_dispatch() V ---- syscall ---- vdso zx_port_wait() user code
Here the thread checks for pending signals and suspends itself. Upon being resumed, the thread returns to user mode (to the vDSO) with the status result ZX_ERR_INTERNAL_INTR_RETRY. The vDSO has special logic for handling syscalls that return ZX_ERR_INTERNAL_INTR_RETRY, it simply reissues the syscall with the original arguments:
suspend_self() ZX_ERR_INTERNAL_INTR_RETRY syscall_dispatch() | ---- syscall ---- | A vdso |______| zx_port_wait() user code
Thread A issues the suspend after thread B has entered the kernel and blocked, waiting for a port packet. Thread A sees that thread B is blocked so it unblocks thread B with the value ZX_ERR_INTERNAL_INTR_RETRY. From this point on the behavior matches that of Scenario 2. The call returns to user mode where it is retried by the vDSO:
blocked ZX_ERR_INTERNAL_INTR_RETRY WaitQueue::BlockEtcPostamble() | WaitQueue::BlockEtc() | PortDispatcher::Dequeue() | sys_port_wait() | syscall_dispatch() | ---- syscall ---- | A vdso |______| zx_port_wait() user code
While thread B was blocked, waiting on a port packet, a packet arrived, unblocking it (with ZX_OK):
blocked ZX_OK WaitQueue::BlockEtcPostamble() | WaitQueue::BlockEtc() | PortDispatcher::Dequeue() V sys_port_wait() syscall_dispatch() ---- syscall ---- vdso zx_port_wait() user code
Thread B is now unwinding toward user mode when thread A issues a suspend. Thread A sets the bit, see that thread B is marked as running so it sends an IPI. Similar to the “Suspend just before syscall” case, the interrupt handler executes:
interrupt_handler() ---- interrupt ---- PortDispatcher::Dequeue() sys_port_wait() syscall_dispatch() ---- syscall ---- vdso zx_port_wait() user code
However, this time it does not check for pending signals because the handler interrupted kernel context rather than user context. The handler completes and thread B continues to unwind. Eventually, thread B reaches the edge and is about to return from the syscall to user mode. Here it checks for pending signals, sees THREAD_SIGNAL_SUSPEND and suspends itself:
suspend_self() syscall_dispatch() ---- syscall ---- vdso zx_port_wait() user code
Upon being resumed, it will return to user mode with the status result that unblocked it (ZX_OK):
syscall_dispatch() ZX_OK ---- syscall ---- | vdso V zx_port_wait() user code
The key points to take away are:
You can't forcibly suspend or kill a thread. You can only ask it to suspend or terminate itself.
Thread signals are the mechanism for asking a thread to suspend or terminate.
Threads should only suspend or terminate their execution at specific points within the kernel. In particular, a thread may only suspend or terminate execution when it holds no resources (e.g. locks) and is about to return from kernel mode to user mode.
In order to remain responsive, long running kernel operations must periodically check for pending signals and return if any are set.
