blob: d1a861fe7feda7355391151491959ca4e86a5885 [file] [view]
# trace_manager object lifecycle
This document describes how `trace_manager` tracks trace providers across
trace sessions: when providers are registered, when they join and leave a
session, and what happens to them when a session times out or ends. For the
overall design of Fuchsia tracing, see
[Fuchsia tracing system](/docs/concepts/kernel/tracing-system.md).
## Objects at a glance
| Object | Defined in | Lifetime | Represents |
| --- | --- | --- | --- |
| `TraceManager` | `trace_manager.h` | The `trace_manager` process | Serves `fuchsia.tracing.provider/Registry` and `fuchsia.tracing.controller/Provisioner`. Owns the provider registrations and at most one `TraceController`. |
| `TraceProviderBundle` (V1) | `trace_provider_bundle.h` | Provider registration to channel close | A registered V1 provider and its `fuchsia.tracing.provider/Provider` client. |
| `ProviderConnection` (V2) | `provider_connection.h` | Provider registration to channel close | A registered V2 provider and its `fuchsia.tracing.provider/ProviderV2` client. Also routes the provider's `OnSaveBuffer` and `OnAlert` events. |
| `TraceController` | `trace_manager.h` | `InitializeTracing` to session close | Serves one `fuchsia.tracing.controller/Session` connection and owns that session's `TraceSession`. |
| `TraceSession` | `trace_session.h` | Same as its `TraceController` | One trace session: its configuration, its state machine, the output socket (through a `BufferForwarder`), and one tracee per participating provider. |
| `Tracee` (V1) | `tracee.h` | Joins a session to leaves it | One V1 provider's participation in one session: its trace buffer and FIFO. |
| `TraceeV2` (V2) | `traceev2.h` | Joins a session to leaves it | One V2 provider's participation in one session: its trace buffer and pending FIDL calls. |
Registrations are long-lived: they belong to `TraceManager` and last as long
as the provider's channel. Tracees are short-lived: they belong to a single
`TraceSession` and never move to another session.
```mermaid
flowchart TD
TM["TraceManager"] -->|"owns providers_"| REG["TraceProviderBundle / ProviderConnection"]
TM -->|"owns at most one"| TC["TraceController"]
TC -->|"owns"| TS["TraceSession"]
TS -->|"owns tracees_"| T["Tracee / TraceeV2"]
T -.->|"borrows a pointer to"| REG
```
A tracee holds a raw pointer to its registration. When a registration goes
away, its session first removes the matching tracee (see
[Provider disconnect](#provider-disconnect)).
## Provider registration
A provider process connects to `fuchsia.tracing.provider/Registry` and calls
one of:
* V1: `RegisterProvider` or `RegisterProviderSynchronously`, handled by
`TraceManager::RegisterProviderWorker`.
* V2: `RegisterV2` or `RegisterV2Synchronously`, handled by
`TraceManager::RegisterProviderV2Worker`.
Each call appends a `TraceProviderBundle` or `ProviderConnection` to
`TraceManager::providers_` and assigns it a new provider id. If a session
exists, the provider is also added to it right away (see
[Joining a session](#joining-a-session)). The `*Synchronously` variants also
report whether that session has already started.
Providers built on `TraceProviderImpl` register once, when the provider is
created, usually at process startup. They do not register again. As a result,
**the registration is the only lasting link between a provider and
`trace_manager`**: anything that drops a registration removes that provider
from tracing until its process creates a new provider.
`GetProviders` and `GetKnownCategories` on `Provisioner` iterate over
`providers_`, so they see every registered provider whether or not a session
exists.
### Provider disconnect
A registration is removed only when its channel unbinds, for example because
the provider process exited or closed the channel. The unbind handler set in
`RegisterProviderWorker` or `RegisterProviderV2Worker`:
1. Calls `TraceSession::RemoveDeadProvider` or `RemoveDeadV2Provider` on the
current session, if there is one. This runs the same path as a provider
finishing termination: it writes the provider's results if they haven't
been written yet and the session writes results on terminate, then removes
the tracee from `tracees_`.
2. Erases the registration from `providers_`.
Nothing else in `trace_manager` erases a registration. In particular,
timeouts and errors within a session affect only the tracee, not the
registration.
## Sessions
A client calls `Provisioner.InitializeTracing` with a `TraceConfig`, a
`Session` server end, and an output socket. `TraceManager::InitializeTracing`:
1. Rejects the request if a `TraceController` already exists. Only one
session can exist at a time.
2. Creates a `BufferForwarder` wrapping the output socket, or a
`DeferredBufferForwarder` when the config sets `defer_transfer` (ignored on
the `astro` board).
3. Creates a `TraceSession` in state `kInitialized` and writes the trace info
records to the socket.
4. Calls `TraceSession::AddProvider` for **every** registration in
`providers_`.
5. Wraps the session in a new `TraceController` bound to the `Session`
channel.
### Session states
```mermaid
stateDiagram-v2
[*] --> kInitialized: InitializeTracing
kInitialized --> kStarting: StartTracing
kStopped --> kStarting: StartTracing
kStarting --> kStarted: all tracees started, or start timeout
kInitialized --> kStopping: StopTracing
kStarting --> kStopping: StopTracing
kStarted --> kStopping: StopTracing
kStopping --> kStopped: all tracees stopped, or stop timeout
kInitialized --> kTerminating: controller disconnect
kStarting --> kTerminating: controller disconnect
kStarted --> kTerminating: controller disconnect
kStopping --> kTerminating: controller disconnect
kStopped --> kTerminating: controller disconnect
kTerminating --> [*]: all tracees terminated, or terminate timeout
```
There is no `kTerminated` state. A terminated session is destroyed, and
`TraceManager` then has no session. A session can also end at any time
through `TraceSession::Abort()`, for example when writing to the output socket
fails. Abort destroys the `TraceController` and its session immediately.
A session can go through several start/stop cycles. It ends when the
controller closes its `Session` channel. `TraceManager::OnControllerDisconnect`
calls `TraceController::TerminateTracing`, which calls
`TraceSession::Terminate`. When termination finishes or times out,
`TraceManager::CloseSession` destroys the `TraceController`, its
`TraceSession`, and every remaining tracee. A new `InitializeTracing` call is
accepted only after that.
### Timeouts
| Timeout | Default | Armed by | When it fires |
| --- | --- | --- | --- |
| Start | `TraceConfig.start_timeout_milliseconds`, 5 s if unset | `TraceSession::Start` | `SessionStartTimeout` logs the tracees that haven't started and moves the session to `kStarted`. Those tracees stay in the session. |
| Stop | 60 s (`kStopTimeout`) | `TraceSession::Stop` | `SessionStopTimeout` evicts every tracee that hasn't stopped (see [Leaving a session](#leaving-a-session)), flushes the output, and moves the session to `kStopped`. |
| Terminate | 60 s (`kStopTimeout`) | `TraceSession::Terminate` | `SessionTerminateTimeout` logs the tracees that haven't terminated and runs the terminate callback, which destroys the session and all remaining tracees. |
## Tracees
### Joining a session
`TraceSession::AddProvider` creates a tracee for a registration and calls its
`Initialize`. Initialization allocates the trace buffer VMO (sized by the
session's buffer size, or by the provider's entry in
`TraceConfig.provider_specs`) and sends the provider `Initialize`:
* V1 (`Tracee::Initialize`): also creates a FIFO and starts waiting on it.
The provider reports `STARTED`, `STOPPED`, `SAVE_BUFFER`, and `ALERT` as
[FIFO packets](FIFO-packets.md).
* V2 (`TraceeV2::Initialize`): there is no FIFO. Start, stop, and terminate
are acknowledged by FIDL replies, and buffer-save and alert requests arrive
as `OnSaveBuffer` and `OnAlert` events on the `ProviderConnection`. After a
successful `Initialize`, `AddProvider` registers callbacks on the
`ProviderConnection` that forward those events to this tracee and session.
Each callback checks that the tracee still exists, so events from a provider
that has left the session are dropped.
If `Initialize` fails, the tracee is discarded and the provider does not take
part in this session. Its registration is unaffected.
A tracee joins a session only through `AddProvider`, which is called in two
places:
1. **At `InitializeTracing`**, for every registered provider.
2. **At registration**, for a provider that registers while a session exists.
If the session is already starting or started, the new tracee is started
immediately with a cleared buffer. If the session is stopping or stopped,
it is marked stopped so the session doesn't wait for it.
`AddProvider` ignores new providers while the session is `kTerminating`.
### Tracee states
`Tracee` and `TraceeV2` share `Tracee::State`:
```mermaid
stateDiagram-v2
[*] --> kReady
kReady --> kInitialized: Initialize
kInitialized --> kStarting: Start
kStopped --> kStarting: Start
kStarting --> kStarted: provider acknowledges start
kInitialized --> kStopped: Stop
kStarting --> kStopping: Stop
kStarted --> kStopping: Stop
kStopping --> kStopped: provider acknowledges stop
kInitialized --> kTerminating: Terminate
kStarting --> kTerminating: Terminate
kStarted --> kTerminating: Terminate
kStopping --> kTerminating: Terminate
kStopped --> kTerminating: Terminate
kTerminating --> kTerminated: provider acknowledges terminate
```
A V1 provider acknowledges start and stop with FIFO packets, and acknowledges
terminate by closing its end of the FIFO. A V2 provider acknowledges each
request by replying to the FIDL call. `Stop` on a tracee that was never
started marks it stopped without contacting the provider.
`Tracee::Start` and `TraceeV2::Start` require the tracee to be in
`kInitialized` or `kStopped`. Restarting a stopped session therefore must not
find a tracee still in `kStopping`, which is one reason the stop timeout
evicts such tracees instead of leaving them in the session.
### Writing trace data
Trace data reaches the output socket in these cases:
* When a tracee stops and the `StopTracing` request set `write_results`, the
session writes that provider's buffer to the socket.
* When a tracee terminates, the session writes its buffer if the tracee was
started, its results weren't already written, and the session writes
results on terminate.
* In streaming mode, the session writes a buffer half each time the provider
sends a buffer-save request while the tracee is starting, started,
stopping, or terminating. If writing fails, the session terminates that
tracee.
With a `DeferredBufferForwarder`, these writes go to a temporary file, which
is copied to the socket when all tracees have stopped (or the stop timeout
fires) and when the forwarder is destroyed.
### Leaving a session
A tracee leaves its session, and is destroyed, in exactly these cases:
| Trigger | Where | Registration |
| --- | --- | --- |
| `Initialize` fails | `TraceSession::AddProvider` | Kept |
| The provider acknowledges terminate, during session termination or after a failed buffer save | `OnProviderTerminated`, `OnV2ProviderTerminated` | Kept |
| The provider's channel unbinds | `RemoveDeadProvider`, `RemoveDeadV2Provider` | Removed |
| The tracee hasn't stopped when the stop timeout fires (eviction) | `TraceSession::SessionStopTimeout` | Kept |
| The session is destroyed | `~TraceSession` | Kept |
#### Eviction on stop timeout
When the stop timeout fires, `SessionStopTimeout` treats each tracee that
hasn't reached `kStopped` as unresponsive. For each one it:
1. Logs a warning naming the provider.
2. Sends the provider a best-effort `Terminate`, so that the provider stops
writing to a buffer that `trace_manager` no longer reads. Nothing waits for
the reply.
3. Removes the tracee from `tracees_`.
The session then flushes the output, moves to `kStopped`, and replies to
`StopTracing` with stats for the remaining providers only.
Eviction has these consequences:
* **The evicted provider sits out the rest of the session.** No code path adds
a provider to an existing session except a new registration, so a later
`StartTracing` on the same `kStopped` session starts only the remaining
tracees.
* **Late events from the evicted provider are ignored.** The session no longer
has a tracee to deliver V1 FIFO packets to, and V2 `OnSaveBuffer` and
`OnAlert` callbacks drop events once the tracee is gone. Late V2 replies
have no effect on the session (see [Pending V2 calls](#pending-v2-calls)).
* **Session termination doesn't wait for the evicted provider.** Only the
remaining tracees need to acknowledge `Terminate`.
* **The next session includes the evicted provider again.** Its registration
is still in `providers_`, so the next `InitializeTracing` calls
`AddProvider` for it.
What the provider does in that next session depends on whether it has
recovered. `TraceProviderImpl` releases its engine session as soon as it
handles `Terminate`, so it accepts the next `Initialize`. If its trace engine
is still stopping from the earlier session, however,
`Session::InitializeEngine` fails. The provider then stays connected, but it
has no engine session: it replies to `Start`, `Stop`, and `Terminate`
immediately and contributes no trace data until its engine finishes stopping.
### Pending V2 calls
`TraceeV2` sends `Start`, `Stop`, and `Terminate` as two-way FIDL calls and
handles each reply with `.Then(...)`. That ties the reply handler to the
lifetime of the `ProviderConnection`'s FIDL client:
* If the client is destroyed (the registration was removed), pending reply
handlers are discarded without being called.
* If the tracee is destroyed first, for example by eviction, the reply
handlers still run when the replies arrive. The `Start` and `Stop` handlers
do nothing once the tracee is gone. The `Terminate` handler still calls the
callback it was given, which reports that the provider acknowledged
`Terminate`. Callers must not assume the tracee still exists at that point.
For an evicted tracee that callback does nothing.
## Source references
* [`trace_manager.cc`](/src/performance/trace_manager/trace_manager.cc):
`TraceManager` and `TraceController`.
* [`trace_session.cc`](/src/performance/trace_manager/trace_session.cc):
`TraceSession`.
* [`tracee.cc`](/src/performance/trace_manager/tracee.cc),
[`traceev2.cc`](/src/performance/trace_manager/traceev2.cc): tracees.
* [`provider_connection.cc`](/src/performance/trace_manager/provider_connection.cc):
V2 registrations and event routing.
* [`//zircon/system/ulib/trace-provider/provider_impl.cc`](/zircon/system/ulib/trace-provider/provider_impl.cc):
the provider-side `TraceProviderImpl`.