| --- | |
| name: fuchsia-debugger | |
| description: Use fx debug cli to debug test failures, processes, and live device drivers. | |
| --- | |
| # Overview | |
| `fx debug cli` is a machine-readable, agent-friendly interface to `zxdb`. It | |
| uses the Debug Adapter Protocol (DAP) to enable turn-based interactions that | |
| agents are already accustomed to and possess knowledge of. Unlike traditional | |
| console-based debuggers, `fx debug cli` exposes a stateless command-line | |
| frontend that issues request-response style commands to the backend, allowing | |
| agents to send and receive structured JSON as input and output. | |
| `fx debug cli` integrates with `fx test` to allow automatic pausing during test | |
| failures, which allows Agents to inspect the state and diagnose the failure. | |
| --- | |
| ## Testing Workflow | |
| This mode delegates the management of the debugging session to the testing | |
| framework. The framework automatically starts the `zxdb-daemon` process and | |
| connects to the active target. | |
| ### Step 1: Start the Test in the Background | |
| Run `fx test` with the `--agent-debugging-mode` flag. Execute this command | |
| asynchronously in the background so you can proceed with other work while | |
| observing the output: | |
| ```bash | |
| fx test <test_suite_name> --agent-debugging-mode | |
| ``` | |
| *Note: This automatically starts the `zxdb-daemon` background process, which | |
| begins listening for messages sent via `fx debug cli`.* | |
| ### Step 2: Poll for Target Transitions | |
| Begin polling the active daemon process for events. Start with `last_seen_seq: | |
| 0` and increment the sequence number as events are received: | |
| ```bash | |
| fx debug cli --json '{"command": "wait-for-event", "last_seen_seq": 0, "timeout": 10}' | |
| ``` | |
| > [!NOTE] | |
| > **Socket Startup Delay**: The background daemon may take 1-2 seconds to | |
| > initialize and bind the listening socket after `fx test` begins. If the | |
| > initial command fails with `Daemon socket not found`, wait briefly and retry. | |
| #### Example `wait-for-event` Response (Test Failure / Breakpoint): | |
| ```json | |
| { | |
| "success": true, | |
| "events": [ | |
| { | |
| "seq": 1, | |
| "event": "stopped", | |
| "body": { | |
| "threadId": 1, | |
| "reason": "breakpoint" | |
| } | |
| } | |
| ] | |
| } | |
| ``` | |
| > [!WARNING] | |
| > **Thread ID Key Casing Mapping Pitfall**: | |
| > Output events returned by `wait-for-event` use **camelCase** keys (e.g. | |
| > `"threadId": 1`). However, subsequent request commands (such as | |
| > `stack-trace`, `continue`, or `pause`) strictly require **snake_case** | |
| > parameters (e.g. `"thread_id": 1`). | |
| > Always map `"threadId"` to `"thread_id"` when programmatically invoking | |
| > requests. | |
| Simultaneously, monitor the output and status of the background `fx test` task. | |
| ### Step 3: Handle Execution Scenarios | |
| Depending on the status of the test run, handle one of the following scenarios: | |
| #### Scenario A: Test Failure (Debugger Suspended) | |
| If `wait-for-event` returns a `"stopped"` event, it indicates a test failure or | |
| breakpoint hit. Perform diagnostics: | |
| 1. **Query Session State:** Retrieve all threads and processes active in the | |
| session to identify the failing thread: | |
| ```bash | |
| fx debug cli --json '{"command": "get-state"}' | |
| ``` | |
| Example Response: | |
| ```json | |
| { | |
| "success": true, | |
| "body": { | |
| "threads": [ | |
| { "id": 1, "name": "initial-thread" } | |
| ], | |
| "processes": { "12345": "my_test_binary" } | |
| } | |
| } | |
| ``` | |
| *Note: In the `get-state` thread list representation, the key is `"id"`. Map | |
| this value to the `"thread_id"` parameter in subsequent requests.* | |
| 2. **Retrieve Stack Trace:** Get the stack trace for a suspended thread | |
| (`thread_id`) or across all threads in a process (`pid`): | |
| ```bash | |
| # Thread-level stack trace | |
| fx debug cli --json '{"command": "stack-trace", "thread_id": 1}' | |
| # Process-level stack trace (all threads in "pid") | |
| fx debug cli --json '{"command": "stack-trace", "pid": 12345}' | |
| ``` | |
| Example Thread Response: | |
| ```json | |
| { | |
| "success": true, | |
| "body": { | |
| "thread_id": 1, | |
| "stack_frames": [ | |
| { | |
| "frame_index": 0, | |
| "name": "my_test_function", | |
| "source": { | |
| "name": "main_test.cc", | |
| "path": "/src/main_test.cc" | |
| }, | |
| "line": 42, | |
| "column": 1 | |
| } | |
| ], | |
| "total_frames": 1 | |
| } | |
| } | |
| ``` | |
| Example Process Response: | |
| ```json | |
| { | |
| "success": true, | |
| "body": { | |
| "process_id": 12345, | |
| "stacks": [ | |
| { | |
| "thread_id": 1, | |
| "stack_frames": [ | |
| { | |
| "frame_index": 0, | |
| "name": "my_test_function", | |
| "source": { | |
| "name": "main_test.cc", | |
| "path": "/src/main_test.cc" | |
| }, | |
| "line": 42, | |
| "column": 1 | |
| } | |
| ], | |
| "total_frames": 1 | |
| }, | |
| { | |
| "thread_id": 2, | |
| "stack_frames": [ | |
| { | |
| "frame_index": 0, | |
| "name": "worker_thread_entry", | |
| "source": { | |
| "name": "worker.cc", | |
| "path": "/src/worker.cc" | |
| }, | |
| "line": 105, | |
| "column": 5 | |
| } | |
| ], | |
| "total_frames": 1 | |
| } | |
| ] | |
| } | |
| } | |
| ``` | |
| 3. **Inspect Local Variables:** Retrieve variables and arguments within a | |
| specific stack frame: | |
| ```bash | |
| fx debug cli --json '{"command": "variables", "thread_id": 1, "frame_index": 0}' | |
| ``` | |
| 4. **Evaluate Dynamic Expressions:** Evaluate expressions directly within a | |
| stack frame via `evaluate`. | |
| Note that `evaluate` requires the target thread to be stopped prior to | |
| invocation. Ensure the thread is suspended via a breakpoint or `pause` | |
| command before evaluating. | |
| Pass the expression directly (do not prefix with `print`): | |
| ```bash | |
| fx debug cli --json '{"command": "evaluate", "thread_id": 1, "frame_index": 0, "expression": "my_var"}' | |
| ``` | |
| Example Response: | |
| ```json | |
| { | |
| "success": true, | |
| "body": { | |
| "result": "42", | |
| "type": "int" | |
| } | |
| } | |
| ``` | |
| To execute raw `zxdb` console commands that do not have a dedicated CLI | |
| subcommand (such as symbol inspection or disassembly), use the `console` | |
| command (optionally scoped to a `thread_id` and `frame_index`): | |
| ```bash | |
| fx debug cli --json '{"command": "console", "thread_id": 1, "frame_index": 0, "input": "sym-info my_var"}' | |
| ``` | |
| 5. **Teardown Session:** Once diagnostics are complete, terminate the debugging | |
| session. This detaches from targets and lets the background test runner exit | |
| cleanly: | |
| ```bash | |
| fx debug cli --json '{"command": "stop"}' | |
| ``` | |
| #### Scenario B: Clean Pass (All Tests Succeed) | |
| If all tests pass cleanly, the background `fx test` task will complete | |
| successfully, exiting with status code `0` and summarizing `FAILED: 0` in its | |
| output. | |
| * **Edge Case: Poll Connection Failures**: Since `fx test` automatically reaps | |
| the background daemon process on a clean exit, any active, blocking | |
| `wait-for-event` command will terminate abruptly with a connection error | |
| (e.g., connection refused or closed socket). | |
| * **Action**: If `wait-for-event` returns a connection failure, check the status | |
| of the background `fx test` task. If it completed successfully (exit code 0), | |
| treat this as a successful clean pass, print a success message, and exit. **No | |
| manual `"stop"` command or teardown is required.** | |
| #### Scenario C: Proactive Breakpoint Installation (Non-Failing Paths) | |
| By default, the testing framework attaches to targets weakly (`attach --weak`). | |
| In this state, symbols are not loaded upfront, and dynamic `break` requests will | |
| remain pending and unresolved until a crash occurs. | |
| To debug a passing path or install breakpoints before test execution begins: | |
| 1. Start the test with the `--breakpoint` option to force a normal attach with | |
| immediate symbol loading (paths must be fully qualified from the workspace | |
| root): | |
| ```bash | |
| fx test <test_name> --agent-debugging-mode --breakpoint <source_file>:<line> | |
| ``` | |
| 2. Poll for the breakpoint hit stopped event via `wait-for-event`. | |
| 3. Once suspended, perform diagnostics and resume execution using `continue`. | |
| --- | |
| ## JSON Request Reference | |
| All commands are sent as serialized JSON payloads to `fx debug cli --json | |
| '<payload>'`. | |
| | Action | Command Input Payload | | |
| |---|---| | |
| | **Wait for Event** | `{"command": "wait-for-event", "last_seen_seq": <seq>, "timeout": <secs>}` | | |
| | **Get State** | `{"command": "get-state"}` *(Returns active threads, processes, breakpoints, and tracepoints)* | | |
| | **List Threads** | `{"command": "threads"}` | | |
| | **Set/Delete Breakpoint** | `{"command": "break", "file": "<workspace_root_path>", "line": <line_num>, "condition": "<optional_expr>", "delete": <optional_bool>}` or `{"command": "break", "function": "<symbol>", "condition": "<optional_expr>", "delete": <optional_bool>}` *(CLI: `break <file>:<line>` or `break --file <file> --line <line>` or `break -f <symbol> [-c <expr>]`)* | | |
| | **Set/Manage Tracepoint** | `{"command": "tracepoint", "file": "<workspace_root_path>", "line": <line_num>, ...}` or `{"command": "tracepoint", "function": "<symbol>", ...}` or `{"command": "tracepoint", "watch": "<expr_or_hex_addr>", "thread_id": <optional_tid>, "frame_index": <optional_idx>, "size": <optional_bytes>, "access_type": "write" \| "readWrite", ...}` with optional `"condition": "<expr>"`, `"expressions": ["<expr>"]`, `"capture": ["frame", "backtrace", "locals", "registers"]`, `"raw_backtrace": <bool>`, `"max_matching_frames": <int>`, `"max_hits": <int>`. For management, pass `{"command": "tracepoint", "list_all": true}`, `{"command": "tracepoint", "clear": true}`, or `{"command": "tracepoint", "tracepoint_id": <id>, "enabled": <bool> \| "delete": true}` *(CLI: `tracepoint (<file>:<line> \| -f <sym> \| -w <expr> [-t <id>]) [-c <expr>] [-e <expr>] [--locals] [--registers] [--backtrace] [-r] [--max-frames <n>] [--max-hits <n>]` or `tracepoint (--list \| --clear \| --enable <id> \| --disable <id> \| <id> -d)`. Aliases: `tp`, `trace`)* | | |
| | **Attach Process / Driver** | `{"command": "attach", "filter": "<name_or_pid>", "weak": <optional_bool>, "exact": <optional_bool>, "recursive": <optional_bool>, "job_only": <optional_bool>}` or `{"command": "attach", "driver": "<driver_name_or_url>", "weak": <optional_bool>, "wait": <optional_bool>}` or `{"command": "attach", "node": "<moniker>", "weak": <optional_bool>, "wait": <optional_bool>}` *(CLI: `attach (<filter> | --driver <drv> | --node <moniker>)`)* | | |
| | **List Driver Hosts** | `{"command": "driver-list", "filter": "<optional_substr>"}` *(CLI: `driver-list [<filter>]` or `driver-list -f <filter>`)* | | |
| | **List Modules** | `{"command": "modules", "pid": <optional_pid>, "filter": "<optional_substr>", "force_reload": <optional_bool>}` *(CLI: `modules [-p <pid>] [-f <filter>] [-r]`)* | | |
| | **Inspect Address Space** | `{"command": "aspace", "pid": <optional_pid>, "address": "<optional_addr>", "filter": "<optional_substr>"}` *(CLI: `aspace [addr] [-p <pid>] [-f <filter>]`)* | | |
| | **Inspect Handles** | `{"command": "handles", "pid": <optional_pid>, "type": "<optional_type>", "koid": <optional_koid>, "detail": <optional_bool>}` *(CLI: `handles [-p <pid>] [--type <type>] [-k <koid>] [--detail]`)* | | |
| | **Read Memory** | `{"command": "mem-read", "address": "<addr_or_expr>", "size": <bytes>, "pid": <optional_pid>, "thread_id": <optional_thread_id>, "frame_index": <frame_index>}` *(CLI: `mem-read <addr_or_expr> [-s <size>] [-p <pid>] [-t <id> [--frame-index <idx>]]`)* | | |
| | **Async Backtrace** | `{"command": "async-backtrace", "pid": <optional_pid>}` *(CLI: `async-backtrace [-p <pid>]`)* | | |
| | **Detach Process** | `{"command": "detach", "pid": <pid>}` or `{"command": "detach", "all": true}` *(CLI: `detach [<pid>] [--all]`)* | | |
| | **Get Stack Trace** | `{"command": "stack-trace", "thread_id": <thread_id>}` or `{"command": "stack-trace", "pid": <pid>, "raw": <optional_bool>}`; optional `"force_remote_unwind": <bool>`, `"unwinder": "cfi" \| "fp" \| "scs" \| "plt" \| "armehabi" \| "sigreturn"` *(CLI: `stack-trace (-t <id> \| -p <pid>) [-r] [--force-remote-unwind] [--unwinder <name>]`)* | | |
| | **List Variables** | `{"command": "variables", "thread_id": <thread_id>, "frame_index": <frame_index>}` *(CLI: `variables -t <id> --frame-index <idx>`)* | | |
| | **Evaluate Expression** | `{"command": "evaluate", "thread_id": <thread_id>, "frame_index": <frame_index>, "expression": "<expr>"}` *(CLI: `evaluate -t <id> [--frame-index <idx>] <expr>`)* | | |
| | **Run Console Command** | `{"command": "console", "input": "<cmd>", "thread_id": <optional_thread_id>, "frame_index": <frame_index>}` *(CLI: `console [-t <id>] [--frame-index <idx>] <cmd>`)* | | |
| | **Step In** | `{"command": "step-in", "thread_id": <thread_id>}` *(CLI: `step-in <id>`)* | | |
| | **Step Over (Next)** | `{"command": "next", "thread_id": <thread_id>}` *(CLI: `next <id>`)* | | |
| | **Step Out (Finish)** | `{"command": "finish", "thread_id": <thread_id>}` *(CLI: `finish <id>`)* | | |
| | **Continue Thread** | `{"command": "continue", "thread_id": <thread_id>}` *(CLI: `continue <id>`)* | | |
| | **Pause Thread / Process** | `{"command": "pause", "thread_id": <thread_id>}` or `{"command": "pause", "pid": <pid>}` *(CLI: `pause -t <id>` or `pause -p <pid>`)* | | |
| | **Stop Session** | `{"command": "stop"}` | | |
| ### Global Parameters | |
| * **Event History Pruning (`ack_seq`)**: Any command payload can include | |
| `"ack_seq": <seq>` (e.g., `{"command": "get-state", "ack_seq": 5}`). The | |
| daemon will prune all event history up to the acknowledged sequence to | |
| optimize memory. | |
| ### Performance & Blocking | |
| * **Smart Blocking**: Commands like `pause`, `stack-trace`, and `wait-for-event` | |
| are blocking operations and may take up to 10 seconds depending on the | |
| target's execution state. Unlike those commands, `evaluate` is non-blocking | |
| and fails immediately if the target thread is not already stopped. | |
| --- | |
| ## Live Process & Driver Debugging (`fx debug cli`) | |
| `fx debug cli` also provides workflows for live process inspection and DFv2 | |
| driver bring-up on running targets. | |
| ### 1. Discover & Safely Attach to a Driver Host | |
| Because pausing a `driver_host` freezes all colocated drivers in that process | |
| (which can sever USB/CDC-NCM/ADB transport if transport or watchdog drivers are | |
| colocated), inspect driver host topology with `driver-list` before attaching: | |
| ```bash | |
| # List running driver_hosts, colocated drivers, and transport hazards | |
| fx debug cli driver-list --filter xhci | |
| # Attach by driver name/URL or device node moniker (warns on transport hazards) | |
| fx debug cli attach --driver xhci | |
| fx debug cli attach --node dev.sys.platform.xhci | |
| ``` | |
| #### Driver Host Transport Hazards & Guidance | |
| When a `driver_host` colocates transport or watchdog drivers (such as USB, | |
| CDC-NCM, ethernet, serial, or watchdog), avoid stopping breakpoints or manual | |
| pause commands to prevent severing host `ffx` communication. Instead: | |
| * Prefer weak attach (`--weak`) to catch crashes without stopping threads. | |
| * Use non-stopping inspection commands (`modules`, `aspace`, `handles`, | |
| `mem-read`) to examine process state safely. | |
| * Use non-stopping tracepoints (`tracepoint`) instead of breakpoints to sample | |
| variables or expressions without pausing threads. | |
| ```bash | |
| # Attach weakly for crash-only catch on transport-sensitive driver hosts | |
| fx debug cli attach --driver xhci --weak | |
| ``` | |
| #### Pending or Unbound Drivers (`--wait`) | |
| When attaching to a driver that has not yet started or bound, pass `--wait`. If | |
| no running `driver_host` matches, `fx debug cli` installs a zxdb process filter | |
| on `"driver_host"` that attaches to every newly started driver host (since zxdb | |
| matches process names rather than driver URLs or device monikers). | |
| Attach flags (`--weak`, `--recursive`, `--job-only`) are forwarded to this | |
| filter, and the response includes `requested_target` and `filter_warning` | |
| alerting the user that all new driver hosts will be attached. | |
| ```bash | |
| # Pre-arm a filter before a driver binds or starts | |
| fx debug cli attach --driver my_driver --wait | |
| ``` | |
| ### 2. Inspect a Live Process | |
| These commands return structured JSON and do not stop the target. When more than | |
| one process is attached, pass `--pid` (`-p`) to choose one. | |
| ```bash | |
| # Inspect loaded modules/DSOs, virtual memory mappings, and Zircon handles | |
| fx debug cli modules --filter xhci | |
| fx debug cli aspace --filter mmio | |
| fx debug cli handles --type interrupt --detail | |
| # Read memory at an address (or at a frame-relative expression with -t) | |
| fx debug cli mem-read 0x7ffee1234000 -s 32 | |
| ``` | |
| `modules` requests the module list from `debug_agent`, which ends a weak attach. | |
| ### 3. Breakpoints and Stack Traces | |
| ```bash | |
| # Set function or conditional breakpoints | |
| fx debug cli break --function usb_xhci::UsbXhci::Init --condition "status != 0" | |
| # If an unwind stops early, retry on the target or force a specific unwinder | |
| fx debug cli stack-trace -p <pid> --force-remote-unwind | |
| fx debug cli stack-trace -t <tid> --unwinder fp | |
| ``` | |
| ### 4. Tracepoints (`tracepoint`) | |
| A tracepoint (`tracepoint`, `tp`, or `trace`) captures state when a thread hits | |
| a code location or hardware watchpoint, then resumes the thread immediately. The | |
| backend emits a `zxdb.Tracepoint` event to `wait-for-event` and disables the | |
| tracepoint after `max_hits` events (20 by default). | |
| ```bash | |
| # Capture an expression and local variables on a function (aliases: tp, trace) | |
| fx debug cli tracepoint usb_xhci::UsbXhci::HandleEvent -e status --locals --max-hits 10 | |
| # Capture a backtrace on a source line when a condition holds | |
| fx debug cli tp src/devices/usb/drivers/xhci/usb-xhci.cc:142 -c "ring != nullptr" --backtrace | |
| # Set a watchpoint tracepoint on an expression (requires -t) or hex address | |
| fx debug cli tp -w my_var -t 1 -a readWrite --size 4 | |
| # List, disable, re-enable (resets hit_count), delete, or clear tracepoints | |
| fx debug cli tp --list | |
| fx debug cli tp --disable 1 | |
| fx debug cli tp --enable 1 | |
| fx debug cli tp 1 -d | |
| fx debug cli tp --clear | |
| ``` | |
| Poll `wait-for-event` to read `zxdb.Tracepoint` hit events: | |
| ```json | |
| { | |
| "success": true, | |
| "events": [ | |
| { | |
| "seq": 2, | |
| "event": "zxdb.Tracepoint", | |
| "body": { | |
| "tracepointId": 1, | |
| "location": "usb_xhci::UsbXhci::HandleEvent", | |
| "hitCount": 1, | |
| "maxHits": 10, | |
| "autoDisabled": false, | |
| "truncated": false, | |
| "threadKoid": 34567, | |
| "dwellMs": 1.4, | |
| "capture": { | |
| "matched": true, | |
| "frame": { | |
| "name": "usb_xhci::UsbXhci::HandleEvent()", | |
| "file": "src/devices/usb/drivers/xhci/usb-xhci.cc", | |
| "line": 98, | |
| "ip": 4398046515200, | |
| "sp": 140732920758272, | |
| "trust": "CFI" | |
| }, | |
| "locals": { | |
| "ring": { "value": "0x20004000", "type": "usb_xhci::EventRing*" } | |
| }, | |
| "evaluations": { | |
| "status": { "result": "0", "type": "zx_status_t", "error": null } | |
| }, | |
| "errors": [] | |
| } | |
| } | |
| } | |
| ] | |
| } | |
| ``` |