name: fuchsia-debugger description: Use fx debug cli to debug test failures, processes, and live device drivers.

Overview

fx debug cli is a machine-readable, agent-friendly interface to zxdb. It uses the Debug Adapter Protocol (DAP) to enable turn-based interactions that agents are already accustomed to and possess knowledge of. Unlike traditional console-based debuggers, fx debug cli exposes a stateless command-line frontend that issues request-response style commands to the backend, allowing agents to send and receive structured JSON as input and output.

fx debug cli integrates with fx test to allow automatic pausing during test failures, which allows Agents to inspect the state and diagnose the failure.


Testing Workflow

This mode delegates the management of the debugging session to the testing framework. The framework automatically starts the zxdb-daemon process and connects to the active target.

Step 1: Start the Test in the Background

Run fx test with the --agent-debugging-mode flag. Execute this command asynchronously in the background so you can proceed with other work while observing the output:

fx test <test_suite_name> --agent-debugging-mode

Note: This automatically starts the zxdb-daemon background process, which begins listening for messages sent via fx debug cli.

Step 2: Poll for Target Transitions

Begin polling the active daemon process for events. Start with last_seen_seq: 0 and increment the sequence number as events are received:

fx debug cli --json '{"command": "wait-for-event", "last_seen_seq": 0, "timeout": 10}'

[!NOTE] Socket Startup Delay: The background daemon may take 1-2 seconds to initialize and bind the listening socket after fx test begins. If the initial command fails with Daemon socket not found, wait briefly and retry.

Example wait-for-event Response (Test Failure / Breakpoint):

{
  "success": true,
  "events": [
    {
      "seq": 1,
      "event": "stopped",
      "body": {
        "threadId": 1,
        "reason": "breakpoint"
      }
    }
  ]
}

[!WARNING] Thread ID Key Casing Mapping Pitfall: Output events returned by wait-for-event use camelCase keys (e.g. "threadId": 1). However, subsequent request commands (such as stack-trace, continue, or pause) strictly require snake_case parameters (e.g. "thread_id": 1). Always map "threadId" to "thread_id" when programmatically invoking requests.

Simultaneously, monitor the output and status of the background fx test task.

Step 3: Handle Execution Scenarios

Depending on the status of the test run, handle one of the following scenarios:

Scenario A: Test Failure (Debugger Suspended)

If wait-for-event returns a "stopped" event, it indicates a test failure or breakpoint hit. Perform diagnostics:

  1. Query Session State: Retrieve all threads and processes active in the session to identify the failing thread:

    fx debug cli --json '{"command": "get-state"}'
    

    Example Response:

    {
      "success": true,
      "body": {
        "threads": [
          { "id": 1, "name": "initial-thread" }
        ],
        "processes": { "12345": "my_test_binary" }
      }
    }
    

    Note: In the get-state thread list representation, the key is "id". Map this value to the "thread_id" parameter in subsequent requests.

  2. Retrieve Stack Trace: Get the stack trace for a suspended thread (thread_id) or across all threads in a process (pid):

    # Thread-level stack trace
    fx debug cli --json '{"command": "stack-trace", "thread_id": 1}'
    
    # Process-level stack trace (all threads in "pid")
    fx debug cli --json '{"command": "stack-trace", "pid": 12345}'
    

    Example Thread Response:

    {
      "success": true,
      "body": {
        "thread_id": 1,
        "stack_frames": [
          {
            "frame_index": 0,
            "name": "my_test_function",
            "source": {
              "name": "main_test.cc",
              "path": "/src/main_test.cc"
            },
            "line": 42,
            "column": 1
          }
        ],
        "total_frames": 1
      }
    }
    

    Example Process Response:

    {
      "success": true,
      "body": {
        "process_id": 12345,
        "stacks": [
          {
            "thread_id": 1,
            "stack_frames": [
              {
                "frame_index": 0,
                "name": "my_test_function",
                "source": {
                  "name": "main_test.cc",
                  "path": "/src/main_test.cc"
                },
                "line": 42,
                "column": 1
              }
            ],
            "total_frames": 1
          },
          {
            "thread_id": 2,
            "stack_frames": [
              {
                "frame_index": 0,
                "name": "worker_thread_entry",
                "source": {
                  "name": "worker.cc",
                  "path": "/src/worker.cc"
                },
                "line": 105,
                "column": 5
              }
            ],
            "total_frames": 1
          }
        ]
      }
    }
    
  3. Inspect Local Variables: Retrieve variables and arguments within a specific stack frame:

    fx debug cli --json '{"command": "variables", "thread_id": 1, "frame_index": 0}'
    
  4. Evaluate Dynamic Expressions: Evaluate expressions directly within a stack frame via evaluate.

    Note that evaluate requires the target thread to be stopped prior to invocation. Ensure the thread is suspended via a breakpoint or pause command before evaluating.

    Pass the expression directly (do not prefix with print):

    fx debug cli --json '{"command": "evaluate", "thread_id": 1, "frame_index": 0, "expression": "my_var"}'
    

    Example Response:

    {
      "success": true,
      "body": {
        "result": "42",
        "type": "int"
      }
    }
    

    To execute raw zxdb console commands that do not have a dedicated CLI subcommand (such as symbol inspection or disassembly), use the console command (optionally scoped to a thread_id and frame_index):

    fx debug cli --json '{"command": "console", "thread_id": 1, "frame_index": 0, "input": "sym-info my_var"}'
    
  5. Teardown Session: Once diagnostics are complete, terminate the debugging session. This detaches from targets and lets the background test runner exit cleanly:

    fx debug cli --json '{"command": "stop"}'
    

Scenario B: Clean Pass (All Tests Succeed)

If all tests pass cleanly, the background fx test task will complete successfully, exiting with status code 0 and summarizing FAILED: 0 in its output.

  • Edge Case: Poll Connection Failures: Since fx test automatically reaps the background daemon process on a clean exit, any active, blocking wait-for-event command will terminate abruptly with a connection error (e.g., connection refused or closed socket).
  • Action: If wait-for-event returns a connection failure, check the status of the background fx test task. If it completed successfully (exit code 0), treat this as a successful clean pass, print a success message, and exit. No manual "stop" command or teardown is required.

Scenario C: Proactive Breakpoint Installation (Non-Failing Paths)

By default, the testing framework attaches to targets weakly (attach --weak). In this state, symbols are not loaded upfront, and dynamic break requests will remain pending and unresolved until a crash occurs.

To debug a passing path or install breakpoints before test execution begins:

  1. Start the test with the --breakpoint option to force a normal attach with immediate symbol loading (paths must be fully qualified from the workspace root):

    fx test <test_name> --agent-debugging-mode --breakpoint <source_file>:<line>
    
  2. Poll for the breakpoint hit stopped event via wait-for-event.

  3. Once suspended, perform diagnostics and resume execution using continue.


JSON Request Reference

All commands are sent as serialized JSON payloads to fx debug cli --json '<payload>'.

ActionCommand Input Payload
Wait for Event{"command": "wait-for-event", "last_seen_seq": <seq>, "timeout": <secs>}
Get State{"command": "get-state"} (Returns active threads, processes, breakpoints, and tracepoints)
List Threads{"command": "threads"}
Set/Delete Breakpoint{"command": "break", "file": "<workspace_root_path>", "line": <line_num>, "condition": "<optional_expr>", "delete": <optional_bool>} or {"command": "break", "function": "<symbol>", "condition": "<optional_expr>", "delete": <optional_bool>} (CLI: break <file>:<line> or break --file <file> --line <line> or break -f <symbol> [-c <expr>])
Set/Manage Tracepoint{"command": "tracepoint", "file": "<workspace_root_path>", "line": <line_num>, ...} or {"command": "tracepoint", "function": "<symbol>", ...} or {"command": "tracepoint", "watch": "<expr_or_hex_addr>", "thread_id": <optional_tid>, "frame_index": <optional_idx>, "size": <optional_bytes>, "access_type": "write" | "readWrite", ...} with optional "condition": "<expr>", "expressions": ["<expr>"], "capture": ["frame", "backtrace", "locals", "registers"], "raw_backtrace": <bool>, "max_matching_frames": <int>, "max_hits": <int>. For management, pass {"command": "tracepoint", "list_all": true}, {"command": "tracepoint", "clear": true}, or {"command": "tracepoint", "tracepoint_id": <id>, "enabled": <bool> | "delete": true} (CLI: tracepoint (<file>:<line> | -f <sym> | -w <expr> [-t <id>]) [-c <expr>] [-e <expr>] [--locals] [--registers] [--backtrace] [-r] [--max-frames <n>] [--max-hits <n>] or tracepoint (--list | --clear | --enable <id> | --disable <id> | <id> -d). Aliases: tp, trace)
Attach Process / Driver{"command": "attach", "filter": "<name_or_pid>", "weak": <optional_bool>, "exact": <optional_bool>, "recursive": <optional_bool>, "job_only": <optional_bool>} or {"command": "attach", "driver": "<driver_name_or_url>", "weak": <optional_bool>, "wait": <optional_bool>} or {"command": "attach", "node": "<moniker>", "weak": <optional_bool>, "wait": <optional_bool>} *(CLI: `attach (
List Driver Hosts{"command": "driver-list", "filter": "<optional_substr>"} (CLI: driver-list [<filter>] or driver-list -f <filter>)
List Modules{"command": "modules", "pid": <optional_pid>, "filter": "<optional_substr>", "force_reload": <optional_bool>} (CLI: modules [-p <pid>] [-f <filter>] [-r])
Inspect Address Space{"command": "aspace", "pid": <optional_pid>, "address": "<optional_addr>", "filter": "<optional_substr>"} (CLI: aspace [addr] [-p <pid>] [-f <filter>])
Inspect Handles{"command": "handles", "pid": <optional_pid>, "type": "<optional_type>", "koid": <optional_koid>, "detail": <optional_bool>} (CLI: handles [-p <pid>] [--type <type>] [-k <koid>] [--detail])
Read Memory{"command": "mem-read", "address": "<addr_or_expr>", "size": <bytes>, "pid": <optional_pid>, "thread_id": <optional_thread_id>, "frame_index": <frame_index>} (CLI: mem-read <addr_or_expr> [-s <size>] [-p <pid>] [-t <id> [--frame-index <idx>]])
Async Backtrace{"command": "async-backtrace", "pid": <optional_pid>} (CLI: async-backtrace [-p <pid>])
Detach Process{"command": "detach", "pid": <pid>} or {"command": "detach", "all": true} (CLI: detach [<pid>] [--all])
Get Stack Trace{"command": "stack-trace", "thread_id": <thread_id>} or {"command": "stack-trace", "pid": <pid>, "raw": <optional_bool>}; optional "force_remote_unwind": <bool>, "unwinder": "cfi" | "fp" | "scs" | "plt" | "armehabi" | "sigreturn" (CLI: stack-trace (-t <id> | -p <pid>) [-r] [--force-remote-unwind] [--unwinder <name>])
List Variables{"command": "variables", "thread_id": <thread_id>, "frame_index": <frame_index>} (CLI: variables -t <id> --frame-index <idx>)
Evaluate Expression{"command": "evaluate", "thread_id": <thread_id>, "frame_index": <frame_index>, "expression": "<expr>"} (CLI: evaluate -t <id> [--frame-index <idx>] <expr>)
Run Console Command{"command": "console", "input": "<cmd>", "thread_id": <optional_thread_id>, "frame_index": <frame_index>} (CLI: console [-t <id>] [--frame-index <idx>] <cmd>)
Step In{"command": "step-in", "thread_id": <thread_id>} (CLI: step-in <id>)
Step Over (Next){"command": "next", "thread_id": <thread_id>} (CLI: next <id>)
Step Out (Finish){"command": "finish", "thread_id": <thread_id>} (CLI: finish <id>)
Continue Thread{"command": "continue", "thread_id": <thread_id>} (CLI: continue <id>)
Pause Thread / Process{"command": "pause", "thread_id": <thread_id>} or {"command": "pause", "pid": <pid>} (CLI: pause -t <id> or pause -p <pid>)
Stop Session{"command": "stop"}

Global Parameters

  • Event History Pruning (ack_seq): Any command payload can include "ack_seq": <seq> (e.g., {"command": "get-state", "ack_seq": 5}). The daemon will prune all event history up to the acknowledged sequence to optimize memory.

Performance & Blocking

  • Smart Blocking: Commands like pause, stack-trace, and wait-for-event are blocking operations and may take up to 10 seconds depending on the target's execution state. Unlike those commands, evaluate is non-blocking and fails immediately if the target thread is not already stopped.

Live Process & Driver Debugging (fx debug cli)

fx debug cli also provides workflows for live process inspection and DFv2 driver bring-up on running targets.

1. Discover & Safely Attach to a Driver Host

Because pausing a driver_host freezes all colocated drivers in that process (which can sever USB/CDC-NCM/ADB transport if transport or watchdog drivers are colocated), inspect driver host topology with driver-list before attaching:

# List running driver_hosts, colocated drivers, and transport hazards
fx debug cli driver-list --filter xhci

# Attach by driver name/URL or device node moniker (warns on transport hazards)
fx debug cli attach --driver xhci
fx debug cli attach --node dev.sys.platform.xhci

Driver Host Transport Hazards & Guidance

When a driver_host colocates transport or watchdog drivers (such as USB, CDC-NCM, ethernet, serial, or watchdog), avoid stopping breakpoints or manual pause commands to prevent severing host ffx communication. Instead:

  • Prefer weak attach (--weak) to catch crashes without stopping threads.
  • Use non-stopping inspection commands (modules, aspace, handles, mem-read) to examine process state safely.
  • Use non-stopping tracepoints (tracepoint) instead of breakpoints to sample variables or expressions without pausing threads.
# Attach weakly for crash-only catch on transport-sensitive driver hosts
fx debug cli attach --driver xhci --weak

Pending or Unbound Drivers (--wait)

When attaching to a driver that has not yet started or bound, pass --wait. If no running driver_host matches, fx debug cli installs a zxdb process filter on "driver_host" that attaches to every newly started driver host (since zxdb matches process names rather than driver URLs or device monikers).

Attach flags (--weak, --recursive, --job-only) are forwarded to this filter, and the response includes requested_target and filter_warning alerting the user that all new driver hosts will be attached.

# Pre-arm a filter before a driver binds or starts
fx debug cli attach --driver my_driver --wait

2. Inspect a Live Process

These commands return structured JSON and do not stop the target. When more than one process is attached, pass --pid (-p) to choose one.

# Inspect loaded modules/DSOs, virtual memory mappings, and Zircon handles
fx debug cli modules --filter xhci
fx debug cli aspace --filter mmio
fx debug cli handles --type interrupt --detail

# Read memory at an address (or at a frame-relative expression with -t)
fx debug cli mem-read 0x7ffee1234000 -s 32

modules requests the module list from debug_agent, which ends a weak attach.

3. Breakpoints and Stack Traces

# Set function or conditional breakpoints
fx debug cli break --function usb_xhci::UsbXhci::Init --condition "status != 0"

# If an unwind stops early, retry on the target or force a specific unwinder
fx debug cli stack-trace -p <pid> --force-remote-unwind
fx debug cli stack-trace -t <tid> --unwinder fp

4. Tracepoints (tracepoint)

A tracepoint (tracepoint, tp, or trace) captures state when a thread hits a code location or hardware watchpoint, then resumes the thread immediately. The backend emits a zxdb.Tracepoint event to wait-for-event and disables the tracepoint after max_hits events (20 by default).

# Capture an expression and local variables on a function (aliases: tp, trace)
fx debug cli tracepoint usb_xhci::UsbXhci::HandleEvent -e status --locals --max-hits 10

# Capture a backtrace on a source line when a condition holds
fx debug cli tp src/devices/usb/drivers/xhci/usb-xhci.cc:142 -c "ring != nullptr" --backtrace

# Set a watchpoint tracepoint on an expression (requires -t) or hex address
fx debug cli tp -w my_var -t 1 -a readWrite --size 4

# List, disable, re-enable (resets hit_count), delete, or clear tracepoints
fx debug cli tp --list
fx debug cli tp --disable 1
fx debug cli tp --enable 1
fx debug cli tp 1 -d
fx debug cli tp --clear

Poll wait-for-event to read zxdb.Tracepoint hit events:

{
  "success": true,
  "events": [
    {
      "seq": 2,
      "event": "zxdb.Tracepoint",
      "body": {
        "tracepointId": 1,
        "location": "usb_xhci::UsbXhci::HandleEvent",
        "hitCount": 1,
        "maxHits": 10,
        "autoDisabled": false,
        "truncated": false,
        "threadKoid": 34567,
        "dwellMs": 1.4,
        "capture": {
          "matched": true,
          "frame": {
            "name": "usb_xhci::UsbXhci::HandleEvent()",
            "file": "src/devices/usb/drivers/xhci/usb-xhci.cc",
            "line": 98,
            "ip": 4398046515200,
            "sp": 140732920758272,
            "trust": "CFI"
          },
          "locals": {
            "ring": { "value": "0x20004000", "type": "usb_xhci::EventRing*" }
          },
          "evaluations": {
            "status": { "result": "0", "type": "zx_status_t", "error": null }
          },
          "errors": []
        }
      }
    }
  ]
}