docs/get-started/learn/components/debugging-components.md

# Debugging components

Developing software means dealing with program crashes and uncovering the
source of bugs. Fuchsia has a suite of tools to help identify and diagnose
issues in your components from analyzing crash logs to full step-through
debugging in core code.


## Analyzing crashes

Fuchsia starts a program at boot called `crashanalyzer` that reports program
crashes and prints a backtrace of the crashing thread to the system log. While
you can explore these directly by reviewing the logs at runtime, the backtrace
content is encoded using the stack memory address references rather than
pointing to the corresponding lines in the program source files.


```none {:.devsite-disable-click-to-copy}
[klog][I] devmgr: crash_analyzer_listener: analyzing exception type 0x108
[klog][I] <== fatal exception: process crasher[42410] thread initial-thread[42424]
[klog][I] <== fatal page fault, PC at 0x1e1888dbbbd7
[klog][I]  CS:                   0 RIP:     0x1e1888dbbbd7 EFL:            0x10246 CR2:                  0
[klog][I]  RAX:                  0 RBX:                0x1 RCX:     0x721ad98697c6 RDX:     0x77accb36f264
[klog][I]  RSI:                  0 RDI:                  0 RBP:     0x2781c4816f90 RSP:     0x2781c4816f80
[klog][I]   R8:                  0  R9:                  0 R10:                  0 R11:              0x246
[klog][I]  R12:     0x773bf11dcda0 R13:     0x773bf11dcdd0 R14:               0x16 R15:         0x78050d69
[klog][I]  errc:               0x6
[klog][I] bottom of user stack:
[klog][I] 0x00002781c4816f80: f11dcda0 0000773b 9ccd2b38 000039b2 |....;w..8+...9..|
[klog][I] 0x00002781c4816f90: c4816fd0 00002781 88dbbba7 00001e18 |.o...'..........|
[klog][I] 0x00002781c4816fa0: 00000008 00000000 9ccd2b38 000039b2 |........8+...9..|
[klog][I] 0x00002781c4816fb0: f11dcf70 0000773b f11dcf70 0000773b |p...;w..p...;w..|
[klog][I] 0x00002781c4816fc0: cb36f570 000077ac f11dcdd0 0000773b |p.6..w......;w..|
[klog][I] 0x00002781c4816fd0: c4816ff0 00002781 cb2b0d0f 000077ac |.o...'....+..w..|
[klog][I] 0x00002781c4816fe0: 00000054 00000000 f11dcf70 0000773b |T.......p...;w..|
[klog][I] 0x00002781c4816ff0: f11dcfe0 0000773b 00000000 00000000 |....;w..........|
[klog][I] arch: x86_64
[klog][I] dso: id=a94c78564173530d51670b6586b1aa471e004f06 base=0x7d3506a49000 name=libfdio.so
[klog][I] dso: id=a61961ba9776a67a00fb322af9ebbdcfd1ce3f62 base=0x77accb297000 name=libc.so
[klog][I] dso: id=760f1e6e47d3dd8b6a19150aa47241279ec75a9c base=0x721ad9863000 name=<vDSO>
[klog][I] dso: id=b18462140c6784a53736105bbf3021852eeda68c base=0x1e1888dbb000 name=app:crasher
[klog][I] bt#01: pc 0x1e1888dbbbd7 sp 0x2781c4816f80 (app:crasher,0xbd7)
[klog][I] bt#02: pc 0x1e1888dbbba7 sp 0x2781c4816fa0 (app:crasher,0xba7)
[klog][I] bt#03: pc 0x77accb2b0d0f sp 0x2781c4816fe0 (libc.so,0x19d0f)
[klog][I] bt#04: pc 0 sp 0x2781c4817000
[klog][I] bt#05: end
```

This is because the **debug symbols** are stripped out of the core binaries
by default at build time. To properly analyze the crash log, you need to
reapply those symbols to the backtrace to see the call stack in terms of source
code line numbers. When you call the `ffx log` command, the developer tools
process the raw log through an additional binary called `symbolizer` that
reapplies the symbols from your local build configuration to any backtraces in
the log.

```posix-terminal
ffx log
```

The output you see includes the symbols reapplied to the backtrace:

```none {:.devsite-disable-click-to-copy}
[klog][I] devmgr: crash_analyzer_listener: analyzing exception type 0x108
... same output as "raw" backtrace ...
start of symbolized stack:
[klog][I] #01: blind_write at ../../src/developer/forensics/crasher/cpp/crasher.c:21
[klog][I] #02: main at ../../src/developer/forensics/crasher/cpp/crasher.c:137
[klog][I] #03: start_main at ../../zircon/third_party/ulib/musl/src/env/__libc_start_main.c:49
[klog][I] #04: unknown, can't find pc, sp or app/library in line
end of symbolized stack
```

<aside class="key-point">
If you have a manually captured backtrace, you can run the same process by
passing the content through the <code>fx symbolize</code> command.
</aside>

With a properly symbolized backtrace, you can directly discover the site of a
crash in your source code.

### Step-through debugging

Just knowing where a program crashed may not be enough information to fully
diagnose the issue. Sometimes it's necessary to walk through the code
step-by-step and even inspect the state of variables in memory. To support this,
Fuchsia has a debugger for core code called `zxdb`.

The `zxdb` tool is a client that connects to a running `debug_agent` component
on the target device. You can use the `zxdb` commands to configure the
`debug_agent` to attach to specific processes and set breakpoints. Once a debug
session is attached to a running process, `zxdb` allows you to step through the
code and inspect the stack frames.

![Diagram showing how the Fuchsia debugger (zxdb) interacts with the
debug_agent service running on a Fuchsia device to perform interactive
debugging of a process.](images/zxdb.png){: width="591"}

Setting up the debug session requires the following high-level steps:

1.  Run the `debug_agent` component on the target device.
1.  Run the `zxdb` client and connect to the target device.
1.  Set the location for `zxdb` to find debug symbols.

The simplest method to start a debug session is to use the `fx debug` command,
which does all of these in the context of your local Fuchsia build. However,
these steps can also be performed manually if you need to configure them
separately.

Once the debug session is active, you are taken to a `[zxdb]` prompt to issue
debugger commands. You can use `zxdb` to configure the `debug_agent` to attach
to a process using a name filter and set pending breakpoints even if no
matching process is currently running.

The following example sets a pending breakpoint on main to stop at the
beginning of execution, and waits for a process called "hello-world" to start:

```none
[zxdb] attach hello-world
Waiting for process matching "hello-world"

[zxdb] break main
Breakpoint 1 (Software) on Global, Enabled, stop=All, @ main
Pending: No matches for location, it will be pending library loads.
```

<aside class="caution">
  <b>Launching components with zxdb</b>
  <p>The Fuchsia debugger supports two main methods of debugging a process:
  <code>run</code> and <code>attach</code>. Avoid using <code>run</code> to
  start and debug components.</p>

  <p>When the debugger launches a component, the component will have the same
  capabilities as the <code>debug_agent</code>. The only way to get the correct
  environment is to launch your component in the way it expects and attach the
  debugger to it.</p>
</aside>

Once the debugger is attached to a process, you can use `zxdb` commands to
control and inspect the state of the process. Here is a short collection of
common commands:

Note: For complete details and reference on `zxdb` commands, see
[The Fuchsia debugger](/docs/development/debugger).

<table>
  <tr>
   <td><code>step</code>
   </td>
   <td>Step over the next line of code in the thread
   </td>
  </tr>
  <tr>
   <td><code>next</code>
   </td>
   <td>Step into the next line of code in the thread
   </td>
  </tr>
  <tr>
   <td><code>continue</code>
   </td>
   <td>Continue execution until the next breakpoint, exception, or exit
   </td>
  </tr>
  <tr>
   <td><code>frame</code>
   </td>
   <td>List or select from the current stack frames
   </td>
  </tr>
  <tr>
   <td><code>print</code>
   </td>
   <td>Evaluate an expression and print the result
   </td>
  </tr>
</table>

## Exercise: Using the Fuchsia debugger

In this exercise, you'll use the Fuchsia debugger (`zxdb`) to inspect a running
instance of the `echo-args` component and understand the cause of a crash.

<<../_common/_start_femu.md>>

### Start a debug session

Once the emulator has started up, start a `zxdb` debugging session with the
`fx debug` command:

```posix-terminal
fx debug
```

```none
Checking for debug agent on [...]:2345.
Debug agent not found. Starting one.
Connecting (use "disconnect" to cancel)...
Connected successfully.
👉 To get started, try "status" or "help".
[zxdb]
```

After successfully connecting, the `zxdb` prompt is ready to accept commands.

### Attach to the component

Before launching the component, configure `zxdb` to attach to an instance of
`greeter`. This enables the debugger to attach as soon as the process starts:

```none
[zxdb] attach echo_args
```

Set a breakpoint on the `greeting()` function:

```none
[zxdb] break greeting
```

With the debugger ready, start a new `echo-args` component instance:


```posix-terminal
ffx component run fuchsia-pkg://fuchsia.com/echo-args#meta/echo_args.cm
```

### Explore the debug session

Upon reaching the breakpoint in `greeting()`, execution stops and the debugger
waits for a new command. Use the `list` command to show where execution is
currently paused:

```none
[zxdb] list
   18
   19 // Return a proper greeting for the list
   20 fn greeting(names: &Vec<String>) -> String {
   21     // Join the list of names based on length
 ▶ 22     match names.len() {
   23         0 => String::from("Nobody"),
   24         1 => names.join(""),
   25         2 => names.join(" and "),
   26         _ => names.join(", "),
   27     }
   28 }
   29
```

The `print` command will output the state of any variables in the current stack
frame. Print the current value of `names`:

```none
[zxdb] print names
vec!["Alice", "Bob", "Spot"]
```

Step through the `greeting()` function a few times using the `next` command:

```none
[zxdb] next
```

To let the program continue to completion, use the `continue` command:

```none
[zxdb] continue
```

Exit the debugging session to return to the terminal:

```none
[zxdb] exit
```

### Introduce some crashing code

Next, you'll add some code to `src/main.rs` to cause the component to crash
(or panic). Simulate this behavior by adding an `assert!(false)` macro just
after the arguments are collected:

`echo-args/src/main.rs`:

```rust
#[fuchsia::component(logging = true)]
async fn main() -> Result<(), anyhow::Error> {
    // ...

    {{ '<strong>' }}// Simulate a crash {{ '</strong>' }}
    {{ '<strong>' }}assert!(false, "fake crash");{{ '</strong>' }}

    // Print a greeting to syslog
    info!("Hello, {}!", greeting(&args));

    Ok(())
}
```

Run `fx build` again to rebuild the component:

```posix-terminal
fx build
```

Start a new debug session with `zxdb`:

```posix-terminal
fx debug
```

### Debug the crashing stack frame

Configure the debugger to attach to the `echo-args` component:

```none
[zxdb] attach echo_args
```

Start a new instance of the component:


```posix-terminal
ffx component run fuchsia-pkg://fuchsia.com/echo-args#meta/echo_args.cm
```

This time, the debugger detects that an exception was thrown. Use the `frame`
command to inspect the stack trace at the point of the crash:

```none
[zxdb] frame
▶ 0 abort() • abort.c:7
  1 panic_abort::__rust_start_panic::abort() • panic_abort/src/lib.rs:43
  2 panic_abort::__rust_start_panic(…) • panic_abort/src/lib.rs:38
  3 std::panicking::rust_panic(…) • library/std/src/panicking.rs:672
  4 std::panicking::rust_panic_with_hook(…) • library/std/src/panicking.rs:642
  5 std::panicking::begin_panic::$({closure#0}<&str>)() • rust/library/std/src/panicking.rs:544
  6 std::sys_common::backtrace::$(__rust_end_short_backtrace<std::panicking::begin_panic::{closure#0}, !>)(…) • rust/library/std/src/sys_common/backtrace.rs:144
  7 std::panicking::begin_panic<…>(…) • rust/library/std/src/panicking.rs:543
  {{ '<strong>' }}8 echo_args::main::component_entry_point::$({generator#0})(…) • main.rs:18{{ '</strong>' }}
  9 core::future::from_generator::$({impl#1})::$(poll<echo_args::main::component_entry_point::{generator#0}>)(…) • rust/library/core/src/future/mod.rs:80
  10 core::future::future::$({impl#1})::$(poll<&mut core::future::from_generator::GenFuture<echo_args::main::component_entry_point::{generator#0}>>)(…) • future/future.rs:119
  11 futures_util::future::future::FutureExt::$(poll_unpin<core::pin::Pin<&mut core::future::from_generator::GenFuture<echo_args::main::component_entry_point::{generator#0}>>>)(…) • future/future/mod.rs:562
  12 fuchsia_async::runtime::fuchsia::executor::local::MainTask::$(poll<core::pin::Pin<&mut core::future::from_generator::GenFuture<echo_args::main::component_entry_point::{generator#0}>>>)(…) • fuchsia/src/lib/fuchsia-async/src/runtime/fuchsia/executor/local.rs:444
  13 fuchsia_async::runtime::fuchsia::executor::local::LocalExecutor::$(run_singlethreaded<core::future::from_generator::GenFuture<echo_args::main::component_entry_point::{generator#0}>>)(…) • fuchsia/src/lib/fuchsia-async/src/runtime/fuchsia/executor/local.rs:73
  14 fuchsia::$(main_singlethreaded<fuchsia::init_logging_for_component_with_executor::{closure#0}, core::future::from_generator::GenFuture<echo_args::main::component_entry_point::{generator#0}>, core::result::Result<(), anyhow::Error>>)(…) • fuchsia/src/lib/fuchsia/src/lib.rs:152
  15 echo_args::main() • main.rs:7
  16 core::ops::function::FnOnce::call_once<…>(…) • /b/s/w/ir/x/w/rust/library/core/src/ops/function.rs:227
  17 std::sys_common::backtrace::__rust_begin_short_backtrace<…>(…) • rust/library/std/src/sys_common/backtrace.rs:125
  18 std::rt::lang_start::$({closure#0}<core::result::Result<(), anyhow::Error>>)() • rust/library/std/src/rt.rs:63
  19 core::ops::function::impls::$({impl#2})::call_once<…>(…) • /b/s/w/ir/x/w/rust/library/core/src/ops/function.rs:259 (inline)
  20 std::panicking::try::do_call<…>(…) • library/std/src/panicking.rs:403 (inline)
  21 std::panicking::try<…>(…) • library/std/src/panicking.rs:367 (inline)
  22 std::panic::catch_unwind<…>(…) • library/std/src/panic.rs:129 (inline)
  23 std::rt::lang_start_internal::$({closure#2})() • library/std/src/rt.rs:45 (inline)
  24 std::panicking::try::$(do_call<std::rt::lang_start_internal::{closure#2}, isize>)(…) • library/std/src/panicking.rs:403 (inline)
  25 std::panicking::$(try<isize, std::rt::lang_start_internal::{closure#2}>)(…) • library/std/src/panicking.rs:367 (inline)
  26 std::panic::$(catch_unwind<std::rt::lang_start_internal::{closure#2}, isize>)(…) • library/std/src/panic.rs:129 (inline)
  27 std::rt::lang_start_internal(…) • library/std/src/rt.rs:45
  28 std::rt::lang_start<…>(…) • rust/library/std/src/rt.rs:62
  29 $elf(main) + 0x1f
  30 «libc startup» (-r expands)
  31 «libc startup» (-r expands)
  32 $elf(_start) + 0x11
```

Notice line 8 in the stack trace indicates the point in `src/main.rs` where the
crash happened, corresponding to the `assert!()` macro line of code.

The current stack frame (frame 0) is deep within the system library, but you
can inspect any stack frame by prefixing the command with the frame number from
the stack trace. Print the value of `args` at the point of the crash by passing
the frame number as follows:

```none
[zxdb] frame 8 print args
vec!["Alice", "Bob", "Spot"]
```

Exit the debugging session to return to the terminal:

```none
[zxdb] exit
```