Diagnosing symbol problems in zxdb

Variable values are unavailable

Usually this is related to the optimization level of the program:

Optimized out indicates that the program symbols declare a variable with the given name, but that it has no value or location. This means the compiler has entirely optimized out the variable and the debugger can not show it. If you need to see it, use a less-optimized build setting.

Unavailable indicates that the variable is not valid at the current address, but that its value is known at other addresses. In optimized code, the compiler will often re-use registers, clobbering previous values, which become unavailable.

You can see the valid ranges for a variable with the “sym-info” command:

[zxdb] sym-info my_variable
Variable: my_variable
  Type: int
  DWARF tag: 0x05
  DWARF location (address range + DWARF expression bytes):
    [0x3e0d0a3e05b, 0x3e0d0a3e0b2): 0x70 0x88 0x78
    [0x3e0d0a3e0b2, 0x3e0d0a3eb11): 0x76 0x48 0x10 0xf8 0x07 0x1c 0x06

Under “DWARF location” it will give a list of address ranges where the value of the variable is known (inclusive at the beginning of the range, non-inclusive at the end). Run to one of these addresses to see the value of the variable (use “di” to see the current address).

You can ignore the “DWARF expression bytes” which are the internal instructions for finding the variable.

Can't find symbols

The sym-stat command will tell you status for symbols. With no running process, it will give information on the different symbol locations you have specified. If your symbols aren't found, make sure this matches your expectations:

[zxdb] sym-stat
Symbol index status

  Indexed  Source path
 (folder)  /home/me/.build-id
 (folder)  /home/me/build/out/x64
        0  my_dir/my_file

If you see “0” in the “Indexed” column of the “Symbol index status” that means that the debugger could not find where your symbols are. See below for how to specify the location of these.

Symbol sources using the “.build-id” hierarchy will list “(folder)” for the indexed symbols since this type of source does not need to be indexed. To check if your hierarchy includes a given build ID, go to “.build-id” inside it, then to the folder with the first to characters of the build ID to see if there is a matching file.

When you have a running program, sym-stat will additionally print symbol information for each binary loaded into the process. If you're not getting symbols, find the entry for the binary or shared library in this list. If it says:

    Symbols loaded: No

then that means it couldn't find the symbolized binary on the local computer for the given build ID in any of the locations listed in “Symbol index status”. You may need to add a new location with -s.

If instead it says something like this:

    Symbols loaded: Yes
    Symbol file: /home/foo/bar/...
    Source files indexed: 1
    Symbols indexed: 0

where “Source files indexed” and “Symbols indexed” is 0 or a very low integer, that means that the debugger found a symbolized file but there are few or no symbols in it. Normally this means the binary was not built with symbols enabled or the symbols were stripped. Check your build, you should be passing the path to the unstripped binary and the original compile line should have a -g in it to get symbols.

Understanding how Zxdb loads symbols

Symbol settings should normally be set automatically by your environment (see About symbol settings below) so most users should not have to do any configuration. This section provides some implementation details to help in diagnosing problems.

About build IDs

Zxdb locates the symbols for a binary on the target device using the binary's “build ID”. If the build ID does not match, Zxdb will not load the symbols even if the file name is the same. To see the build ID for a binary on Linux (Mac users will have to install readelf separately), dump the “notes” for the ELF binary:

$ readelf -n my_binary

  ... (some other notes omitted) ...

Displaying notes found in: .note.gnu.build-id
  Owner                Data size 	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: 18cec080fc47cdc07ec554f946f2e73d38541869

The sym-stat Zxdb command will show the build IDs for each binary and library loaded in the currently attached process and the corresponding symbol file if found.

Symbol servers

Zxdb can load symbols for prebuilt libraries from Google servers or upstream debuginfod servers. This is how symbols arrive for SDK users for anything not built locally. See Downloading symbols for more.

For large binaries, symbols can be several gigabytes so the download process can take many minutes. The sym-stat command will display “Downloading...” during this time.

Downloaded symbols are stored in the symbol cache. The symbol-cache setting contains the name of this directory:

[zxdb] get symbol-cache
symbol-cache = /home/me/.fuchsia/debug/symbol-cache

When downloading symbols from prebuilt binaries fetched from upstream packages, the debuginfo may not have a corresponding ELF binary (depending on the server and the package). You may see errors such as this in zxdb:

...
binary for build_id 26820458adaf5d95718fb502d170fe374ae3ee70 not found on 5 servers
binary for build_id 53eaa845e9ca621f159b0622daae7387cdea1e97 not found on 5 servers
binary for build_id f3fd699712aae08bbaae3191eedba514c766f9d2 not found on 5 servers
binary for build_id 4286bd11475e673b194ee969f5f9e9759695e644 not found on 5 servers
binary for build_id 2d28b51427b49abcd41dcf611f8f3aa6a2811734 not found on 5 servers
binary for build_id 0401bd8da6edab3e45399d62571357ab12545133 not found on 5 servers
...

Indicating the ELF binary file was not found on the debuginfod server(s). This means that ELF symbols will not be available for this particular binary, which is typically okay for most debugging scenarios. The most commonly used ELF specific symbols that would be unavailable in this case are PLT symbols.

Use sym-stat to verify that DWARF information has been loaded (which is downloaded separately from the ELF symbols):

  libc.so.6
    Base: 0x1c85fdc2000
    Build ID: 0401bd8da6edab3e45399d62571357ab12545133
    Symbols loaded: Yes
    Symbol file: /home/me/.fuchsia/debug/symbol-cache/04/01bd8da6edab3e45399d62571357ab12545133.debug
    Source files indexed: 1745
    Symbols indexed: 10130

“.build-id” directory symbol databases

Many build environments, including the main “fuchsia.git” repository, add symbolized binaries in a standard directory structure called “.build-id”. This directory contains subdirectories named according to the first two characters of the binary's build ID, and inside those directories will be the symbol files named according to the remaining characters of the build ID.

You can set one or more build ID directories (they do not need to be named “.build-id”) on the command line or interactively using the build-id-dirs setting (a list of directory paths):

[zxdb] set build-id-dirs += "/home/me/project/out/x64/.build-id"

These directories will appear in the output of the sym-stat command but will be annotated with “(folder)” rather than the number of binaries found in it, and the binaries will not appear in the sym-stat --dump-index output. This is because Zxdb searches these directories on demand when searching for symbols rather than enumerating them in advance.

Individual files and directories

If you have a single binary file without one of the other symbol database formats, you can tell Zxdb about the file individually. You can use a command-line flag or set it interactively using the symbol-paths setting (a list of files or directories):

[zxdb] set symbol-paths += /home/me/project/a.out

This setting also accepts directory names. In this case, Zxdb will non-recursively enumerate all files in that directory and look for binaries with build IDs:

[zxdb] set symbol-paths += /home/me/project/build/

To see the status of the locations you provided:

[zxdb] sym-stat
Symbol index status

  This command just refreshed the index.
  Use "sym-stat --dump-index" to see the individual mappings.

   Indexed  Source path
         1  /home/me/a.out
         2  /home/me/project/build/

You can also see the build IDs and file names of the binaries added in this way with the sym-stat --dump-index command.

“ids.txt” symbol index

Some older internal Google projects generate a file called “ids.txt”. This provides a mapping from a binary's build ID to the symbol path on the local system. If your build produces such a file and it is not automatically loaded, you can provide it to Zxdb via a command-line flag or interactively using the ids-txts setting (a list of file names):

[zxdb] set ids-txts += "/home/me/project/build/ids.txt"

The symbol files from ids.txt files will also be reflected in the sym-stat and sym-stat --dump-index commands described in the previous section.

About symbol settings

The settings described in the above Understanding how Zxdb loads symbols section should get automatically applied by your environment. This section describes how they get set for help debugging symbol load problems.

The symbol-index-files setting contains one or more JSON-format files that should be set by the development environment:

[zxdb] get symbol-index-files
symbol-index-files =
   /home/me/.fuchsia/debug/symbol-index.json

This file can contain some global settings and also refer to other symbol-index files. Typically each build environment you are actively using will have a similar file that is included by reference from this global file. If you are switching between build environments and find symbols aren't loading, please make sure your environment is registered by checking the ffx debug symbol-index list command. Typically, these are automatically inserted into the included list when you run any ffx debug ... subtool.

Mismatched source lines

Sometimes the source file listings may not match the code. The most common reason is that the build is out-of-date and no longer matches the source. The debugger will check that the symbol file modification time is newer than the source file, but it will only print the warning the first time the file is displayed. Check for this warning if you suspect a problem.

Some people have multiple checkouts. If it's finding a file in the wrong one, override the build-dirs option as described above in the setup guide.

To display the file name of the file it found from list, use the -f option:

[zxdb] list -f
/home/me/fuchsia/out/x64/../../src/foo/bar.cc
 ... <source code> ...

You can also set the show-file-paths option. This will increase file path information:

  • It will show the full resolved path in source listings as in list -f.
  • It will show the full path instead of just the file name in other places such as backtraces.
[zxdb] set show-file-paths true

You may notice a mismatch when setting a breakpoint on a specific line where the displayed breakpoint location doesn't match the line number you typed. In most cases, this is because this symbols did not identify any code on the specified line so the debugger used the next line. It can happen even in unoptimized builds, and is most common for variable declarations.

[zxdb] b file.cc:138
Breakpoint 1 (Software) @ file.cc:138
   138   int my_value = 0;          <- Breakpoint was requested here.
  139   DoSomething(&my_value);    <- But ended up here.
   140   if (my_value > 0) {