| # Diagnosing symbol problems in zxdb |
| |
| ## Variable values are unavailable |
| |
| Usually this is related to the optimization level of the program: |
| |
| _Optimized out_ indicates that the program symbols declare a variable with the given name, but |
| that it has no value or location. This means the compiler has entirely optimized out the variable |
| and the debugger can not show it. If you need to see it, use a less-optimized build setting. |
| |
| _Unavailable_ indicates that the variable is not valid at the current address, but that its value |
| is known at other addresses. In optimized code, the compiler will often re-use registers, clobbering |
| previous values, which become unavailable. |
| |
| You can see the valid ranges for a variable with the "sym-info" command: |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] sym-info my_variable |
| Variable: my_variable |
| Type: int |
| DWARF tag: 0x05 |
| DWARF location (address range + DWARF expression bytes): |
| [0x3e0d0a3e05b, 0x3e0d0a3e0b2): 0x70 0x88 0x78 |
| [0x3e0d0a3e0b2, 0x3e0d0a3eb11): 0x76 0x48 0x10 0xf8 0x07 0x1c 0x06 |
| |
| ``` |
| |
| Under "DWARF location" it will give a list of address ranges where the value of the variable is |
| known (inclusive at the beginning of the range, non-inclusive at the end). Run to one of these |
| addresses to see the value of the variable (use "di" to see the current address). |
| |
| You can ignore the "DWARF expression bytes" which are the internal instructions for finding the |
| variable. |
| |
| ## Can't find symbols |
| |
| The `sym-stat` command will tell you status for symbols. With no running process, it will give |
| information on the different symbol locations you have specified. If your symbols aren't found, make |
| sure this matches your expectations: |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] sym-stat |
| Symbol index status |
| |
| Indexed Source path |
| (folder) /home/me/.build-id |
| (folder) /home/me/build/out/x64 |
| 0 my_dir/my_file |
| ``` |
| |
| If you see "0" in the "Indexed" column of the "Symbol index status" that means that the debugger |
| could not find where your symbols are. See below for how to specify the location of these. |
| |
| Symbol sources using the ".build-id" hierarchy will list "(folder)" for the indexed symbols since |
| this type of source does not need to be indexed. To check if your hierarchy includes a given build |
| ID, go to ".build-id" inside it, then to the folder with the first to characters of the build ID to |
| see if there is a matching file. |
| |
| When you have a running program, `sym-stat` will additionally print symbol information for each |
| binary loaded into the process. If you're not getting symbols, find the entry for the binary or |
| shared library in this list. If it says: |
| |
| ```none {:.devsite-disable-click-to-copy} |
| Symbols loaded: No |
| ``` |
| |
| then that means it couldn't find the symbolized binary on the local computer for the given build ID |
| in any of the locations listed in "Symbol index status". You may need to add a new location with |
| `-s`. |
| |
| If instead it says something like this: |
| |
| ```none {:.devsite-disable-click-to-copy} |
| Symbols loaded: Yes |
| Symbol file: /home/foo/bar/... |
| Source files indexed: 1 |
| Symbols indexed: 0 |
| ``` |
| |
| where "Source files indexed" and "Symbols indexed" is 0 or a very low integer, that means that the |
| debugger found a symbolized file but there are few or no symbols in it. Normally this means the |
| binary was not built with symbols enabled or the symbols were stripped. Check your build, you should |
| be passing the path to the unstripped binary and the original compile line should have a `-g` in it |
| to get symbols. |
| |
| ## Understanding how Zxdb loads symbols |
| |
| Symbol settings should normally be set automatically by your environment (see [About symbol |
| settings](#about_symbol_settings) below) so most users should not have to do any configuration. This |
| section provides some implementation details to help in diagnosing problems. |
| |
| #### About build IDs |
| |
| Zxdb locates the symbols for a binary on the target device using the binary's "build ID". If the |
| build ID does not match, Zxdb will not load the symbols even if the file name is the same. To see |
| the build ID for a binary on Linux (Mac users will have to install readelf separately), dump the |
| "notes" for the ELF binary: |
| |
| ```none {:.devsite-disable-click-to-copy} |
| $ readelf -n my_binary |
| |
| ... (some other notes omitted) ... |
| |
| Displaying notes found in: .note.gnu.build-id |
| Owner Data size Description |
| GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) |
| Build ID: 18cec080fc47cdc07ec554f946f2e73d38541869 |
| ``` |
| |
| The `sym-stat` Zxdb command will show the build IDs for each binary and library loaded in the |
| currently attached process and the corresponding symbol file if found. |
| |
| #### Symbol servers |
| |
| Zxdb can load symbols for prebuilt libraries from Google servers or upstream debuginfod servers. |
| This is how symbols arrive for SDK users for anything not built locally. See [Downloading |
| symbols](attaching.md#downloading_symbols) for more. |
| |
| For large binaries, symbols can be several gigabytes so the download process can take many minutes. |
| The `sym-stat` command will display "Downloading..." during this time. |
| |
| Downloaded symbols are stored in the symbol cache. The `symbol-cache` setting contains the name of |
| this directory: |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] get symbol-cache |
| symbol-cache = /home/me/.fuchsia/debug/symbol-cache |
| ``` |
| |
| When downloading symbols from prebuilt binaries fetched from upstream packages, the debuginfo may |
| not have a corresponding ELF binary (depending on the server and the package). You may see errors |
| such as this in zxdb: |
| |
| ```none {:.devsite-disable-click-to-copy} |
| ... |
| binary for build_id 26820458adaf5d95718fb502d170fe374ae3ee70 not found on 5 servers |
| binary for build_id 53eaa845e9ca621f159b0622daae7387cdea1e97 not found on 5 servers |
| binary for build_id f3fd699712aae08bbaae3191eedba514c766f9d2 not found on 5 servers |
| binary for build_id 4286bd11475e673b194ee969f5f9e9759695e644 not found on 5 servers |
| binary for build_id 2d28b51427b49abcd41dcf611f8f3aa6a2811734 not found on 5 servers |
| binary for build_id 0401bd8da6edab3e45399d62571357ab12545133 not found on 5 servers |
| ... |
| ``` |
| |
| Indicating the ELF binary file was not found on the debuginfod server(s). This means that ELF |
| symbols will not be available for this particular binary, which is typically okay for most debugging |
| scenarios. The most commonly used ELF specific symbols that would be unavailable in this case are |
| PLT symbols. |
| |
| Use `sym-stat` to verify that DWARF information has been loaded (which is downloaded separately from |
| the ELF symbols): |
| |
| ```none {:.devsite-disable-click-to-copy} |
| libc.so.6 |
| Base: 0x1c85fdc2000 |
| Build ID: 0401bd8da6edab3e45399d62571357ab12545133 |
| Symbols loaded: Yes |
| Symbol file: /home/me/.fuchsia/debug/symbol-cache/04/01bd8da6edab3e45399d62571357ab12545133.debug |
| Source files indexed: 1745 |
| Symbols indexed: 10130 |
| ``` |
| |
| #### ".build-id" directory symbol databases |
| |
| Many build environments, including the main "fuchsia.git" repository, add symbolized binaries in a |
| standard directory structure called ".build-id". This directory contains subdirectories named |
| according to the first two characters of the binary's build ID, and inside those directories will be |
| the symbol files named according to the remaining characters of the build ID. |
| |
| You can set one or more build ID directories (they do not need to be named ".build-id") on the |
| command line or interactively using the `build-id-dirs` setting (a list of directory paths): |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] set build-id-dirs += "/home/me/project/out/x64/.build-id" |
| ``` |
| |
| These directories will appear in the output of the `sym-stat` command but will be annotated with |
| "(folder)" rather than the number of binaries found in it, and the binaries will not appear in the |
| `sym-stat --dump-index` output. This is because Zxdb searches these directories on demand when |
| searching for symbols rather than enumerating them in advance. |
| |
| #### Individual files and directories |
| |
| If you have a single binary file without one of the other symbol database formats, you can tell Zxdb |
| about the file individually. You can use a command-line flag or set it interactively using the |
| `symbol-paths` setting (a list of files or directories): |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] set symbol-paths += /home/me/project/a.out |
| ``` |
| |
| This setting also accepts directory names. In this case, Zxdb will non-recursively enumerate all |
| files in that directory and look for binaries with build IDs: |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] set symbol-paths += /home/me/project/build/ |
| ``` |
| |
| To see the status of the locations you provided: |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] sym-stat |
| Symbol index status |
| |
| This command just refreshed the index. |
| Use "sym-stat --dump-index" to see the individual mappings. |
| |
| Indexed Source path |
| 1 /home/me/a.out |
| 2 /home/me/project/build/ |
| ``` |
| |
| You can also see the build IDs and file names of the binaries added in this way with the |
| `sym-stat --dump-index` command. |
| |
| #### "ids.txt" symbol index |
| |
| Some older internal Google projects generate a file called "ids.txt". This provides a mapping from a |
| binary's build ID to the symbol path on the local system. If your build produces such a file and it |
| is not automatically loaded, you can provide it to Zxdb via a command-line flag or interactively |
| using the `ids-txts` setting (a list of file names): |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] set ids-txts += "/home/me/project/build/ids.txt" |
| ``` |
| |
| The symbol files from ids.txt files will also be reflected in the `sym-stat` and |
| `sym-stat --dump-index` commands described in the previous section. |
| |
| ## About symbol settings |
| |
| The settings described in the above [Understanding how Zxdb loads |
| symbols](#understanding_how_zxdb_loads_symbols) section should get automatically applied by your |
| environment. This section describes how they get set for help debugging symbol load problems. |
| |
| The `symbol-index-files` setting contains one or more JSON-format files that should be set by the |
| development environment: |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] get symbol-index-files |
| symbol-index-files = |
| • /home/me/.fuchsia/debug/symbol-index.json |
| ``` |
| |
| This file can contain some global settings and also refer to other symbol-index files. Typically |
| each build environment you are actively using will have a similar file that is included by reference |
| from this global file. If you are switching between build environments and find symbols aren't |
| loading, please make sure your environment is registered by checking the |
| `ffx debug symbol-index list` command. Typically, these are automatically inserted into the included |
| list when you run any `ffx debug ...` subtool. |
| |
| ## Mismatched source lines |
| |
| Sometimes the source file listings may not match the code. The most common reason is that the build |
| is out-of-date and no longer matches the source. The debugger will check that the symbol file |
| modification time is newer than the source file, but it will only print the warning the first time |
| the file is displayed. Check for this warning if you suspect a problem. |
| |
| Some people have multiple checkouts. If it's finding a file in the wrong one, override the |
| `build-dirs` option as described above in [the setup guide](running.md). |
| |
| To display the file name of the file it found from `list`, use the `-f` option: |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] list -f |
| /home/me/fuchsia/out/x64/../../src/foo/bar.cc |
| ... <source code> ... |
| ``` |
| |
| You can also set the `show-file-paths` option. This will increase file path information: |
| |
| * It will show the full resolved path in source listings as in `list -f`. |
| * It will show the full path instead of just the file name in other places such as backtraces. |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] set show-file-paths true |
| ``` |
| |
| You may notice a mismatch when setting a breakpoint on a specific line where the displayed |
| breakpoint location doesn't match the line number you typed. In most cases, this is because this |
| symbols did not identify any code on the specified line so the debugger used the next line. It can |
| happen even in unoptimized builds, and is most common for variable declarations. |
| |
| ```none {:.devsite-disable-click-to-copy} |
| [zxdb] b file.cc:138 |
| Breakpoint 1 (Software) @ file.cc:138 |
| 138 int my_value = 0; <- Breakpoint was requested here. |
| ◉ 139 DoSomething(&my_value); <- But ended up here. |
| 140 if (my_value > 0) { |
| ``` |
| |