| # Handling results found through fuzzing |
| |
| When [your fuzzer runs](run-a-fuzzer.md), it searches for inputs that crash the program or violate |
| checked conditions. When the fuzzer finds and reports such a test input, it is evidence of a bug |
| that needs to be resolved. |
| |
| Typically, you might use `ffx fuzz` when first developing your fuzzer. This can often produce |
| results immediately. After a fuzzer has been submitted, it is run at scale by |
| [ClusterFuzz][clusterfuzz]{:.external} and any results it finds will be filed as bugs. |
| |
| ## Handle results from `ffx fuzz` {#ffx-fuzz-results} |
| |
| When [running a fuzzer with `ffx fuzz`](run-a-fuzzer.md#run-on-device), output logs and results are |
| stored to the output directory. By default, this is the current working directory. A different |
| location can be set passing the `--output` option to `attach`. |
| |
| Crashes and other artifacts will have file name like |
| `{{ "<var>" }}type-of-finding{{ "</var>" }}.{{ "<var>" }}SHA2-hash-of-input{{ "</var>" }}`. The file |
| contents will be the input bytes themselves. |
| |
| For example, examining a crash produced by a [toy example][toy-example] might look like the |
| following: |
| |
| <pre class="prettyprint devsite-disable-click-to-copy"> |
| <code class="devsite-terminal">ffx fuzz run -o results fuchsia.pkg://fuchsia.com/example-fuzzers#meta/toy_example_arbitrary.cm</code> |
| <code class="devsite-terminal">cd test_data/fuzzing/example-fuzzers/toy_example_arbitrary/latest</code> |
| <code class="devsite-terminal">hd results/crash-2fda3f03bb699c8a2151724b64b6e36c3b986aea</code> |
| 00000000 2a 48 49 21 2a 00 08 00 00 00 2a 48 49 00 0a 66 |*HI!*.....*HI..f| |
| 00000010 4a 33 00 0a f9 |J3...| |
| 00000015 |
| </pre> |
| |
| ### Reproduce a result {#repro} |
| |
| You can execute the fuzzer with this input again using `ffx fuzz try`. For example: |
| |
| <pre class="devsite-terminal devsite-disable-click-to-copy"> |
| ffx fuzz try fuchsia.pkg://fuchsia.com/example-fuzzers#meta/toy_example_arbitrary.cm crash-2fda3f03bb699c8a2151724b64b6e36c3b986aea |
| </pre> |
| |
| If the result is reproducible, this will produce a symbolized log including a stack trace. The top |
| of this stack trace is likely to be the error handling by libFuzzer and/or the sanitizer, and the |
| bottom will likely be the fuzzer engine itself. |
| |
| For example, the relevant function where the `panic` occurred in the following stack trace is |
| `_toy_example_arbitrary_lib_rustc_static::toy_example`: |
| |
| <pre class="prettyprint devsite-disable-click-to-copy"> |
| #0 0x000023c56780a61e in _$LT$std..sys_common..backtrace.._print..DisplayBacktrace$u20$as$u20$core..fmt..Display$GT$::fmt::h510ae2e0fe71c88c <>+0x19161e |
| #1 0x000023c56783399c in core::fmt::write::hb61ef49191e76a74 <>+0x1ba99c |
| #2 0x000023c5678009b1 in std::io::Write::write_fmt::h41df81fb2b8460af <>+0x1879b1 |
| #3 0x000023c56780eb92 in std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::h4e9a8e3c4f33b3f4 <>+0x195b92 |
| #4 0x000023c56780e87c in std::panicking::default_hook::hd85edcd963c04eae <>+0x19587c |
| #5 0x000023c56780f271 in std::panicking::rust_panic_with_hook::h8960558cc7e69505 <>+0x196271 |
| #6 0x000023c5677973d5 in std::panicking::begin_panic::h97c6d4cd722282c5 /b/s/w/ir/k/rust/src/libstd/panicking.rs:397 <>+0x11e3d5 |
| #7 0x000023c56777f2d0 in _toy_example_arbitrary_lib_rustc_static::toy_example::h573322211ba71c22 ../../out/default/../../examples/fuzzers/rust/src/lib.rs:22 <>+0x1062d0 |
| #8 0x000023c567780a03 in _toy_example_arbitrary_lib_rustc_static::_::toy_example_arbitrary::hc02c288d17b25ac2 ../../out/default/../../examples/fuzzers/rust/src/lib.rs:35 <>+0x107a03 |
| #9 0x000023c56778136c in LLVMFuzzerTestOneInput ../../out/default/../../examples/fuzzers/rust/src/lib.rs:33 <>+0x10836c |
| #10 0x000023c56772ab86 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) <>+0xb1b86 |
| #11 0x000023c567716ae5 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) ../recipe_cleanup/clangshYTOG/llvm_build_dir/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:296 <>+0x9dae5 |
| #12 0x000023c56771c535 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) <>+0xa3535 |
| #13 0x000023c5677457e3 in main ../recipe_cleanup/clangshYTOG/llvm_build_dir/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19 <>+0xcc7e3 |
| #14 0x000041fad9a9243b in start_main ./../../zircon/third_party/ulib/musl/src/env/__libc_start_main.c:112 <libc.so>+0x9343b |
| </pre> |
| |
| For unreproducible results, you can still examine symbolized log from the original fuzzer execution |
| for clues. |
| |
| ### Attach a debugger {#debug} |
| |
| You may also want to attach a debugger when reproducing fuzzer results. By default, libFuzzer on |
| Fuchsia creates a [debug exception channel][exception-channel] attached to the fuzzing thread in |
| order to detect and handle crashes during fuzzing. Only one process may do this per thread, so |
| debuggers are prevented from attaching. |
| |
| To prevent `libfuzzer` from creating a debug exception channel, set the `debug` fuzzer option to |
| true. |
| |
| For example, to use [zxdb] while reproducing a specific test case: |
| |
| <pre class="prettyprint devsite-disable-click-to-copy"> |
| <code class="devsite-terminal">ffx debug connect</code> |
| [zxdb] attach my-fuzzer |
| [zxdb] break LLVMFuzzerTestOneInput |
| </pre> |
| |
| Now, in a separate terminal, start the fuzzer with your test case: |
| |
| <pre class="prettyprint devsite-disable-click-to-copy"> |
| <code class="devsite-terminal">ffx fuzz shell</code> |
| fuzz >> attach fuchsia-pkg://fuchsia.com/my-fuzzers#meta/my-fuzzer.cm |
| fuzz >> set debug true |
| fuzz >> try <var>my_input_file</var> |
| </pre> |
| |
| ### File fuzzing bugs {#bug-filing} |
| |
| It may be tempting to immediately fix the bug related to the fuzzer result, especially if the bug is |
| obvious. No matter how trivial the bug is, please file a bug report! |
| |
| To file a bug, please use the [Fuzzing Bug template][fuzzing-bug-template]. This ensures you include |
| certain labels, such as `found-by-fuzzing`, `libfuzzer` and `Sec-TriageMe`. This in turn helps the |
| security team see where fuzzers are being used and stay aware of any critical issues they are |
| finding. |
| |
| Note: Due to their potential security implications, fuzzing bugs are not public until they have been |
| triaged and found to have no security impact or have been mitigated. |
| |
| Important: As with other potential security issues, bugs should be filed in the component of the |
| code under test, and _not_ in the `Security` component. |
| |
| If you encounter problems or shortcomings in the fuzzing framework _itself_, open bugs or |
| feature requests in the [`Security>Fuzzing` component][security-fuzzing]. |
| |
| As with all potential security issues, you do not need to wait for triage to begin fixing the bug! |
| Once fixed, reference the bug number in the commit message. |
| |
| ## Handle bugs from ClusterFuzz {#clusterfuzz-bugs} |
| |
| ClusterFuzz will file bugs automatically when it finds reproducible fuzzer results. If you are |
| assigned such a bug, look for the following: |
| |
| * The _Detailed Report_ will contain details about the result, including: |
| |
| * What type of result it is. |
| * Whether it has security implications. |
| * What revisions exhibited the behavior. |
| * What stack frames appear to uniquely identify the crash. |
| |
| * The _Reproducer Testcase_ will link to a fuzzer artifact. You can download this artifact and then |
| use it to reproduce the fuzzer result as described [above](#repro). |
| |
| When you submit a fix so that a fuzzer stops producing an artifact from the input, ClusterFuzz will |
| automatically close the bug. |
| |
| ## Bugs found by fuzzing {#found-by-fuzzing} |
| |
| Note: The bug tracker is currently only open to Googlers. |
| |
| {% dynamic if user.is_googler %} |
| |
| You can see bugs found in Fuchsia by fuzzing in [Monorail]. |
| |
| You can also see graphs of this information using the Fuchsia fuzzing bug [dashboard]. |
| |
| {% dynamic endif %} |
| |
| [clusterfuzz]: https://google.github.io/clusterfuzz/ |
| [dashboard]: https://goto.google.com/fuchsia-fuzzing-bugs |
| [exception-channel]: /docs/concepts/kernel/exceptions.md |
| [fuzzing-bug-template]: https://bugs.fuchsia.dev/p/fuchsia/issues/entry?template=Fuzzing+Bug |
| [monorail]: https://goto.google.com/fuchsia-found-by-fuzzing |
| [security-fuzzing]: https://bugs.fuchsia.dev/p/fuchsia/issues/list?q=component%3ASecurity%3EFuzzing&can=2 |
| [toy-example]: /examples/fuzzers/rust/src/lib.rs |
| [zxdb]: /docs/development/debugger |