# Handling results found through fuzzing

When [your fuzzer runs](run-a-fuzzer.md), it searches for inputs that crash the program or violate
checked conditions. When the fuzzer finds and reports such a test input, it is evidence of a bug
that needs to be resolved.

Typically, you might use `ffx fuzz` when first developing your fuzzer. This can often produce
results immediately. After a fuzzer has been submitted, it is run at scale by
[ClusterFuzz][clusterfuzz]{:.external} and any results it finds will be filed as bugs.

## Handle results from `ffx fuzz` {#ffx-fuzz-results}

When [running a fuzzer with `ffx fuzz`](run-a-fuzzer.md#run-on-device), output logs and results are
stored to the output directory. By default, this is the current working directory.  A different
location can be set passing the `--output` option to `attach`.

Crashes and other artifacts will have file name like
`{{ "<var>" }}type-of-finding{{ "</var>" }}.{{ "<var>" }}SHA2-hash-of-input{{ "</var>" }}`. The file
contents will be the input bytes themselves.

For example, examining a crash produced by a [toy example][toy-example] might look like the
following:

<pre class="prettyprint devsite-disable-click-to-copy">
<code class="devsite-terminal">ffx fuzz run -o results fuchsia.pkg://fuchsia.com/example-fuzzers#meta/toy_example_arbitrary.cm</code>
<code class="devsite-terminal">cd test_data/fuzzing/example-fuzzers/toy_example_arbitrary/latest</code>
<code class="devsite-terminal">hd results/crash-2fda3f03bb699c8a2151724b64b6e36c3b986aea</code>
00000000  2a 48 49 21 2a 00 08 00  00 00 2a 48 49 00 0a 66  |*HI!*.....*HI..f|
00000010  4a 33 00 0a f9                                    |J3...|
00000015
</pre>

### Reproduce a result {#repro}

You can execute the fuzzer with this input again using `ffx fuzz try`. For example:

<pre class="devsite-terminal devsite-disable-click-to-copy">
ffx fuzz try fuchsia.pkg://fuchsia.com/example-fuzzers#meta/toy_example_arbitrary.cm crash-2fda3f03bb699c8a2151724b64b6e36c3b986aea
</pre>

If the result is reproducible, this will produce a symbolized log including a stack trace. The top
of this stack trace is likely to be the error handling by libFuzzer and/or the sanitizer, and the
bottom will likely be the fuzzer engine itself.

For example, the relevant function where the `panic` occurred in the following stack trace is
`_toy_example_arbitrary_lib_rustc_static::toy_example`:

<pre class="prettyprint devsite-disable-click-to-copy">
#0    0x000023c56780a61e in _$LT$std..sys_common..backtrace.._print..DisplayBacktrace$u20$as$u20$core..fmt..Display$GT$::fmt::h510ae2e0fe71c88c <>+0x19161e
#1    0x000023c56783399c in core::fmt::write::hb61ef49191e76a74 <>+0x1ba99c
#2    0x000023c5678009b1 in std::io::Write::write_fmt::h41df81fb2b8460af <>+0x1879b1
#3    0x000023c56780eb92 in std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::h4e9a8e3c4f33b3f4 <>+0x195b92
#4    0x000023c56780e87c in std::panicking::default_hook::hd85edcd963c04eae <>+0x19587c
#5    0x000023c56780f271 in std::panicking::rust_panic_with_hook::h8960558cc7e69505 <>+0x196271
#6    0x000023c5677973d5 in std::panicking::begin_panic::h97c6d4cd722282c5 /b/s/w/ir/k/rust/src/libstd/panicking.rs:397 <>+0x11e3d5
#7    0x000023c56777f2d0 in _toy_example_arbitrary_lib_rustc_static::toy_example::h573322211ba71c22 ../../out/default/../../examples/fuzzers/rust/src/lib.rs:22 <>+0x1062d0
#8    0x000023c567780a03 in _toy_example_arbitrary_lib_rustc_static::_::toy_example_arbitrary::hc02c288d17b25ac2 ../../out/default/../../examples/fuzzers/rust/src/lib.rs:35 <>+0x107a03
#9    0x000023c56778136c in LLVMFuzzerTestOneInput ../../out/default/../../examples/fuzzers/rust/src/lib.rs:33 <>+0x10836c
#10   0x000023c56772ab86 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) <>+0xb1b86
#11   0x000023c567716ae5 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) ../recipe_cleanup/clangshYTOG/llvm_build_dir/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:296 <>+0x9dae5
#12   0x000023c56771c535 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) <>+0xa3535
#13   0x000023c5677457e3 in main ../recipe_cleanup/clangshYTOG/llvm_build_dir/tools/clang/stage2-bins/runtimes/runtimes-x86_64-unknown-fuchsia-bins/compiler-rt/lib/fuzzer/FuzzerMain.cpp:19 <>+0xcc7e3
#14   0x000041fad9a9243b in start_main ./../../zircon/third_party/ulib/musl/src/env/__libc_start_main.c:112 <libc.so>+0x9343b
</pre>

For unreproducible results, you can still examine symbolized log from the original fuzzer execution
for clues.

### Attach a debugger {#debug}

You may also want to attach a debugger when reproducing fuzzer results. By default, libFuzzer on
Fuchsia creates a [debug exception channel][exception-channel] attached to the fuzzing thread in
order to detect and handle crashes during fuzzing. Only one process may do this per thread, so
debuggers are prevented from attaching.

To prevent `libfuzzer` from creating a debug exception channel, set the `debug` fuzzer option to
true.

For example, to use [zxdb] while reproducing a specific test case:

<pre class="prettyprint devsite-disable-click-to-copy">
<code class="devsite-terminal">ffx debug connect</code>
[zxdb] attach my-fuzzer
[zxdb] break LLVMFuzzerTestOneInput
</pre>

Now, in a separate terminal, start the fuzzer with your test case:

<pre class="prettyprint devsite-disable-click-to-copy">
<code class="devsite-terminal">ffx fuzz shell</code>
fuzz >> attach fuchsia-pkg://fuchsia.com/my-fuzzers#meta/my-fuzzer.cm
fuzz >> set debug true
fuzz >> try <var>my_input_file</var>
</pre>

### File fuzzing bugs {#bug-filing}

It may be tempting to immediately fix the bug related to the fuzzer result, especially if the bug is
obvious. No matter how trivial the bug is, please file a bug report!

To file a bug, please use the [Fuzzing Bug template][fuzzing-bug-template]. This ensures you include
certain labels, such as `found-by-fuzzing`, `libfuzzer` and `Sec-TriageMe`. This in turn helps the
security team see where fuzzers are being used and stay aware of any critical issues they are
finding.

Note: Due to their potential security implications, fuzzing bugs are not public until they have been
triaged and found to have no security impact or have been mitigated.

Important: As with other potential security issues, bugs should be filed in the component of the
code under test, and _not_ in the `Security` component.

If you encounter problems or shortcomings in the fuzzing framework _itself_, open bugs or
feature requests in the [`Security>Fuzzing` component][security-fuzzing].

As with all potential security issues, you do not need to wait for triage to begin fixing the bug!
Once fixed, reference the bug number in the commit message.

## Handle bugs from ClusterFuzz {#clusterfuzz-bugs}

ClusterFuzz will file bugs automatically when it finds reproducible fuzzer results. If you are
assigned such a bug, look for the following:

 * The _Detailed Report_ will contain details about the result, including:

   * What type of result it is.
   * Whether it has security implications.
   * What revisions exhibited the behavior.
   * What stack frames appear to uniquely identify the crash.

 * The _Reproducer Testcase_ will link to a fuzzer artifact. You can download this artifact and then
   use it to reproduce the fuzzer result as described [above](#repro).

When you submit a fix so that a fuzzer stops producing an artifact from the input, ClusterFuzz will
automatically close the bug.

## Bugs found by fuzzing {#found-by-fuzzing}

Note: The bug tracker is currently only open to Googlers.

{% dynamic if user.is_googler %}

You can see bugs found in Fuchsia by fuzzing in [Monorail].

You can also see graphs of this information using the Fuchsia fuzzing bug [dashboard].

{% dynamic endif %}

[clusterfuzz]: https://google.github.io/clusterfuzz/
[dashboard]: https://goto.google.com/fuchsia-fuzzing-bugs
[exception-channel]: /docs/concepts/kernel/exceptions.md
[fuzzing-bug-template]: https://bugs.fuchsia.dev/p/fuchsia/issues/entry?template=Fuzzing+Bug
[monorail]: https://goto.google.com/fuchsia-found-by-fuzzing
[security-fuzzing]: https://bugs.fuchsia.dev/p/fuchsia/issues/list?q=component%3ASecurity%3EFuzzing&can=2
[toy-example]: /examples/fuzzers/rust/src/lib.rs
[zxdb]: /docs/development/debugger