Sanitizers are tools for detecting certain classes of bugs in code. The operating principle varies, though sanitizers commonly (but not always) rely on some form of compile-time instrumentation to change code in ways that expose bugs at runtime. Fuchsia uses a variety of sanitizers to discover and diagnose dangerous bugs that are otherwise difficult to find.
Sanitizers are enabled by build-time flags. Sanitizer builds are continuously exercised on Fuchsia's Continuous Integration (CI) and Commit Queue (CQ), and serve Fuchsia C/C++ and Rust developers.
Developers typically benefit from sanitizers with no special action required, and only need to pay attention to sanitizers when they detect a bug. However certain limitations apply. Continue reading to learn what sanitizers are supported and how to use them.
Fuchsia currently supports the following sanitizers:
The following sanitizer behavior is available in the Zircon kernel:
pmm_checker) detects use-after-free bugs and stray DMAs.The following C/C++ compilation options are added by default to detect or prevent bugs at runtime:
-ftrivial-auto-var-init=pattern (see RFC) initializes automatic variables to a non-zero pattern to expose bugs related to reads from uninitialized memory.Lastly, Fuchsia uses libFuzzer{:.external} and syzkaller{: .external} to perform coverage-directed fuzz testing. Fuzzers are similar to sanitizers in that they attempt to expose bugs in the code at runtime, and they are usually used in conjunction. Fuzzers are different from sanitizers in that fuzzers attempt to force the execution of production code into paths that may expose a bug.
Sanitizers are currently supported in local builds and in CI/CQ under the following configurations:
bringup.x64bringup.arm64core.x64core.arm64zbi_tests.x64zbi_tests.arm64In addition, sanitizers apply to host tools.
Tests for all of the above are exercised on CI/CQ on qemu{:.external} and Intel NUC. Other platforms are not tested with sanitizers due to resources and capacity issues, but you may test on these platforms locally using the build workflow below.
Additional tryjobs for configurations defined under //vendor may be shown in Gerrit and in CI consoles for certain signed-in users. Look for configurations with -asan in their name.
The sanitizers listed above are applied to C/C++ code. In addition, LSan is applied to Rust code for detecting Rust memory leaks{:.external}.
To reproduce a sanitizer build, specify the sanitizer variants:
fx set {{ '<var label="product">product</var>' }} --variant asan-ubsan --variant host_asan-ubsan
Alternatively you may select to only instrument certain binaries:
fx set {{ '<var label="product">product</var>' }} --variant asan-ubsan/{{ '<var>executable_name</var>' }}
The selective instrumentation workflow is useful for testing locally on hardware where a fully instrumented build does not fit on the device.
Specifically to detect use-after-free bugs in kernel code you will need to enable the kernel PMM checker.
When compiling with the Fuchsia toolchain it is sufficient to pass the -fsanitize= flag to indicate which sanitizers to use. See the compiler documentation{:.external}.
When creating a Fuchsia package with instrumented components, you need to ensure that your package contains all runtime dependencies including the sanitizer runtime, which is distributed as part of the Clang toolchain, and instrumented C library, which is distributed as part of the Fuchsia SDK under sysroot.
Test as you normally would, in your local workflow or on a CQ builder which has sanitizers enabled (the tryjob has asan in the name). If a sanitizer detects an issue then messages will be printed to the logs containing one of the following strings:
ERROR: AddressSanitizerERROR: LeakSanitizerSUMMARY: UndefinedBehaviorSanitizerWARNING: ThreadSanitizerFollowing these messages you will find stack traces that identify the nature of the problem and point to the root cause. You can find these messages in fx log.
Note that the test that triggered the sanitizer may still appear as passing. Sanitizer issues don't manifest as test failures.
Issues detected by sanitizers typically have similar root causes. You may be able to find references for prior work by searching Fuchsia bugs for a bug with some of the same keywords that you're seeing in the sanitizer output.
See also: UBSan issues on Open Projects.
#[should_panic]Fuchsia‘s Rust builds abort on panic!. This significantly reduces binary size. An unfortunate side effect is that tests that use the #[should_panic] attribute may falsely detect memory leaks. These tests emit an expected panic and then exit without unwinding, which means they don’t free their heap allocations. To LeakSanitizer this is indistinguishable from a real memory leak.
If this issue affects your test then you can disable it in sanitizer builds by following this example.
See: Issue 88496: Rust tests that should_panic trigger leaksanitizer
Sanitizers expose bugs at runtime. Unless your code runs, such as within a test or generally on Fuchsia in such a way that‘s exercised by CI/CQ, sanitizers won’t be able to expose bugs in your code.
The best way to ensure sanitizer coverage for your code is to ensure test coverage under the same configuration. Consult the guide on test coverage.
Sanitizers may be suppressed for certain build targets. Most commonly this is used for issues that predate the introduction of sanitizer support, especially for issues in third party code that the Fuchsia project doesn't own.
Suppressed sanitizers should be considered tech debt, as they not only hide old bugs but keep you from discovering new bugs as they're introduced to your code. Ideally new suppressions should not be added, and existing suppressions should be removed and the underlying bugs fixed.
Suppressing sanitizers may be done by editing the BUILD.gn file that defines your executable target as follows:
executable("please_fix_the_bugs") { ... # TODO(fxbug.dev/12345): delete the below and fix the memory bug. deps += [ "//build/config/sanitizers:suppress-asan-stack-use-after-return" ] # TODO(fxbug.dev/12345): delete the below and fix the memory bug. deps += [ "//build/config/sanitizers:suppress-asan-container-overflow" ] # TODO(fxbug.dev/12345): delete the below and fix the memory leak. deps += [ "//build/config/sanitizers:suppress-lsan.DO-NOT-USE-THIS" ] # TODO(fxbug.dev/12345): delete the below and fix undefined behavior. configs += [ "//build/config:temporarily_disable_ubsan_do_not_use" ] }
The examples above demonstrate suppressing all sanitizers. However you should at most suppress sanitizers that are causing failures. Please track suppressions by filing a bug and referencing it in the comment as shown above.
Another common approach for disabling sanitizers works as follows:
executable("too_slow_when_built_with_asan") { ... exclude_toolchain_tags = [ "asan" ] }
Both examples above suppress at the granularity of an entire executable. For finer-grained suppressions you may detect the presence of sanitizers in code. This is useful for instance for suppressing sanitizers in a particular test case, but not more broadly. For instance this is used by tests that intentionally introduce memory errors and test the sanitizer runtime itself.
For C/C++ see:
__has_feature(address_sanitizer){:.external}__attribute__((no_sanitize("address"))){:.external}For Rust, you can follow this pattern:
#[cfg(test)] mod tests { #[test] // TODO(fxbug.dev/12345): delete the below and fix the leak #[cfg_attr(feature = "variant_asan", ignore)] fn test_that_leaks() { // ... } }
Sanitizer errors may be flaky if the code under test's behavior is non-deterministic. For instance a memory leak may only happen under certain race conditions. If sanitizer errors appear flaky, consult the guide on testing for flakiness in CQ.
When encountering a sanitizer issue, file a bug containing all the troubleshooting information that's available.
Example: Issue 73214: ASAN use-after-scope in blobfs
The bug report contains:
Ongoing work:
Areas for future work:
unsafe {} code blocks or across FFI{:.external} calls, or detecting undefined behavior bugs.See also: sanitizers in the 2021 roadmap.