Bazel Host Tests

Scope

This documents describes how to define, build and run Fuchsia host tests that exclusively exist in the Bazel graph.

These host tests must meet the following requirements:

These tests must be buildable exclusively with Bazel; i.e. they do not depend on artifacts generated by a prior Ninja invocation.
These tests only include binaries and runtime files that can run on a compatible host system that is NOT connected to any Fuchsia device or emulator.

Defining Fuchsia host tests in Bazel

Regular Bazel test rules (e.g. rustc_test, go_test) are not compatible with Fuchsia test runners. They will not be visible to fx test and will not be run by infra builders.

Hence, to define a Fuchsia-compatible host test in Bazel:

Use language-specific host test wrapper from //build/bazel/host_tests:host_<xxx>_test.bzl, such as host_rustc_test(), host_go_test(), etc.
These are designed to be drop-in replacements for regular Bazel test rules, with extra arguments like test_data, test_args, so Fuchsia test runners can invoke them with the right runtime environment and arguments.
Only if no language-specific wrappers are available for your test, use the generic host_test() wrapper from //build/bazel/host_tests:host_test.bzl. For example:
```
load("//build/bazel/host_tests:host_test.bzl", "host_test")
load("@rules_sh//sh:sh_binary.bzl", "sh_binary")

sh_binary(
    name = "test_bin",
    ...
)

host_test(
    name = "shell_test",
    binary = ":test_bin",
)
```
NOTE: ALWAYS prefer using language-specific wrappers if available, as they also implement extra Fuchsia-specific features, which are critical for parsing test results in infra.
Do NOT assume the execution location of a test binary (e.g. from out/default), as it is launched from a directory whose exact path is not predictable.
Bazel host tests MUST be hermetic and list all their runtime dependencies explicitly, just like GN host tests. Runtime dependencies can be provided through one of the mechanisms:
- As test_data dependencies to a host_test_data_files() or host_test_data_map() target (see //build/bazel/host_tests:host_test_data.bzl). These make the files available at runtime at fixed locations, relative to the test's execution directory, similar to the GN host_test_data() template.
- As data attribute on a host_xxx_test() target. In this case, the files will be added to the test's runfiles, and can be accessed using a runfiles library.
Use test_suite() instead of filegroup() to define groups of tests. Test suites can depend on tests and other test suites.
WARNING: Bazel skips tests from test_suites if they are incompatible with the current configuration. See Bazel test compatibility for more details.

Exporting Bazel host tests to `fx test` and infra builders

Defining the host_xxx_test() target alone will NOT make it visible to fx test and, more importantly, to infra builders.

To make a Bazel host test visible to fx test and infra builders:

Make the Bazel target reachable from //:bazel_host_test_suites in GN. This can be achieved in a few ways:
- Write a bazel_host_test_suite() target in GN that lists the Bazel labels of one or more Bazel host tests or suites.
- Add the Bazel test to a test_suite (in a BUILD.bazel) that is already listed in a bazel_host_test_suite() (in a BUILD.gn).
- In rare cases, add the Bazel test directly to default_host_tests in //build/bazel/host_tests/BUILD.bazel, if the test needs to be built for ALL build configurations. These tests should be small, fast to run, and critical to the build.
Ensure export_bazel_host_tests=true is set in args.gn. For example, when running fx set:
```
fx set ... --args export_bazel_host_tests=true
```

After these steps, an entry for the test will appear in out/default/tests.json. When in doubt, examine the content of this file after fx set or fx gen. The label field for Bazel tests begins with @.

During development, it is possible to build and run test targets once they are defined, before they are exported as described above. See the Running tests directly with Bazel section below.

Running Fuchsia Bazel host tests

Local testing

Running tests with `fx test`

NOTE: To run Bazel host tests with fx test, they MUST be exported by following the instructions in the Exporting Bazel host tests to fx test and infra builders section above.

After the tests are exported, use fx test --host <name_or_label> to build and run a Bazel host test locally, same as with GN-defined host tests.

The only difference is that the test will be built on demand by invoking Bazel directly, skipping Ninja entirely.

Note that when using fx test:

<name_or_label> will be subject to fuzzy matching, and fx test will report multiple candidates in case of ambiguity.
<name_or_label> cannot reference a Bazel test_suite() label, just like it cannot reference a GN group() one.

See https://fuchsia.dev/fuchsia-src/reference/testing/fx-test for more details on how to use fx test.

Running tests directly with `fx bazel test`

NOTE: Only use this during local developments, as it bypasses the mechanisms that infra builders use to run tests.

You can use fx bazel test --config=host <target_pattern> to build and run tests locally directly with Bazel. This does not require the extra plumbing to make tests visible to fx test as described in the previous section, and has the following advantages:

<target_pattern> is a set of Bazel target patterns that can match “all test targets in a given package”, in which case Bazel will skip non-test targets automatically.
<target_pattern> can match test_suite() labels which can be convenient to run a set of host tests repeatedly during development.
bazel test caches test results, and will only re-run the tests impacted by your latest changes since the last invocation.
(Use --nocache_test_results flag to disable this).
<target_pattern> can reference labels that are not visible in tests.json. This can be handy during development when adding new tests.

WARNING: Bazel skips tests matched by <target_pattern> if they are incompatible with the current configuration. See Bazel test compatibility for more details.

Pro Tip: If you have RBE enabled in your args.gn, use fx bazel test --config=remote --config=test <target_pattern> to run your host tests on remote test bots. This will be noticeably slower but is a good way to verify that your tests are truly hermetic before running them on infra.

Use fx bazel test --config=host //build/bazel/host_tests to run all reachable Bazel host tests locally in one go. This is useful when modifying build rules or macros that may affect many such targets.

Infra testing

Only host tests that are listed in tests.json can be launched on infra test bots. This applies equally to GN and Bazel host tests. So you MUST ensure they are visible in tests.json by following Exporting Bazel host tests to fx test and infra builders section above, otherwise they will NOT be continuously tested on infra.

There is no explicit distinction between GN and Bazel-defined host tests when running them on infra test runners, or when collecting results.

FAQ

Bazel test compatibility

If any test in a test_suite() (or any of its dependencies) is incompatible with the current configuration (e.g., using target_compatible_with), that test will be skipped during evaluation of the suite instead of failing the build.

The same is true when using patterns such as //... and :all on the command line.

For more information, see the handling of incompatible targets.

When running a test_suite() or using a target pattern, developers must verify that the test(s) of interest are reported as PASSED (and not SKIPPED).