docs/development/build/bazel_concepts/tests.md

# Bazel Tests

## Scope

This document describes how to define, build and run tests that exclusively
exist in the Bazel graph.

Fuchsia supports Bazel **host tests** that meet the following requirements:

- These tests must be buildable exclusively with Bazel; i.e. they do not depend
  on artifacts generated by a prior Ninja invocation.

- These tests only include binaries and runtime files that can run on a
  compatible host system that is **NOT** connected to any Fuchsia device or
  emulator.

Fuchsia also supports Bazel **device tests**, declared with `fx_test()` and
run on a Fuchsia device or emulator. These have the same requirement of
being buildable exclusively with Bazel, and additionally:

- Aside from `environments` (which defaults to the build's default test
  environments), `build_only`, `max_log_severity` (which defaults to `WARN`),
  and `test_type` (for running in a non-hermetic test realm such as
  `"starnix"` or `"system"`), test spec fields (timeouts, etc.) cannot be
  customized per test yet.

[fxbug.dev/564574581](https://fxbug.dev/564574581) tracks the remaining work for
target tests in Bazel.

## Defining Fuchsia host tests in Bazel

Regular Bazel test rules (e.g. `rustc_test`, `go_test`) are not compatible with
Fuchsia test runners. They will not be visible to `fx test` and will not
be run by infra builders.

Hence, to define a Fuchsia-compatible host test in Bazel:

- Use language-specific host test wrapper from
  `//build/bazel/rules/host_tests:host_<xxx>_test.bzl`, such as `host_rustc_test()`,
  `host_go_test()`, etc.

  These are designed to be drop-in replacements for regular Bazel test rules,
  with extra arguments like `test_data`, `test_args`, so Fuchsia test runners
  can invoke them with the right runtime environment and arguments.

- Only if no language-specific wrappers are available for your test, use
  the generic `host_test()` wrapper from
  `//build/bazel/rules/host_tests:host_test.bzl`. For example:

  ```starlark
  load("//build/bazel/rules/host_tests:host_test.bzl", "host_test")
  load("@rules_sh//sh:sh_binary.bzl", "sh_binary")

  sh_binary(
      name = "test_bin",
      ...
  )

  host_test(
      name = "shell_test",
      binary = ":test_bin",
  )
  ```

  Note: **ALWAYS** prefer using language-specific wrappers if available, as they
  also implement extra Fuchsia-specific features, which are critical for parsing
  test results in infra.

- Do **NOT** assume the execution location of a test binary (e.g. from
  `out/default`), as it is launched from a directory whose exact path is not
  predictable.

  Bazel host tests **MUST** be hermetic and list all their runtime dependencies
  explicitly, just like GN host tests. Runtime dependencies can be provided
  through one of the mechanisms:

  - As `test_data` dependencies to a `host_test_data_files()` or
    `host_test_data_map()` target (see `//build/bazel/rules/host_tests:host_test_data.bzl`).
    These make the files available at runtime at fixed locations, relative to the
    test's execution directory, similar to the GN `host_test_data()` template.

  - As `data` attribute on a `host_xxx_test()` target.
    In this case, the files will be added to the test's runfiles, and can be
    accessed using a [runfiles library][bazel_runfiles].

  [bazel_runfiles]: /build/bazel/BAZEL_RUNFILES.md

- Use `test_suite()` instead of `filegroup()` to define groups of tests.
  Test suites can depend on tests and other test suites.

  Warning: Bazel skips tests from `test_suite`s if they are incompatible
  with the current configuration. See
  [Bazel test compatibility](#bazel-test-compatibility) for more details.

## Exporting Bazel host tests to `fx test` and infra builders

Defining the `host_xxx_test()` target alone will **NOT** make it visible to
`fx test` and, more importantly, to infra builders.

To make a Bazel host test visible to `fx test` and infra builders, write a
`bazel_test_suite` target in GN that lists the Bazel labels of one or more Bazel
host tests or suites, and add that target to a "tests" group in GN. For example:

`//src/foo/bar/BUILD.bazel`:

```bazel
load("//build/bazel/platforms:constraints.bzl", "HOST_OS_CONSTRAINTS")

test_suite(
  name = "host_tests",
  host_tests = [
    "//src/foo/bar/lib:lib_tests",
    "//src/foo/bar/util:util_tests",
  ]
  target_compatible_with = HOST_OS_CONSTRAINTS,
)
```

`//src/foo/bar/BUILD.gn`:

```gn
import("//build/bazel/bazel_test_suite.gni")

bazel_test_suite("bazel_tests") {
  host_tests = [
    # This references the target in BUILD.bazel.
    "//src/foo/bar:host_tests",
  ]
}

group("tests") {
  deps = [
    ":bazel_tests",
    ...
  ]
}
```

This will cause entries for the Bazel tests to appear in
`out/default/tests.json`. When in doubt, examine the content of this file after
`fx set` or `fx gen`. The `label` field for Bazel tests begins with `@`.

During development, it is possible to build and run test targets once they are
defined, before they are exported as described above. See the
[Running tests directly with Bazel](#running-tests-directly-with-bazel) section
below.

Note: Bazel tests do not show up in `tests.json` if the `export_bazel_tests`
GN argument is overridden to `false`.

### Best practices

It's recommended to avoid deep nesting of Bazel `test_suite` targets. Instead,
use just one `test_suite` Bazel target and one corresponding `bazel_test_suite`
GN target per source code sub-area (e.g. host tool or Fuchsia component).

## Defining Fuchsia device tests in Bazel

Device tests are components in a Fuchsia package, exactly like in the GN build.
Declare the test component with `fx_test_component()`, package it with
`fx_package()`, and expose it as a test with `fx_test()`:

`//src/foo/bar/BUILD.bazel`:

```bazel
load("//build/bazel/rules/packages:fx_packaged_binary.bzl", "fx_packaged_binary")
load(
    "//build/bazel/rules/components:fx_component.bzl",
    "fx_component_manifest",
    "fx_test_component",
)
load("//build/bazel/rules/packages:fx_package.bzl", "fx_package")
load("//build/bazel/rules/testing:fx_test.bzl", "fx_test")

fx_packaged_binary(
    name = "bar_test_packaged_bin",
    testonly = True,
    binary = ":bar_test_bin",
    binary_name = "bar_test",
)

fx_component_manifest(
    name = "bar_test_manifest",
    testonly = True,
    component_name = "bar_test",
    includes = ["//src/sys/test_runners/gtest:default.shard.cml"],
    manifest = "meta/bar_test.cml",
)

fx_test_component(
    name = "bar_test_component",
    compiled_manifest = ":bar_test_manifest",
    component_name = "bar_test",
    deps = [":bar_test_packaged_bin"],
)

fx_package(
    name = "bar_tests_package",
    package_name = "bar_tests",
    test_components = [":bar_test_component"],
)

fx_test(
    name = "bar_tests",
    package = ":bar_tests_package",
    # Optional: defaults to the build's default test environments when omitted.
    environments = [
        "//build/testing/environments:aemu_env",
        "//build/testing/environments:nuc11_env",
    ],
)
```

Every component in `test_components` becomes a separate test, whose name is its
package URL, e.g. `fuchsia-pkg://fuchsia.com/bar_tests#meta/bar_test.cm`.

To customize which target environments a test runs in, pass `environments` to
`fx_test()` as a list of environment targets. `//build/testing/environments`
has one target for each individual environment in
`//build/testing/environments.gni`, with the same name (e.g. `:emu_env` or
`:nuc11_env`). When `environments` is omitted, the test runs in the build's
default test environments (GN's `basic_envs`), exactly like a GN test that
doesn't set `environments`. To build a test without running it, set
`build_only = True` instead.

To add infra settings, such as tags, to one of these environments, use
`fx_test_environment_variant()`:

```bazel
load(
    "//build/bazel/rules/testing:fx_test_environment.bzl",
    "fx_test_environment_variant",
)

fx_test_environment_variant(
    name = "nuc11_isolated_env",
    base = "//build/testing/environments:nuc11_env",
    env_tags = ["e2e-isolated"],
)
```

New environments should be rare; most tests should use one of the existing
environments in `//build/testing/environments`. When a new environment is
needed, add it to both `//build/testing/environments.gni` and
`//build/testing/environments/BUILD.bazel`, rather than defining it with
`fx_test_environment()` next to a test. Environments' Swarming dimensions are
an interface with infra, so they're kept in one place.

To run a test in a non-hermetic test realm (such as `"starnix"`, `"system"`, or
`"vulkan"`), pass `test_type` to `fx_test_component()` (or to `fx_test()` to
apply to all test components in the package). Valid `test_type` values match
`type_moniker_map` in `//build/components/fuchsia_test_component.gni`.

Note: components marked as tests **MUST** be listed in `test_components`, and
non-test components **MUST** be listed in `components`. Packaging fails
otherwise.

## Exporting Bazel device tests to `fx test` and infra builders

Like host tests, device tests must be listed in a GN `bazel_test_suite()`
target, using the `target_tests` parameter instead of `host_tests`:

`//src/foo/bar/BUILD.gn`:

```gn
import("//build/bazel/bazel_test_suite.gni")

bazel_test_suite("bazel_tests") {
  target_tests = [
    # This references the target in BUILD.bazel.
    "//src/foo/bar:bar_tests",
  ]
}
```

Device tests are tracked separately from host tests because they are queried
and built in the `fuchsia_platform` Bazel configuration, rather than the host
one.

In addition to the `tests.json` entries, this writes
`${root_build_dir}/bazel_test_packages.list`, which enumerates the package
manifests of all Bazel test packages in the build graph. Full `fx build`
invocations merge this list into `all_package_manifests.list` and publish its
packages into `amber-files`, while `fx test` also merges it directly when
building specific test targets on demand.

## Running Fuchsia Bazel device tests

Once exported, run a Bazel device test the same way as a GN one, by package URL
or by package name:

```posix-terminal
fx test fuchsia-pkg://fuchsia.com/bar_tests#meta/bar_test.cm
```

`fx test` builds the test by invoking Bazel directly (as
`fx build --fuchsia_platform {{ '<var>' }}bazel_label{{ '</var>' }}`), then
publishes the package manifests listed in `bazel_test_packages.list` before
running the test. A running package server and a connected device or emulator
are required, exactly as for GN device tests.

## Running Fuchsia Bazel host tests

### Local testing

#### Running tests with `fx test`

To run Bazel host tests with `fx test`, they **MUST** be exported by following
the instructions in the
[Exporting Bazel host tests to `fx test` and infra builders](#exporting-bazel-host-tests-to-fx-test-and-infra-builders)
section above.

After the tests are exported, use `fx test --host <name_or_label>` to build and
run a Bazel host test locally, same as with GN-defined host tests.

The only difference is that the test will be built on demand by invoking Bazel
directly, skipping Ninja entirely.

Note that when using `fx test`:

- `<name_or_label>` will be subject to fuzzy matching, and `fx test` will report
  multiple candidates in case of ambiguity.

- `<name_or_label>` cannot reference a Bazel `test_suite()` label, just like it
  cannot reference a GN `group()` one.

See the [User guide for fx test](/docs/reference/testing/fx-test.md) for more
details about how to use `fx test`.

#### Running tests directly with `fx bazel test` {#running-tests-directly-with-bazel}

Caution: Only use this during local development, as it bypasses the mechanisms
that infra builders use to run tests.

You can use `fx bazel test --config=host <target_pattern>` to build and run
tests locally directly with Bazel. This does not require the extra plumbing to
make tests visible to `fx test` as described in the previous section, and has
the following advantages:

- `<target_pattern>` is a set of Bazel target patterns that can match "all test
  targets in a given package", in which case Bazel will skip non-test targets
  automatically.

- `<target_pattern>` can match `test_suite()` labels which can be convenient to
  run a set of host tests repeatedly during development.

- `bazel test` caches test results, and will only re-run the tests impacted by
  your latest changes since the last invocation.

  (Use `--nocache_test_results` flag to disable this).

- `<target_pattern>` can reference labels that are not visible in `tests.json`.
  This can be handy during development when adding new tests.

Warning: Bazel skips tests matched by `<target_pattern>` if they are
incompatible with the current configuration. See
[Bazel test compatibility](#bazel-test-compatibility) for more details.

Tip: If you have RBE enabled in your `args.gn`, use `fx bazel test
--config=remote --config=test <target_pattern>` to run your host tests on remote
test bots. This is noticeably slower but is a good way to verify that your
tests are truly hermetic before running them on infra.

### Infra testing

Only host tests that are listed in `tests.json` can be launched on infra test
bots. This applies equally to GN and Bazel host tests. So you **MUST** ensure
they are visible in `tests.json` by following the
[Exporting Bazel host tests to `fx test` and infra builders](#exporting-bazel-host-tests-to-fx-test-and-infra-builders)
section above, otherwise they will **NOT** be continuously tested on infra.

Infra builds Bazel tests using `fint`, which directly calls `bazel build` after
running `ninja`, to build all Bazel tests that are listed in `bazel_test_suite`
GN targets.

There is no explicit distinction between GN and Bazel-defined host tests when
running them on infra test runners, or when collecting results. In particular,
infra does not run tests using `bazel test`. Instead, it takes the binaries
produced by `bazel build` and runs them directly, just like it does with host
tests from ninja.

## Debug symbol propagation

Host test binaries compiled with debug symbols must have their unstripped ELF
binaries registered so that tooling can symbolize backtraces and correlate LLVM
code coverage profiles (`-profile-correlate=binary`).

For general background on how Fuchsia manages Bazel debug symbols, see
[Technical Note on Debug Symbol Generation](/build/bazel/debug_symbols/README.md).

### Authoring host test rules with debug symbols

The generic `host_test()` rule exposes the `unstripped_binary` field in its
`FuchsiaHostTestInfo` provider:

- If `unstripped_binary` is not explicitly set, `host_test()` inspects the
  underlying `binary` attribute:
  - If `binary` provides `DebugPackageInfo` (C++ targets), it uses
    `DebugPackageInfo.unstripped_file`.
  - If `binary` provides `CrateInfo` (Rust targets), it uses
    `CrateInfo.output`.

- If the test uses a wrapper script launcher (such as `rust_test_parser` in
  `host_rustc_test()`), the macro **MUST** explicitly forward the original
  unstripped binary target via `unstripped_binary`:

  ```starlark
  host_test(
      name = name,
      binary = wrapper_script,
      unstripped_binary = ":" + binary_name,
      ...
  )
  ```

### Propagation to infra and debuginfod

When Bazel host tests are exported to the GN build:

1. **Test Query**: `bazel_tests_utils.py` runs a `bazel cquery` using
   `//build/bazel/starlark/FuchsiaHostTestInfo.cquery`, extracting the
   execroot path of `unstripped_binary` for all registered host tests.

2. **Host Test Manifest**: The paths are normalized relative to the Ninja build
   directory and written to
   `${root_build_dir}/bazel_host_tests.debug_symbols.json`.

3. **GN Build API Integration**: `//:bazel_host_test_suites` in `BUILD.gn`
   attaches `debug_symbol_manifests` metadata pointing to
   `bazel_host_tests.debug_symbols.json`, which is consumed by the
   `build_api_module("debug_symbols")` target.

4. **Artifactory Upload**: In CI/CQ builds, Artifactory processes
   `debug_symbols.json` and uploads the unstripped ELF binaries to cloud
   storage / debuginfod servers, allowing tools like `covargs` to resolve
   Build IDs.

### Local development and `fx coverage`

When running coverage locally at desk, uncommitted binaries have newly generated
Build IDs that are not present on debuginfod servers.

To support local symbol correlation, `fx coverage` runs:

```bash
python3 ${FUCHSIA_DIR}/build/bazel/scripts/copy_bazel_debug_symbols.py ${FUCHSIA_BUILD_DIR}
```

This script reads `bazel_host_tests.debug_symbols.json`, computes the GNU Build
ID for each ELF binary, and copies or symlinks the unstripped binary into
`${root_build_dir}/.build-id/xx/yyyyyyyy.debug`. Tooling such as `ffx coverage`
and `llvm-profdata` then resolves these binaries via `.symbol-index.json`.

### FAQ

#### Bazel test compatibility

If any test in a `test_suite()` (or any of its dependencies) is incompatible
with the current configuration (e.g., using `target_compatible_with`), that test
will be skipped during evaluation of the suite instead of failing the build.

The same is true when using patterns such as `//...` and `:all` on the command line.

For more information, see the [handling of incompatible targets][bazel_target_compatible_with].

When running a `test_suite()` or using a target pattern, developers must verify that
the test(s) of interest are reported as `PASSED` (and not `SKIPPED`).

[bazel_target_compatible_with]: https://bazel.build/reference/be/common-definitions#common.target_compatible_with