docs/reference/testing/tests-json-format.md

# Fuchsia tests.json file format

Fuchsia's build system exports structured information about all
tests included in a build in a `tests.json` file in the build output
directory (e.g. `$FUCHSIA_DIR/out/default/tests.json`).

Fuchsia's continuous integration infrastructure uses `tests.json`
to determine which tests to run, and `fx test` uses it to determine
which tests may be run locally. `tests.json` and its schema are
implementation details of the platform build and test and are not
part of the SDK.

This document describes the file's contents and how the file is generated.

## Generating tests.json

The `tests.json` file is generated immediately after gn-gen in
[regenerator.py](/build/build_tests_json.py).

The fields for `tests.json` entries are collected using the [GN
metadata
mechanism](https://gn.googlesource.com/gn/+/main/docs/reference.md#var_metadata).
`tests.json` entries are registered from GN files by using the
[`test_spec` GN template](/build/testing/test_spec.gni).

`tests.json` is automatically generated by `fx set`. The `tests.json`
format is limited by what can readily be generated from GN using
GN's relatively limited metadata mechanism. To get around these
limitations, the Fuchsia build also produces file called `test-list.json`,
which is generated by post-processing `tests.json` as described below.

## tests.json consumers

Various tools and scripts ingest the `tests.json` file since it is
the canonical list of available tests for a given Fuchsia build.
This section surveys the most notable consumers of the format.

### test_list_tool

The [test_list_tool](/tools/test_list_tool/) processes `tests.json`.
Using the canonical list of tests, it uses the contained package
manifest information to load the build packages and annotate each
test with information about how it should be executed and tags for
additional categorization. For example, it can determine if a test
component runs hermetically, and tags the test with `"hermetic":
true`.

The test_list_tool outputs this information to a `test-list.json`
file adjacent to `tests.json`, and it is run as one of the last
build steps after all packages are built. It uses the same unique
`name` key as `tests.json` so that entries can be matched.

### fx search-tests

The `fx search-tests` script provides fuzzy matching on test names
to help developers find which tests are available to run. It uses
the values in `tests.json` as keys for fuzzy matching, in addition
to searching for non-included tests through the source tree.

### fx test

The `fx test` command is the developer-facing entrypoint to test
execution in the Fuchsia source tree. It uses `tests.json` to
determine the canonical set of tests, and then finds associated
data in `test-list.json` and `test_components.json` to determine
how a test should be executed.

### botanist / testsharder

In Infra, [testsharder](/tools/integration/testsharder/) depends on
`tests.json` to shard test entries based on environment dimensions.
The output of that process is another file containing fields
derived from `tests.json` which is then passed to [botanist](/tools/botanist/)
on the bots where tests run.

## tests.json structure

`tests.json` is a JSON file consisting of a single array. Each
element of the array corresponds to a single Test Target entry in
the build.

Test Target entries have two top-level fields, `environments` and
`test` corresponding to Test Environments and Test Info respectively.

### Fields

#### environments field

The `environments` field is an array of Test Environment entries.
Each Test Environment entry specifies a single environment the test
may run in, and this field is used by testsharder to shard tests between
device configurations.

Each Test Environment entry has a single field `dimensions` with
any number of arbitrary dimensions. Typically, however, there are
two sets of fields that are most common:

- `device_type` - Specifies the required device type for the test.
For example:

  ```json
  "environments": [
      {
          "dimensions": {
              "device_type": "QEMU"
          }
      }
  ]
  ```

  This denotes a test that can run on the QEMU device configuration.
  **Note:** This does not mean that the test runs on *any* QEMU
  device, rather it means that the test runs on a QEMU instance
  based on the configuration used by `botanist`.

- `cpu` and `os` - Specifies the required CPU architecture and
operating system for a host test. For example:

  ```json
  "environments": [
      {
          "dimensions": {
              "cpu": "x64",
              "os": "Linux"
          }
      }
  ]
  ```

  This specifies a test that can run on an x64 Linux system.

- `tags` - Specifies other properties for sharding. For example:

  ```json
  "environments": [
    {
      "tags": [ "e2e-isolated" ]
    }
  ]
  ```

  This specifies a test with tag "e2e-isolated".

  Tests with an environment tag will not be run in continuous
  integration builds unless the build configuration specifically
  opts in to running tests with that tag, in which case *only* tests
  with that tag will run.

#### test field

The `test` field provides information about how to categorize, run,
and configure the test. Test executors like `fx test` and `botanist`
use this data to determine the correct binary and command line to
execute the test. It also contains information that is useful for
other tooling integrations, for instance, to rebuild test targets
between test invocations.

There are generally two types of test: Host and Device. Host tests
refer to a binary path on the host system that is run to execute
the test. Device tests refer to a component URL that is run on the
target system using Test Manager.

Some host tests require a Fuchsia device and some do not.  An
*end-to-end test* (E2E) has an `os` that is not "fuchsia," but still
has a `device_type` specified in its `environments`.

The `test` object contains the following fields, split by type of test:

**All tests**

- `name` - Name for the test. This value is unique within `tests.json`.
Typically contains the test component URL for device tests and the
path to the test binary relative to the output directory for host
tests.
- `cpu` - The CPU architecture this test runs on. Redundant with
the `environments` entry for host tests, but the only source of
this information for device tests.
- `os` - The OS this test runs on. Redundant with the `environments`
entry for host tests, but the only source of this information for
device tests. This field is used by `botanist` to determine if a
test should be run as a subprocess on the host machine or run over
SSH to a target device.
- `label` - The GN label that produced this test entry. Used to
know which targets to rebuild when executing tests from `fx test`,
but only for host tests for which this label actually outputs the
real test binary. Other types of tests use different label fields
such as `package_label`.
- `source_label` - The build-system agnostic (either GN or Bazel) label
corresponding to the test. Used to determine the location of the test's
source code for determining ownership and other metadata.

**Host tests only**

- `path` - The path to the test binary, relative to the output
directory. Present only for host tests.
- `runtime_deps` - The path (relative to the output directory) to
a JSON file containing a list of file paths (relative to the output
directory) to files that should be put in a place known to a host
test for use at runtime. The format of the file is a JSON array
of file path strings.
- `list_cases_argument` - (Optional) The command line argument used to list
individual test cases within the test suite. This is used by `fx test --list`
to discover specific test cases for reporting and selection. Currently, this
is supported for **host tests only**, specifically for Mobly tests and Python
unit tests.

**Device tests only**

- `has_generated_manifest` - True if this is a test component whose
manifest was generated by a GN rule, omitted if the test is a host
test, false otherwise. Used for test categorization and tracking
of tests using custom manifests.
- `build_rule` - Generated by the build for further test categorization.
For example, it is meaningful to know if a test was was created by
the `fuchsia_unittest_package` or the `fuchsia_test_package` rule.
- `log_settings` - A dict that may contain these fields:
  - `max_severity` - This field overrides the default maximum log
  severity for tests. By default tests that emit an ERROR log will
  fail, but this field instructs test executors to run in a mode
  where a different level of logs causes failure.
  - `min_severity` - Used to instruct the test components to emit
  logs at this level rather than the default. By default tests
  choose their own minimum log severity.
- `package_label` - The GN label that produced the package this
test is contained in. Only present for device tests. Used by `fx
test` to rebuild the packages containing tests before they are run.
- `component_label` - The GN label that produces the test component
for this test. Used for fuzzy matching test names when no matching
tests are found in `fx search-tests` (in addition to all other label
fields). Only present for device tests.
- `package_manifests` - List of paths (relative to output directory)
for each package manifest involved in constructing the test package.
Used to determine which packages to republish after building tests
in `fx test`. Only present for device tests.
- `package_url` - The test component URL corresponding to this test.
This is the URL that Test Manager is requested to execute for device
tests.
- `parallel` - Override for default test runner parallelism (as an
integer). Typically used to instruct test executors to tell `ffx
test` not to run test cases in parallel for tests that have crosstalk.
Setting this value to `1` disables parallel test case execution,
while a number `> 1` forces parallel test case execution if supported
by the individual test runner.

### Examples

**Host Test**

```json5
[
  {
    "environments": [
      {
        // This test runs on the infra shard for Linux x64.
        "dimensions": {
          "cpu": "x64",
          "os": "Linux"
        }
      }
    ],
    "test": {
      "cpu": "x64",
      "label": "//src/performance/trace2json:trace2json_tests(//build/toolchain:host_x64)",
      "source_label": "//src/performance/trace2json:trace2json_tests",
      "name": "host_x64/trace2json_tests",
      "os": "linux",
      // This is the path to execute the test both locally and in infra.
      "path": "host_x64/trace2json_tests",
      // The deps listed in this file must be placed in a known
      // location for the test.
      "runtime_deps": "host_x64/gen/src/performance/trace2json/trace2json_tests.deps.json"
    }
  }
]
```

**Device Test**

```json5
[
  {
    "environments": [
      // This test runs on the AEMU shard in infra.
      {
        "dimensions": {
          "device_type": "AEMU"
        }
      }
    ],
    "test": {
      // This test was created using fuchsia_unittest_package
      "build_rule": "fuchsia_unittest_package",
      "component_label": "//src/diagnostics/lib/sampler-config:sampler-config-tests_component(//build/toolchain/fuchsia:x64)",
      "cpu": "x64",
      // This test specified its own manifest.
      "has_generated_manifest": false,
      "label": "//src/diagnostics/lib/sampler-config:sampler-config-tests_test_sampler-config-tests_component(//build/toolchain/fuchsia:x64)",
      "source_label": "//src/diagnostics/lib/sampler-config:sampler-config-tests_test_sampler-config-tests_component",
      "log_settings": {
        // Any ERROR or FATAL logs will force this test to fail.
        "max_severity": "WARN"
      },
      "name": "fuchsia-pkg://fuchsia.com/sampler-config-tests#meta/sampler-config-tests.cm",
      "os": "fuchsia",
      "package_label": "//src/diagnostics/lib/sampler-config:sampler-config-tests(//build/toolchain/fuchsia:x64)",
      // These manifests can be used for deep inspection of the
      // contents of this test.
      "package_manifests": [
        "obj/src/diagnostics/lib/sampler-config/sampler-config-tests/package_manifest.json"
      ],
      "package_url": "fuchsia-pkg://fuchsia.com/sampler-config-tests#meta/sampler-config-tests.cm"
    }
  }
]
```