Fuchsia's build system exports structured information about all tests included in a build in a tests.json
file in the build output directory (e.g. $FUCHSIA_DIR/out/default/tests.json
).
Fuchsia's continuous integration infrastructure uses tests.json
to determine which tests to run, and fx test
uses it to determine which tests may be run locally. tests.json
and its schema are implementation details of the platform build and test and are not part of the SDK.
This document describes the file's contents and how the file is generated.
The tests.json
file is generated by the “tests” api module at the root of the source tree.
The fields for tests.json
entries are collected using the GN metadata mechanism. tests.json
entries are registered from GN files by using the test_spec
GN template.
tests.json
is automatically generated by fx set
. The tests.json
format is limited by what can readily be generated from GN using GN's relatively limited metadata mechanism. To get around these limitations, the Fuchsia build also produces file called test-list.json
, which is generated by post-processing tests.json
as described below.
Various tools and scripts ingest the tests.json
file since it is the canonical list of available tests for a given Fuchsia build. This section surveys the most notable consumers of the format.
The test_list_tool processes tests.json
. Using the canonical list of tests, it uses the contained package manifest information to load the build packages and annotate each test with information about how it should be executed and tags for additional categorization. For example, it can determine if a test component runs hermetically, and tags the test with "hermetic": true
.
The test_list_tool outputs this information to a test-list.json
file adjacent to tests.json
, and it is run as one of the last build steps after all packages are built. It uses the same unique name
key as tests.json
so that entries can be matched.
The fx search-tests
script provides fuzzy matching on test names to help developers find which tests are available to run. It uses the values in tests.json
as keys for fuzzy matching, in addition to searching for non-included tests through the source tree.
The fx test
command is the developer-facing entrypoint to test execution in the Fuchsia source tree. It uses tests.json
to determine the canonical set of tests, and then finds associated data in test-list.json
and test_components.json
to determine how a test should be executed.
In Infra, testsharder depends on tests.json
to shard test entries based on environment dimensions. The output of that process is another file containing fields derived from tests.json
which is then passed to botanist on the bots where tests run.
tests.json
is a JSON file consisting of a single array. Each element of the array corresponds to a single Test Target entry in the build.
Test Target entries have two top-level fields, environments
and test
corresponding to Test Environments and Test Info respectively.
The environments
field is an array of Test Environment entries. Each Test Environment entry specifies a single environment the test may run in, and this field is used by testsharder to shard tests between device configurations.
Each Test Environment entry has a single field dimensions
with any number of arbitrary dimensions. Typically, however, there are two sets of fields that are most common:
device_type
- Specifies the required device type for the test. For example:
"environments": [ { "dimensions": { "device_type": "QEMU" } } ]
This denotes a test that can run on the QEMU device configuration. Note: This does not mean that the test runs on any QEMU device, rather it means that the test runs on a QEMU instance based on the configuration used by botanist
.
cpu
and os
- Specifies the required CPU architecture and operating system for a host test. For example:
"environments": [ { "dimensions": { "cpu": "x64", "os": "Linux" } } ]
This specifies a test that can run on an x64 Linux system.
tags
- Specifies other properties for sharding. For example:
"environments": [ { "tags": [ "e2e-isolated" ] } ]
This specifies a test with tag “e2e-isolated”.
Tests with an environment tag will not be run in continuous integration builds unless the build configuration specifically opts in to running tests with that tag, in which case only tests with that tag will run.
The test
field provides information about how to categorize, run, and configure the test. Test executors like fx test
and botanist
use this data to determine the correct binary and command line to execute the test. It also contains information that is useful for other tooling integrations, for instance, to rebuild test targets between test invocations.
There are generally two types of test: Host and Device. Host tests refer to a binary path on the host system that is run to execute the test. Device tests refer to a component URL that is run on the target system using Test Manager.
Some host tests require a Fuchsia device and some do not. An end-to-end test (E2E) has an os
that is not “fuchsia,” but still has a device_type
specified in its environments
.
The test
object contains the following fields, split by type of test:
All tests
name
- Name for the test. This value is unique within tests.json
. Typically contains the test component URL for device tests and the path to the test binary relative to the output directory for host tests.cpu
- The CPU architecture this test runs on. Redundant with the environments
entry for host tests, but the only source of this information for device tests.os
- The OS this test runs on. Redundant with the environments
entry for host tests, but the only source of this information for device tests. This field is used by botanist
to determine if a test should be run as a subprocess on the host machine or run over SSH to a target device.label
- The GN label that produced this test entry. Used to know which targets to rebuild when executing tests from fx test
, but only for host tests for which this label actually outputs the real test binary. Other types of tests use different label fields such as package_label
.Host tests only
path
- The path to the test binary, relative to the output directory. Present only for host tests.runtime_deps
- The path (relative to the output directory) to a JSON file containing a list of file paths (relative to the output directory) to files that should be put in a place known to a host test for use at runtime. The format of the file is a JSON array of file path strings.Device tests only
has_generated_manifest
- True if this is a test component whose manifest was generated by a GN rule, omitted if the test is a host test, false otherwise. Used for test categorization and tracking of tests using custom manifests.build_rule
- Generated by the build for further test categorization. For example, it is meaningful to know if a test was was created by the fuchsia_unittest_package
or the fuchsia_test_package
rule.log_settings
- A dict that may contain these fields:max_severity
- This field overrides the default maximum log severity for tests. By default tests that emit an ERROR log will fail, but this field instructs test executors to run in a mode where a different level of logs causes failure.min_severity
- Used to instruct the test components to emit logs at this level rather than the default. By default tests choose their own minimum log severity.package_label
- The GN label that produced the package this test is contained in. Only present for device tests. Used by fx test
to rebuild the packages containing tests before they are run.component_label
- The GN label that produces the test component for this test. Used for fuzzy matching test names when no matching tests are found in fx search-tests
(in addition to all other label fields). Only present for device tests.package_manifests
- List of paths (relative to output directory) for each package manifest involved in constructing the test package. Used to determine which packages to republish after building tests in fx test
. Only present for device tests.package_url
- The test component URL corresponding to this test. This is the URL that Test Manager is requested to execute for device tests.parallel
- Override for default test runner parallelism (as an integer). Typically used to instruct test executors to tell ffx test
not to run test cases in parallel for tests that have crosstalk. Setting this value to 1
disables parallel test case execution, while a number > 1
forces parallel test case execution if supported by the individual test runner.Host Test
[ { "environments": [ { // This test runs on the infra shard for Linux x64. "dimensions": { "cpu": "x64", "os": "Linux" } } ], "test": { "cpu": "x64", "label": "//src/performance/trace2json:trace2json_tests(//build/toolchain:host_x64)", "name": "host_x64/trace2json_tests", "os": "linux", // This is the path to execute the test both locally and in infra. "path": "host_x64/trace2json_tests", // The deps listed in this file must be placed in a known // location for the test. "runtime_deps": "host_x64/gen/src/performance/trace2json/trace2json_tests.deps.json" } } ]
Device Test
[ { "environments": [ // This test runs on the AEMU shard in infra. { "dimensions": { "device_type": "AEMU" } } ], "test": { // This test was created using fuchsia_unittest_package "build_rule": "fuchsia_unittest_package", "component_label": "//src/diagnostics/lib/sampler-config:sampler-config-tests_component(//build/toolchain/fuchsia:x64)", "cpu": "x64", // This test specified its own manifest. "has_generated_manifest": false, "label": "//src/diagnostics/lib/sampler-config:sampler-config-tests_test_sampler-config-tests_component(//build/toolchain/fuchsia:x64)", "log_settings": { // Any ERROR or FATAL logs will force this test to fail. "max_severity": "WARN" }, "name": "fuchsia-pkg://fuchsia.com/sampler-config-tests#meta/sampler-config-tests.cm", "os": "fuchsia", "package_label": "//src/diagnostics/lib/sampler-config:sampler-config-tests(//build/toolchain/fuchsia:x64)", // These manifests can be used for deep inspection of the // contents of this test. "package_manifests": [ "obj/src/diagnostics/lib/sampler-config/sampler-config-tests/package_manifest.json" ], "package_url": "fuchsia-pkg://fuchsia.com/sampler-config-tests#meta/sampler-config-tests.cm" } } ]