test_list_tool is a host tool that takes tests.json as input and generates a new file called test-list.json. This file contains metadata about the tests, and can include data scraped from generated component manifest files.
See the test-list crate for the latest test-list.json schema.
This tool extends the test-list.json schema with a specific set of tags that will be present for all tests in the list. The specific tags are as follows:
Key | Description | Source |
---|---|---|
os | The operating system the test is intended to run on. For example: “linux” or “fuchsia” | Copied from tests.json, which is output by the build. |
cpu | The CPU this test is intended to run on. For example: “x64” or “arm64”. | Copied from tests.json, which is output by the build. |
scope | The scope of the test. For example: “unit” or “integration”. | See description below this table. |
realm | The test realm that the on-device Test Manager will run this test in. For example: “hermetic” or “system”. | Copied from the “fuchsia.test” facet of the test's component manifest file. |
hermetic | “true” if this test is hermetic, “false” if not hermetic, empty if not applicable. | A test is hermetic if its realm is “hermetic”. |
legacy_test | “true” if this test is a legacy test, empty otherwise. | A test is a legacy test if its component URL ends in “.cmx” |
This section describes how we categorize a Test's Scope, which tells us if a test is treated as a “unit test,” “integration test,” “system test,” etc.
The valid scopes supported by this tool are:
Scope | Heuristic |
---|---|
host | The test points to a host binary, not a Fuchsia package. |
host_shell | The test points to a host shell binary, not a Fuchsia package. |
unit | The test runs in a hermetic realm and uses a generated default manifest. |
integration | The test runs in a hermetic realm and uses a non-generated custom manifest. |
system | The test runs in a non-hermetic realm. |
wrapped_legacy | The test uses the legacy_test_runner and wraps a v1 test. These tests are not hermetic. |
fuzzer | The test uses the “fuzzer_package” build rule, and it is hermetic. |
prebuilt | The test uses the “prebuilt_test_package” build rule and does not run in a non-hermetic realm. |
bootfs | The test is an ELF binary executed out of /boot on the device. |
unknown | The test uses an unknown build rule (see algorithm below). |
uncategorized | The test does not fall into any of the above categories. |
We apply these heuristics as follows:
host_
or linux_
. These are host tests.host_
and no other tests do. There are some tests for Linux that start with linux_
, and those tests also are scoped to the host._host_test.sh
. These are host_shell tests.hermetic
or hermetic-tier-2
realms it is a system test.fuzzer_package
build rule, it is a fuzzer test.prebuilt_test_package
build rule, it is a prebuilt test.bootfs_test
build rule, it is a bootfs test./boot
on the device. We want to track how many tests do this.The supported build rules are:
fuchsia_unittest_package
- Convenience rule for wrapping a single test in a package with a generated manifest.fuchsia_test_package
- Rule for including one or more test components in a package.fuchsia_test
- Rule for wrapping a single test component in a package.fuzzer_package
- Rule for defining a fuzzer and associated test.prebuilt_test_package
- Rule for wrapping a prebuilt test binary from another repository as a test package.