test_list_tool is a host tool that takes tests.json as input and generates a new file called test-list.json. This file contains metadata about the tests, and can include data scraped from generated component manifest files.
See the test-list crate for the latest test-list.json schema.
This tool extends the test-list.json schema with a specific set of tags that will be present for all tests in the list. The specific tags are as follows:
|os||The operating system the test is intended to run on. For example: “linux” or “fuchsia”||Copied from tests.json, which is output by the build.|
|cpu||The CPU this test is intended to run on. For example: “x64” or “arm64”.||Copied from tests.json, which is output by the build.|
|scope||The scope of the test. For example: “unit” or “integration”.||See description below this table.|
|realm||The test realm that the on-device Test Manager will run this test in. For example: “hermetic” or “system”.||Copied from the “fuchsia.test” facet of the test's component manifest file.|
|hermetic||“true” if this test is hermetic, “false” if not hermetic, empty if not applicable.||A test is hermetic if it does not depend on system capabilities or packages.|
|legacy_test||“true” if this test is a legacy test, empty otherwise.||A test is a legacy test if its component URL ends in “.cmx”|
This section describes how we categorize a Test's Scope, which tells us if a test is treated as a “unit test,” “integration test,” “system test,” etc.
The valid scopes supported by this tool are:
|host||The test points to a host binary, not a Fuchsia package.|
|host_shell||The test points to a host shell binary, not a Fuchsia package.|
|unit||The test runs in a hermetic realm and uses a generated default manifest.|
|integration||The test runs in a hermetic realm and uses a non-generated custom manifest.|
|system||The test runs in a non-hermetic realm.|
|wrapped_legacy||The test uses the legacy_test_runner and wraps a v1 test. These tests are not hermetic.|
|fuzzer||The test uses the “fuzzer_package” build rule, and it is hermetic.|
|prebuilt||The test uses the “prebuilt_test_package” build rule and does not run in a non-hermetic realm.|
|bootfs||The test is an ELF binary executed out of /boot on the device.|
|unknown||The test uses an unknown build rule (see algorithm below).|
|uncategorized||The test does not fall into any of the above categories.|
We apply these heuristics as follows:
linux_. These are host tests.
host_and no other tests do. There are some tests for Linux that start with
linux_, and those tests also are scoped to the host.
_host_test.sh. These are host_shell tests.
hermeticrealm it is a system test.
fuzzer_packagebuild rule, it is a fuzzer test.
prebuilt_test_packagebuild rule, it is a prebuilt test.
bootfs_testbuild rule, it is a bootfs test.
/booton the device. We want to track how many tests do this.
The supported build rules are:
fuchsia_unittest_package- Convenience rule for wrapping a single test in a package with a generated manifest.
fuchsia_test_package- Rule for including one or more test components in a package.
fuchsia_test- Rule for wrapping a single test component in a package.
fuzzer_package- Rule for defining a fuzzer and associated test.
prebuilt_test_package- Rule for wrapping a prebuilt test binary from another repository as a test package.