The Fuchsia Component Framework allows developers to create components in a variety of languages and runtimes. Fuchsia's own code uses a diverse mix of programming languages for components, including C/C++, Rust, Dart, and Go.
The Test Runner Framework uses Component Framework runners as an integration layer between various testing runtimes and a common Fuchsia protocol for launching tests and receiving their results. This makes for an inclusive design that on one hand allows developers to bring their language and testing framework of choice, and on the other hand allows building and testing Fuchsia on a variety of systems and targeting different hardware.
The test_manager
component is responsible for running tests on a Fuchsia device. Test manager exposes the fuchsia.test.manager.RunBuilder
protocol, which allows launching test suites.
Each test suite is launched as a child of test manager. Test suites are offered capabilities by test manager that enable them to do their work while maintaining isolation between the test and the rest of the system. For instance hermetic tests are given the capability to log messages, but are not given the capability to interact with real system resources outside of their sandbox. Test manager uses only one capability from the test realm, a controller protocol that test suites expose. This is done to ensure hermeticity (test results aren‘t affected by anything outside of their intended sandbox) and isolation (tests don’t affect each other or the rest of the system).
The test manager controller itself is offered to other components in the system in order to integrate test execution with various developer tools. Tests can then be launched with such tools as fx test
and ffx
.
The test suite protocol, fuchsia.test.Suite
, is used by the test manager to control tests, such as to invoke test cases and to collect their results.
Test authors typically don't need to implement this protocol. Instead, they rely on a test runner to do this for them. For instance, you might write a test in C++ using the GoogleTest framework, and then use gtest_runner
in your component manifest to integrate with the Test Runner Framework.
Test runners are reusable adapters between the Test Runner Framework and common languages & frameworks used by developers to write tests. They implement the fuchsia.test.Suite
protocol on behalf of the test author, allowing developers to write idiomatic tests for their language and framework of choice.
Component manifests for simple unit tests can be generated by the build rules. Generated component manifests for v2 tests will include the appropriate test runner based on their build definition. For instance a test executable that depends on the GoogleTest library will include the GoogleTest runner in its generated manifest.
The following test runners are currently available for general use:
A runner for tests written in C/C++ using the GoogleTest framework. Use this for all tests written using GoogleTest.
Common GoogleTest features are supported, such as disabling tests, running only specified tests, running the same test multiple times, etc'. Standard output, standard error, and logs are captured from the test.
In order to use this runner, add the following to your component manifest:
{ include: [ "//src/sys/test_runners/gtest/default.shard.cml" ] }
By default GoogleTest test cases run serially (one test case at a time).
A runner for tests written in C/C++ using the GUnit framework. Use this for all tests written using the gUnit flavor of GoogleTest.
Note: Gtest and Gunit testing framework differ in flag names, so we have a separate runner for gunit.
Common GoogleTest features are supported, such as disabling tests, running only specified tests, running the same test multiple times, etc'. Standard output, standard error, and logs are captured from the test.
In order to use this runner, add the following to your component manifest:
{ include: [ "sys/testing/gunit_runner.shard.cml" ] }
By default test cases run serially (one test case at a time).
A runner for tests written in the Rust programming language and following Rust testing idioms. Use this for all idiomatic Rust tests (i.e. tests with modules that set the attribute [cfg(test)]
).
Common Rust testing features are supported, such as disabling tests, running only specified tests, running the same test multiple times, etc'. Standard output, standard error, and logs are captured from the test.
In order to use this runner, add the following to your component manifest:
{ include: [ "//src/sys/test_runners/rust/default.shard.cml" ] }
By default Rust test cases run in parallel, at most 10 cases at a time.
A runner for tests written in the Go programming language and following Go testing idioms. Use this for all tests written in Go using import "testing"
.
Common Go testing features are supported, such as disabling tests, running only specified tests, running the same test multiple times, etc'. Standard output, standard error, and logs are captured from the test.
In order to use this runner, add the following to your component manifest:
{ include: [ "//src/sys/test_runners/gotests/default.shard.cml" ] }
By default Go test cases run in parallel, at most 10 cases at a time.
The simplest test runner - it waits for your program to terminate, then reports that the test passed if the program returned zero or that it failed for any non-zero return value.
Use this test runner if your test is implemented as an ELF program (for instance an executable written in C/C++) but it does not use a common testing framework that‘s supported by existing runners and you’d rather not implement a bespoke test runner.
In order to use this runner, add the following to your component manifest:
{ include: [ "sys/testing/elf_test_runner.shard.cml" ] }
If you are using in-tree unit test GN templates, and you are not already using a test framework with a dedicated test runner, add the following to your build deps:
fuchsia_unittest_package("my-test-packkage") { // ... deps = [ // ... "//src/sys/testing/elftest", ] }
Note: If you see the error message “Component has a `program` block defined, but doesn't specify a `runner`” for your test, this indicates you are not using a test framework with a dedicated test runner, and you should add the above dependency.
When using fx test
to launch tests, they may run each test case in sequence or run multiple test cases in parallel up to a given limit. The default parallelism behavior is determined by the test runner. To manually control the number of test cases to run in parallel use test spec:
fuchsia_test_package("my-test-pkg") { test_components = [ ":my_test_component" ] test_specs = { # control the parallelism parallel = 10 } }
To run a test multiple times use:
fx test --count=<n> <test_url>
If an iteration times out, no further iteration will be executed.
Custom arguments to the tests can be passed using fx test
:
fx test <test_url> -- <custom_args>
Individual test runners have restrictions on these custom flags:
Note the following known behavior changes:
--gtest_break_on_failure - Instead use:
fx test --break-on-failure <test_url>
The following flags are restricted and the test fails if any are passed as fuchsia.test.Suite provides equivalent functionality that replaces them.
fx test --test-filter=<glob_pattern> <test_url>
--test-filter
may be specified multiple times. Tests that match any of the given glob patterns will be executed.
fx test --also-run-disabled-tests <test_url>
Note the following known behavior changes:
--gunit_break_on_failure - Instead use:
fx test --break-on-failure <test_url>
The following flags are restricted and the test fails if any are passed as fuchsia.test.Suite provides equivalent functionality that replaces them.
fx test --test-filter=<glob_pattern> <test_url>
--test-filter
may be specified multiple times. Tests that match any of the given glob patterns will be executed.
fx test --also-run-disabled-tests <test_url>
The following flags are restricted and the test fails if any are passed as fuchsia.test.Suite provides equivalent functionality that replaces them.
fx test --test-filter=<glob_pattern> <test_url>
--test-filter
may be specified multiple times. Tests that match any of the given glob patterns will be executed.
Note the following known behavior change:
-test.failfast: As each test case is executed in a different process, this flag will only influence sub-tests.
The following flags are restricted and the test fails if any are passed as fuchsia.test.Suite provides equivalent functionality that replaces them
fx test --test-filter=<glob_pattern> <test_url>
--test-filter
may be specified multiple times. Tests that match any of the given glob patterns will be executed.
Fuchsia aims to be inclusive, for instance in the sense that developers can create components (and their tests) in their language and runtime of choice. The Test Runner Framework itself is language-agnostic by design, with individual test runners specializing in particular programming languages or test runtimes and therefore being language-inclusive. Anyone can create and use new test runners.
Creating new test runners is relatively easy, with the possibility of sharing code between different runners. For instance, the GoogleTest runner and the Rust runner share code related to launching an ELF binary, but differ in code for passing command line arguments to the test and parsing the test's results.
To use temporary storage in your test, add the following to your component manifest:
{ include: [ "//src/sys/test_runners/tmp_storage.shard.cml" ] }
At runtime, your test will have read/write access to /tmp
. The contents of this directory will be empty when the test starts, and will be deleted after the test finishes.
Tests that don't specify a custom manifest and instead rely on the build system to generate their component manifest can add the following dependency:
fuchsia_unittest_package("foo-tests") { deps = [ ":foo_test", "//src/sys/test_runners:tmp_storage", ] }
To export custom files from your test, use the custom_artifacts storage capability. The contents of custom_artifacts are copied out at the conclusion of a test.
To use custom_artifacts in your test, add the following to your component manifest:
{ use: [ { storage: "custom_artifacts", rights: [ "rw*" ], path: "/custom_artifacts", }, ], }
At runtime, your test will have read/write access to /custom_artifacts
. The contents of this directory will be empty when the test starts, and will be deleted after the test finishes.
See the custom artifact test example. To run it, add //examples/tests/rust:tests
to your build, then run:
fx test --ffx-output-directory <output-dir> custom_artifact_user
After the test concludes, <output-dir>
will contain the artifact.txt
file produced by the test.
In the context of software testing, Hermeticity refers to the isolation of a test or test suite from external factors and dependencies, ensuring that it produces consistent and reliable results regardless of changes in the surrounding environment. A hermetic test is self-contained and doesn't rely on external systems or data that might change unexpectedly, leading to flaky or non-deterministic test outcomes.
Hermeticity does not mean protection from in-stable platform or routed capabilities/APIs. If an API surface is in-stable for certain components in the system, then they will be instable for tests and dependent components and will help catch regressions due to any direct or in direct changes to system's API surface.
Ability to write fully, provably hermetic tests is Fuchsia's Testing Superpower. There are two type of test hermeticity:
Capability: The Test does not use or offer any capabilities from the test root's parent. These tests don't have access to any system capabilities which can affect larger system. Due to this property of hermetic tests they can be run in parallel and will not flake due to cross-talk or shared state, it improves stability and performance of tests.
Package: The Test does not resolve any components outside of the test package. A hermetically packaged test does not have any implicit contract with platform packages. This provides a way to update them without affecting system packages and avoids dependence on incompatible packaged dependencies.
Hermeticity does not apply to platform APIs/capabilities which are available to all components in the system. For eg
The tests should be careful when using these APIs/capabilities.
The tests are by default hermetic unless explicitly stated otherwise.
There are some capabilities which all tests can use which do not violate test hermeticity:
Protocol | Description |
---|---|
fuchsia.boot.WriteOnlyLog | Write to kernel log |
fuchsia.logger.LogSink | Write to syslog |
fuchsia.process.Launcher | Launch a child process from the test package |
fuchsia.diagnostics.ArchiveAccessor | Read diagnostics output by components in the test |
The hermeticity is retained because these capabilities are carefully curated to not allow tests to affect the behavior of system components outside the test realm or of other tests.
To use these capabilities, there should be a use declaration added to test's manifest file:
// my_test.cml { use: [ ... { protocol: [ "{{ '<var label="protocol">fuchsia.logger.LogSink</var>' }}" ], }, ], }
Tests are also provided with some default storage capabilities which are destroyed after the test finishes execution.
Storage Capability | Description | Path |
---|---|---|
data | Isolated data storage directory | /data |
cache | Isolated cache storage directory | /cache |
tmp | Isolated in-memory temporary storage directory | /tmp |
Add a use declaration in test's manifest file to use these capabilities.
// my_test.cml { use: [ ... { storage: "{{ '<var label="storage">data</var>' }}", path: "{{ '<var label="storage path">/data</var>' }}", }, ], }
The framework also provides some capabilities to all the components and can be used by test components if required.
Hermetic test components are launched in a realm that utilizes the hermetic component resolver. This resolver disallows resolving URLs outside of the test‘s package. This is necessary for enforcing hermeticity, as we don’t want the availability of an arbitrary component on the system or in an associated package server to affect the outcome of a test.
Attempts to resolve a component not in the test's package will be met with a PackageNotFound
error and the following message in the syslog:
failed to resolve component fuchsia-pkg://fuchsia.com/[package_name]#meta/[component_name]: package [package_name] is not in the set of allowed packages...
You can avoid this error by including any components your test relies on to the test package - see this CL for an example of how to do this, or by using subpackages:
# BUILD.gn import("//build/components.gni") fuchsia_test_package("simple_test") { test_components = [ ":simple_test_component" ] subpackages = [ "//path/to/subpackage:subpackage" ] }
// test.cml { ... children: [ { name: "child", url: "subpackage#meta/subpackaged_component.cm", }, ], ... }
See this CL as an example of using subpackages.
Note: Subpackages are not available in the sdk yet. If you are not able to include a dependent component in your test package you can add below option to your test manifest file to selectively allow resolution of some packages:
// my_component_test.cml { ... facets: { "fuchsia.test": { "deprecated-allowed-packages": [ "non_hermetic_package" ], }, }, ... }
These tests can access some pre-defined capabilities outside of the test realm. A capability accessed by non-hermetic test from outside its test realm is called a system capability.
To use a system capability, a test must explicitly mark itself to run in non-hermetic realm as shown below.
# BUILD.gn (in-tree build rule) fuchsia_test_component("my_test_component") { component_name = "my_test" manifest = "meta/my_test.cml" deps = [ ":my_test_bin" ] # This runs the test in "system-tests" non-hermetic realm. test_type = "system" }
After integrating with the build rule, the test can be executed as
fx test <my_test>
Or for out-of-tree developers
ffx test run --realm <realm_moniker> <test_url>
where realm_moniker
should be replaced with /core/testing/system-tests
for above example.
Possible values of test_type
:
Value | Description |
---|---|
chromium | Chromium test realm |
ctf | CTF test realm |
device | Device tests |
drm | DRM tests |
starnix | Starnix tests |
system_validation | System Validation Tests |
system | Legacy non hermetic realm with access to some system capabilities. |
test_arch | Test Architecture Tests |
vfs-compliance | VFS compliance tests |
vulkan | Vulkan tests |
Learn how to create your own test realm.
These are legacy test realms created before we had Test Manager as a Service. We are in process of porting these realms. If your tests depend on one of these realms, it should explicitly mark itself to run in the legacy realm as shown below.
// my_component_test.cml { include: [ // Select the appropriate test runner shard here: // rust, gtest, go, etc. "//src/sys/test_runners/rust/default.shard.cml", // This includes the facet which marks the test type as 'starnix'. {{ '<strong>' }}"//src/devices/testing/starnix_test.shard.cml",{{ '</strong>' }} ], program: { binary: "bin/my_component_test", }, {{ '<strong>' }} use: [ { protocol: [ "fuchsia.vulkan.loader.Loader" ], }, ],{{ '</strong>' }} }
The shard includes following facet in the manifest file:
{% includecode gerrit_repo="fuchsia/fuchsia" gerrit_path="src/starnix/tests/starnix_test.shard.cml" %}
Possible values of fuchsia.test.type
:
Value | Description |
---|---|
hermetic | Hermetic realm |
chromium-system | Chromium system test realm |
google | Google test realm |
By default, a test will fail if it logs a message with a severity of ERROR
or higher. See this guide for more information.
When writing a test runner that launches processes, the runner needs to provide a library loader implementation.
Test runners typically launch individual test cases in separate processes to achieve a greater degree of isolation between test cases. However this can come at a significant performance cost. To mitigate this, the test runners listed above use a caching loader service which reduces the extra overhead per process launched.
Components in the test realm may play various roles in the test, as follows:
fuchsia.test.Suite
exposed by this component to drive the test.fuchsia.test.Suite
protocol. Note that the test driver and test root may be, but are not necessarily, the same component: the test driver could be a subcomponent of the test root which re-exposes its fuchsia.test.Suite
, for example.This section contains common issues you may encounter while developing test components with the Test Runner Framework. If one of your test components fails to run, you may see an error like the following from fx test
:
Test suite encountered error trying to run tests: getting test cases Caused by: The test protocol was closed. This may mean `fuchsia.test.Suite` was not configured correctly.
To address the issue, explore the following options:
fuchsia.test.Suite
to test managerfuchsia.test.Suite
to the rootIf you encountered this error during test enumeration then probably you are using the wrong test runner.
For example: your Rust test file might be running the test without using Rust test framework (i.e it is a simple Rust binary with its own main function). In this case change your test manifest file to use elf_test_runner.
Read more about in-built test runners.
fuchsia.test.Suite
to test managerThis happens when the test root fails to expose fuchsia.test.Suite
from the test root. The simple fix is to add an expose
declaration:
// test_root.cml expose: [ ... { protocol: "fuchsia.test.Suite", from: "self", // If a child component is the test driver, put `from: "#driver"` }, ],
fuchsia.test.Suite
to the rootYour test may fail with an error similar to the following if the fuchsia.test.Suite
protocol is not properly exposed:
ERROR: Failed to route protocol `/svc/fuchsia.test.Suite` from component `/test_manager/...`: An `expose from #driver` declaration was found at `/test_manager/...` for `/svc/fuchsia.test.Suite`, but no matching `expose` declaration was found in the child
If the test driver and test root are different components, the test driver must also expose fuchsia.test.Suite
to its parent, the test root.
To address this issue, ensure the test driver component manifest includes the following expose
declaration:
// test_driver.cml expose: [ ... { protocol: "fuchsia.test.Suite", from: "self", }, ],
The test driver must use the appropriate test runner corresponding to the language and test framework the test is written with. For example, the driver of a Rust test needs the following declaration:
// test_driver.cml include: [ "//src/sys/test_runners/rust/default.shard.cml" ]
Also, if the test driver is a child of the test root, you need to offer it to the driver:
// test_root.cml offer: [ { runner: "rust_test_runner", to: [ "#driver" ], }, ],