This page provides best practices, examples, and reference materials for using the fx test
command for running tests in a Fuchsia source checkout setup (fuchsia.git
).
To get started, simply run fx test
:
fx test
This will do several things:
If you did not include any tests in your build, fx test
will exit. Try fx set core.x64
--with //src/diagnostics:tests
on your fx set
command line to include some tests as an example.
For more details on the current status of fx test
, see this README
page.
fx test
is a Test Executor, which means it ingests a list of available tests and is responsible for scheduling and observing their execution. The source of this data is tests.json
.
Each test listed in tests.json
is a Test Suite which may each contain any number of Test Cases. That is, a Test Suite is a single binary or Fuchsia Component, and it contains Test Cases which are defined in a way specific to each test framework (e.g. C++ TEST
, Rust #[test]
, Python unittest.TestCase
). Enumerating and executing on-device Test Cases is the responsibility of the Test Runner Framework.
fx test
supports selecting individual Test Suites using command line options. This allows you to include a large number of tests in your build and then only execute a subset of those tests.
Any non-flag argument to fx test
is a selection that is fuzzy-matched against each test in the input:
fx test archivist --dry
Note: The examples in this section pass --dry
to do a dry-run. This will simply list selected tests without running them.
By default, the following fields are searched:
Field | Description |
---|---|
name | The full name of the test. This is component URL for on-device tests and test binary path for host tests. |
label | The build label for the test. For example, //src/examples:my_test . |
component name | The name of the component manifest (excluding .cm ) for on-device tests only. |
package name | The name of the Fuchsia package for on-device tests only. |
You can select all tests below a directory in the source tree by listing the prefix:
fx test //src/diagnostics/tests --dry
By default all of the above fields are matched, but you can select specific fields using --package
or --component
:
fx test --package archivist_unittests --dry
By default, multiple selections on the command line implement an inclusive-OR operation. Test selection supports composite AND operations as follows:
fx test --package archivist --and unittests --dry
This command selects all tests where the package matches archivist
and any field matches unittests
.
If you know the exact name of the test you want to execute, you may use the --exact
flag to select only that test:
fx test --exact fuchsia-pkg://fuchsia.com/archivist-tests#meta/archivist-unittests.cm --dry
If no tests match your selection, fx test
will try to heuristically match tests in your source checkout and suggest fx set
arguments to include them:
$ fx test driver-tests --dry ... For `driver-tests`, did you mean any of the following? driver_tools_tests (91.67% similar) --with //src/devices/bin/driver_tools:driver_tools_tests driver-runner-tests (90.96% similar) --with //src/devices/bin/driver_manager:driver-runner-tests driver-inspect-test (90.96% similar) --with //src/devices/tests/driver-inspect-test:driver-inspect-test
You can then add the necessary packages to your build.
fx test
stores its output in log files for later analysis. You can view a summary of this log file in text form using the -pr/--previous
argument. For example, to see test logs from the previous run:
$ fx test -pr log previous-log-file.json.gz: 4 tests were run [START first_test] ... [END first_test]
For a full list of options for processing previous log files, run fx test -pr help
.
By default this command processes the most recent log stored in your Fuchsia output directory, but you may pass --logpath
to choose a specific log.
This command is resilient to corrupt or incomplete log files, so it should still work even if you terminate the fx test
command running the tests.
fx test
integrates with zxdb
to provide a simple and easy way to debug your test failures, without needing to recompile anything. Pass --break-on-failure
to your fx test
invocation to automatically have test failures break into the debugger:
$ fx test --break-on-failure rust_crasher_test.cm ... ⚠️ zxdb caught test failure in rust_crasher_test.cm, type `frame` to get started. 14 LLVM_LIBC_FUNCTION(void, abort, ()) { 15 for (;;) { ▶ 16 CRASH_WITH_UNIQUE_BACKTRACE(); 17 _zx_process_exit(ZX_TASK_RETCODE_EXCEPTION_KILL); 18 } ══════════════════════════ Invalid opcode exception ══════════════════════════ Process 1 (koid=107752) thread 1 (koid=107754) Faulting instruction: 0x4159210ab797 🛑 process 1 __llvm_libc::__abort_impl__() • abort.cc:16 [zxdb] // Now you can debug why the test failed!
You can also use the --breakpoint=<location>
option to set a breakpoint at a specific location anywhere in your code. <location>
takes standard zxdb breakpoint syntax, typically a file and line number or a function name:
--breakpoint=my_file.rs:123
sets a breakpoint on line 123 of my_file.rs.--breakpoint=some_function
sets a breakpoint on some_function
.Note that this option will cause your tests to run significantly slower, since zxdb will need to load all of the symbols for your test to be able to install the breakpoint. It is highly recommended to only use this option in addition to --test-filter
.
When you're finished debugging the test failure, you can type quit
, ctrl+d
, or detach *
to resume running your tests. Note, if there were multiple test case failures, this will not pause to let you debug those tests as well. See debugging tests for details about how to debug multiple test failures that occur in parallel.
fx test
is highly configurable, and a full list of options is available at fx test --help
.
This section describes how configuration options are specified and what they mean. Configuration options are categorized as Utility, Build, Test Selection, Execution, or Output Options. They may be specified on the command line or in a configuration file.
All arguments for fx test
are set on the command line, but defaults may be set per-user. If you place a file called .fxtestrc
in your HOME directory, the arguments in that file will be the new defaults for future fx test
invocations.
For example:
# ~/.fxtestrc # Lines starting with "#" are comments and ignored. # The below config roughly matches the behavior of the old Dart-based `fx test`. # Default parallel to 1. --parallel 1 # Disable status output. --no-status # Print output for tests taking longer than 2 seconds. --slow 2
The above file overrides the defaults for --parallel
and --status
flags, which normally default to 4
and false
respectively. The new defaults may still be overridden on the command line when invoking fx test
.
Utility options change the overall behavior of fx test
.
--dry
performs a “dry-run.” fx test
will complete test selection, but will then simply print the list of selected test suites rather than executing any of them.
--list
runs fx test
in “list mode.” Rather than executing tests, this command lists all test cases within each test suite. It outputs the appropriate command line to run each individual case. Note that this does require access to a Fuchsia device or emulator because cases are enumerated by Test Manager on device.
-pr/--prev/--previous COMMAND
will process the log file from a previous execution of fx test
, and will print information depending on the value of COMMAND
. No new tests are executed. This command respects --logpath
to specify the log to read from.
The following COMMAND
s are implemented:
log
prints the command line and output for each test recorded in the log file.help
prints a summary of available commands.fx test
builds and updates selected tests by default. This is useful when running fx -i test
, which will detect changes to your source directory and re-invoke fx test
following each file modification. Test rebuilding works as follows (with overrides listed inline).
fx build <targets>
for each fx test
invocation.--[no-]build
to toggle this behavior.fx set --with-base
), the updates
package will be built and an OTA will be performed.--[no-]updateifinbase
to toggle this behavior.The following options affect which tests are selected by fx test
and how selections are applied.
--host
and --device
select only host or device tests respectively. This is a global setting and they cannot be combined.
--[no-]e2e
controls whether to run end-to-end (E2E) tests. E2E tests are not run by default because they have the potential to put the device in an invalid state. --only-e2e
implies --e2e
, and ensures that only E2E tests are selected.
--package
(-p
) and --component
(-c
) select within package or component names respectively. Names preceded by neither select any test field. Multiple selections may be changed by --and
(-a
). For example:
fx test --package foo -a --component bar //src/other --and --package my-tests
The above command line contains two selection clauses:
Tests matching either of the above clauses are selected.
Test selections are fuzzy-matched using a Damerau-Levenshtein distance of 3 by default (e.g. “my_tset” will match “my-test”). --fuzzy <N>
can be used to override this value to N
, where 0 means not to do fuzzy matching.
Suggestions are shown by default if no test matches a selection clause. The number of suggestions (default 6) can be overridden using --suggestions-count N
, and suggestions can be disabled or enabled using --[no-]show-suggestions
.
Tests are executed in a specific way that maximizes throughput and stability, but each element of this default may be overridden. Tests are executed as follows (with overrides listed inline):
Each selected test is executed in the order they appear within tests.json
--random
to randomize this execution order.All selected tests are run, starting at the beginning of the ordered list above.
--offset N
to skip N
tests at the beginning of the list. Default is 0.--limit N
to run at most N
tests from the offset. Default is no limit.At most 4 tests may run in parallel, such that at most one of those tests is “non-hermetic” (as determined by test-list.json
).
--parallel N
to change this default. --parallel 1
means to execute each test serially.Tests run until they terminate themselves.
--timeout N
to wait at most N
seconds per test.Each test runs one time.
--count N
to run each test N
times.All test cases are run from each test.
--test-filter
to run only specifically named test cases.Failed tests are recorded and execution continues with the next selected test.
--fail
(-f
) to terminate all tests following the first failure.Tests that specify a maximum log level in tests.json
will fail if logs at a higher severity are seen.
--[no-]restrict-logs
to toggle this behavior.Tests components themselves choose the minimum log severity to emit.
--min-severity-logs
to override this minimum for all test components.Test components are run using the Merkle root hash from build artifacts, which ensures that the latest version built was successfully pushed to the target and is being run.
--[no-]use-package-hash
to toggle this behavior.Test cases that are disabled are not run.
--also-run-disabled-tests
to run disabled test cases anyway.Test output logs contain only the last segment of the component moniker, so they are easier to visually inspect.
--[no-]show-full-moniker-in-logs
to toggle this behavior.Failing tests terminate following failure without waiting
--break-on-failure
to catch failing tests with zxdb.--breakpoint=<location>
to install breakpoints at specific [locations][#basic-test-debugging].Note that using the --breakpoint
option will significantly slow down your tests. It is highly recommended to only use this option in conjunction with --test-filter
. --break-on-failure
may be used with many tests with minimal impact to performance.
Command line arguments to the test are completely controlled by test runners
--
to your arguments to pass remaining arguments verbatim to the test. For example: fx test foo -- --argument_for_test
will pass --argument_for_test
to the test itself.Host tests will inherit a limited set of environment variables from the user's environment automatically
--env
(-e
) to add new KEY=VALUE
environment variables to tests. This flag may be specified multiple times.fx test
is intended for developer use cases and includes a simple terminal UI that displays the status of tests as they are executing. The default output behavior is as follows (with overrides listed inline):
--[no-]status
to toggle status display.--status-lines N
to change the number of status output lines.--status-delay N
to change the refresh rate (default is 0.033 or approximately 30hz). If your terminal is slow you may want to change this to 0.5 or 1.--[no-]style
to toggle this behavior.--simple
as shorthand for --no-style --no-status
.--output
(-o
) to show all test output (combine with --parallel 1
to prevent interleaving).--no-output
to hide output explicitly, such as to override --output
set in config.--slow N
(-s N
) to show output only for test suites that take longer than N
seconds to execute..json.gz
file under the build directory specified by fx status
.--[no-]log
to toggle logging entirely.--logpath
to change the output path of the log.--ffx-output-directory
to specify a directory where artifacts may be streamed in the ffx test
output format.--verbose
(-v
) to print debug information to the console. This data is extremely verbose, and is only useful to debug fx test
itself.fx test
is designed to support external tooling by representing every user-visible output as an “event” which is logged to a file during execution.
Log files are compressed using gzip. Each line of the decompressed file is a single JSON object representing one event. The event schema is currently defined in this Python file.
When the format is stabilized, it will be possible to build interactive viewers and converters to other formats (such as Build Event Protocol{:.external}).
The emacs compilation window does not emulate an xterm-compatible terminal, resulting in an error like below:
in _make_progress_bar raise ValueError("Width must be at least 3")
To solve this problem, run fx test
with the --no-status
option to disable the status bar.
Your terminal may not support ANSI color codes, which fx test
fails to detect.
Pass the --no-style
option to fx test
to disable color output or the --no-status
option to disable the updating status bar. Passing the --simple
option to fx test
is equivalent to --no-style --no-status
.
You can set the location of the log by passing --logpath
to fx test
, though this is recommended only for non-interactive use.
By default, your logs are stored in your Fuchsia output directory as timestamped files. Print the path to the previous logs using fx test -pr path
.
fx test
logs are gzipped by default. Use the following command to pretty print the most recent log to your terminal:
cat `fx test -pr path` | gunzip | jq -C | less -R
This command does the following:
fx test -pr path
).gunzip
to decompress the log.jq
to pretty-print it with color output (-C
).less
configured to display color (-R
).For convenience, you can add an alias for this command in your .bashrc
file:
alias testlog='cat `fx test -pr path` | gunzip | jq -C | less -R'
The new fx test
command is currently set to be the default.
To opt out of this setting, set the following environment variable:
export FUCHSIA_DISABLED_legacy_fxtest=0