A system enabling a Fuchsia test to encode pass/fail expectations and skipped cases for that test. This is particularly helpful for conformance-style test suites where the test suite itself is cleanly separable from the system under test.
With this system enabled, expected-to-fail tests that unexpectedly pass will register as test failures, and vice versa, which is helpful to prompt changes to test expectations to prevent backsliding.
There are two ways to specify expectations for a set of tests: per-package and per-component. Per-package expectations apply to all test components within a package. Per-test expectations apply to only the test cases in that test component. If both mechanisms are used in the same package the expectations for a test component are used instead of the expectations for the package.
To use test expectations for a particular component, fuchsia_test_component_with_expectations
must be used instead of fuchsia_test_component
:
import("//src/lib/testing/expectation/fuchsia_test_component_with_expectations.gni") fuchsia_test_component_with_expectations("expectation-example-test") { expectations = "expectations-example-test-expectations.json5" manifest = "meta/expectation-example-test.cml" } fuchsia_test_package("expectation-example") { test_components = [ ":expectation-example-test" ] }
Test components using this template must specify a path to a manifest file, cm_label is not supported. Test components may be placed in a regular fuchsia_test_package
or a fuchsia_test_with_expectations
package (see below).
Each component specified using fuchsia_test_component_with_expectations
must have its manifest include
the expectation client shard (./meta/client.shard.cml).
To use test expectations for all tests within a package, fuchsia_test_with_expectations_package
must be used instead of fuchsia_test_package
:
import( "//src/lib/testing/expectation/fuchsia_test_with_expectations_package.gni") fuchsia_test_with_expectations_package("expectation-example-package") { test_components = [ ":some-integration-test" ] expectations = "expectations_file.json5" }
Each component specified in test_components
must have its manifest include
the expectation client shard (./meta/client.shard.cml). This is enforced by the fuchsia_test_with_expectations_package
GN template.
See the doc comments in ./fuchsia_test_with_expectations_package.gni for more information.
Pass/fail/skip expectations are specified via a JSON5 file. See ./example_expectations.json5
for a simple example.
generated_expectations_file
allows to merge multiple test expectations file. It is useful in the case expecations depend on the product configuration, e.g.:
import("//src/lib/testing/expectation/generated_expectations_file.gni") import( "//src/lib/testing/expectation/fuchsia_test_with_expectations_package.gni") generated_expectations_file("test-expectations") { includes = [ "expectations_file.json5" ] if (current_cpu == "riscv") { includes += [ "expectations_file_riscv.json5"] } } fuchsia_test_with_expectations_package("expectation-example-package") { test_components = [ ":some-integration-test" ] generated_expectations = ":test-expectations" }
Due to limitations (see https://fxbug.dev/42144060), the testing framework fails any test suite in which error logs are generated, even if the test case generating those logs is expected to fail. There are a couple of options here.
Pros: Simple. Cons: No signal from skipped tests.
Run the tests in a package with:
test_specs { log_settings = { max_severity = "ERROR" } }
Pro: Retain a signal from skipped tests. Con: Lose the signal from error logs.
expect_{pass|failure}
In the expectation's file, include expects with one of the following type
s:
expect_pass_with_err_logs
expect_failure_with_err_logs
When instantiating a test package, pass one of the following flags:
with_err_logs
expect type will be run.with_err_logs
expect type will NOT be run.By default, all cases will be run.
Pros:
Con:
Test expectations provide a natural way of tracking progress over time, but the use of glob matchers makes it hard to quantify the number of tests that are expected to [pass|fail|skip].
To mitigate this challenge, we provide fx list_test_expectations
, a command line tool that makes it easy to exhaustively list expectations as applied to a set of cases.
This tool converts ExpectFailureWithErrLogs
and ExpectPassWithErrLogs
to Skip
when listing expected outcomes.
TODO(https://fxbug.dev/42064224): Support listing expects with error logs.
.json5
expectation files.// test_cases.txt test1 test2 // always_passes.json5 { actions: [ { type: "expect_pass", matchers: [ "*", ], }, ], } // always_fails.json5 { actions: [ { type: "expect_failure", matchers: [ "*", ], }, ], } // test1_pass_test2_fail.json5 { actions: [ { type: "expect_failure", matchers: [ "test2", ], }, { type: "expect_pass", matchers: [ "test1", ], }, ], } $ fx list_test_expectations always_passes.json5 always_fails.json5 test1_pass_test2_fail.json5 < test_cases.txt test1,Pass,Fail,Pass test2,Pass,Fail,Fail