Please see the CTS overview for an explanation of what CTS is.
CTS has multiple releases with separate cadences:
Release | Schedule |
---|---|
Canary | ~4 hrs |
Milestone | ~6 weeks |
The canary release is created when new canary releases of the Fuchsia platform are created. Likewise, milestone releases are created when new milestone releases of the Fuchsia platform are created.
Milestone branches (e.g. releases/f7) often receive cherry picks. When this happens, a new CTS for that milestone is generated and automatically rolled into CI/CQ.
{% dynamic if user.is_googler %}
Internal contributors: Look for builders named cts*prebuilt-roller in Milo to monitor new releases.
{% dynamic endif %}
The tip-of-tree version of your test will immediately begin running on CI/CQ. This version of the test does not guarantee backward compatibility.
When the next CTS release is cut, it will contain a snapshot of your test from tip-of-tree which will begin running as soon as that CTS release rolls into CI/CQ. This version of your test guarantees backward compatibility.
See this section above for the release schedule.
See go/fuchsia-builder-viz. Look for builders whose names end in “-cts”.
At minimum, all CTS tests run on the core.x64 image in the Fuchsia emulator.
CQ may run several versions of the same CTS test at a time: The version from tip-of-tree, from the latest canary release, and from a previous milestone release.
The tip-of-tree version of the test has a package name without a release version. For example:
fuchsia-pkg://fuchsia.com/memfs-test-package#meta/memfs-component.cmx
The canary or milestone versions of the test include the release version. For example:
fuchsia-pkg://fuchsia.com/memfs-test-package_6.20211109.1.3166058#meta/memfs-component.cmx
This depends on the version of the test you'd like to run.
To run the tip-of-tree version locally, you can do:
fx set //sdk/cts/tests fx test TEST_NAME
For example:
fx set //sdk/cts/tests fx test memfs-test-package
To run the release version locally, you can do:
fx set //sdk/cts/release:tests fx test TEST_NAME
For example:
fx set //sdk/cts/release:tests fx test memfs-test-package_6.20211109.1.3166058
Please see Run Fuchsia Tests for more information about how to run tests.
This is a sign that your CL is breaking a part of the platform surface area. Please verify that there are no projects in downstream repositories that rely on the APIs and ABIs modified by your CL. If so, you will need to make a soft transition. The general worklow is as follows:
Yes! See //sdk/cts/examples for some examples, or peruse the complete set of tests under //sdk/cts/tests.
You should write a CTS test if:
A CTS test should stop guarding against breaking changes once the SDK element it covers is removed (deprecated and no longer supported by the platform, even for legacy clients). This process is called test “retirement” and allows Fuchsia contributors to remove things from the SDK.
To retire an entire CTS test, delete the test at HEAD before the upcoming milestone release. The version of the test from the previous CTS release will continue running in CQ until the next release is cut.
To retire a few test cases, follow the same procedure: Delete the test cases at HEAD and wait for the next milestone release.
If you must immediately make changes to a previously released version of a test, you'll need to get approval from the Release Team to have the change cherry picked onto the appropriate release branch.
To verify that your change will succeed, you should sync your local Fuchsia checkout to the release branch and test the change yourself, first. After verifying, submit the CL and file a bug against the Release Team.
You can disable a test by adding the test's package and component name to the list of disabled_tests
on the appropriate compatibility_test_suite
target in //sdk/cts/release/BUILD.gn
.
For example, a test running in Fuchsia's canary release might have the package URL:
fuchsia-pkg://fuchsia.com/my_test_canary#meta/my_test_component.cm
This can be disabled as follows:
compatibility_test_suite("canary") { {{ '<strong>' }}disabled_tests = [ { package = "my_test_canary" component_name = "my_test_component" }, ]{{ '</strong>' }} }
Please include a comment with a bug ID as a reminder to enable the test again. Tests should be enabled again within 72 hours.
If you need to disable a test for an extended period of time, please instead remove the test from the CTS release by submitting a change like the following:
cts_fuchsia_test_package("my_test") { test_components = [ ":my_test_component" ] {{ '<strong>' }}internal_only_skip_on_cq = true{{ '</strong>' }} }
Then ask the release team to cherry pick this CL onto the appropriate release branch which will generate a new CTS without the test.
For questions and clarification on this document, please reach out to this directory's owners or file a bug in the CTS bug component.