{% set rfcid = “RFC-0015” %} {% include “docs/contribute/governance/rfcs/_common/_rfc_header.md” %}
This document presents requirements, design, and implementation strategy for a Compatibility Test Suite (CTS) for the Fuchsia platform. A CTS will offer a way of testing platform implementations to make sure that they behave according to Fuchsia's specifications. Fuchsia developers will write tests that guarantee compatibility across changes to both the source and behavior of the platform. When these tests pass, they will guarantee that a particular release, when run on a particular device, is compatible with a target API level and target ABI revision, as defined in RFC-0002.
For the purposes of this document, we refer to API and ABI jointly as the platform surface area. A platform surface area element is a named and versioned identifier associated with the platform surface area (e.g., a method name). Future RFCs may formalize these definitions.
All of the open-source tests for Fuchsia platform behavior are currently (December 2020) built and run as part of the platform build. As the platform evolves, we keep the tests passing at head. As a result, we have no tests that guarantee backwards compatibility with older versions of the platform.
Currently, we use a number of product tests to ensure compatibility and stability. These product tests are difficult to triage: because they rely on the stability of the product, and target many different parts of the platform, it is difficult for platform engineers to determine where the bug may be.
At the same time, developers are writing code that targets older versions of the Fuchsia platform surface area. In this document, we refer to such developers as end developers.
As we roll out API breaking changes, we have no safeguards in place that raise a flag when we break compatibility with end developers' code. Over the course of the project, undocumented API changes have frequently been released that cause external code to stop building.
Furthermore, we are currently building out strong guarantees about backwards ABI compatibility. As of 9 November 2020, we require a six-week ABI compatibility window, but have no enforcement mechanism for it.
We need a set of tests that we can execute independently from the platform build that identify clearly when we break our contracts with end developers. This will help ensure that we maintain compatibility with externally developed code, and provide more easily triaged, targeted test coverage for the parts of the platform that are currently only exercised by product tests.
In the long term, we will also need a set of tests that system integrators can execute to know if they are producing a compliant Fuchsia implementation.
Fuchsia's CTS will offer a way of testing platform implementations to make sure that they are compatible with particular platform releases (as defined in [FTP-0002]). We aspire to have a test for each documented behavior in the platform surface area.
When we create a release, we can use the CTS to tell us about the compatibility of its surface area with that of other release versions.
When someone is developing a device running Fuchsia, and wants to see if it is compatible with a given SDK, they can take the CTS and the SDK with which it wants to demonstrate compatibility, pass the tests, and have confidence that their product correctly implements Fuchsia's semantics - it will be “Fuchsia compatible”.
When a developer wants to understand how to write code against a particular API, or using a particular ABI, they will be able to use these tests as reference.
RFC-0002 allows a platform implementation to provide partial support for backwards compatibility. CTS will provide a way to test partial compatibility.
Note that the CTS is not intended as a complete solution for platform evolution and backwards compatibility. It's not likely that CTS tests will cover every last use case. API and ABI will still have to be designed with future use cases in mind. See the section on drawbacks and alternatives for additional discussion.
The CTS design involves balancing ease of development with the need to build and run the CTS itself outside of the Fuchsia repository. The requirements are as follows:
There should be a CTS test for every documented behavior of every platform surface area element. Although we expect this to become a hard requirement eventually, this RFC does not specify such a requirement.
CTS tests may not rely on any internal details of a particular system image. To the extent that they rely on other platform code, that code must be bundled as part of the CTS and also not rely on any internal details of a particular system image.
CTS tests must be updated by developers (that is, tests must be added or modified) when adding or changing elements of the platform surface area.
It must be possible to determine the API level and ABI revision of Fuchsia that a given CTS artifact targets.
CTS tests that are not included in the test suite as prebuilt artifacts must be written in languages supported by the SDK being used to test (see the supported languages document and the Language section below for more details).
We develop CTS tests alongside their corresponding SDK elements. Today, that means we develop the tests in fuchsia.git. While it would be nice if CTS developers had the same experience as the out of tree developers who use the SDK, there are too many advantages to in-tree development to ignore:
Because feature development is done alongside test development, in-tree development of the tests will allow test authors to use a workflow with which they are familiar, as well as submit the test in the same CL as the feature.
Because the feature will be submitted at the same time as the test, there is no need for any machinery to align the CTS and the version that it qualifies.
CTS tests will use build-time enforcement to ensure that CTS tests can only depend on SDK elements or other pre-approved CTS code. One of the dangers of developing in-tree is that we may accidentally take on dependencies on platform implementation details that are not exposed via the SDK. CTS tests must only access publicly facing elements of the platform to prevent accidentally relying on implementation details. CTS tests may use platform code that is appropriate for writing tests (e.g., zxtest); such code will ship as part of the CTS artifact.
CTS tests must not take dependencies on third party libraries that rely on the SDK for their Fuchsia support. Third party libraries that require SDK elements to support Fuchsia are going to be built against a particular SDK. We must make sure that our tests are as decoupled as possible from anyone else's SDK dependencies, as third party code may rely on platform features that we need to exclude from the tests. For example, if we rely on a test suite that heavily uses locking, it may be inappropriate for testing features of Zircon used to implement locking. Because of this restriction, we will use zxtest rather than gtest.
An artifact containing the CTS tests relevant to a given SDK will be published alongside that SDK. This artifact will also contain build system support sufficient to build and run the CTS tests outside of the tree. It will not contain a toolchain.
The tests must exercise language support thoroughly. See the section on Language Support for more details.
All updates to Fuchsia platform surface area elements should include tests that exercise the documented surface. This includes, but is not limited to, C/C++ headers, FIDL API, the FIDL wire format, and any surface described by the Fuchsia System Interface document. If the surface area element can be accessed by developers via an SDK, it must be tested.
We recognize that it may not be practical to require tests at this point. As the CTS and platform grows, we expect this requirement will become more strict.
Almost all changes that require API review should have CTS tests, and API reviewers should review with that in mind. The final review will be made by testability reviewers, who should only approve platform surface area changes if they are appropriately covered by CTS tests.
All tests are subject to the same review requirements as any other code submitted to the tree. Note that this does not mean that tests must be run as part of the commit queue, although we expect most will be. Examples of tests that might not be run as part of the commit queue include manual tests and tests that take longer than the commit queue allows.
The structure of the //sdk/cts/tests directory mirrors the structure of released SDKs. Tests go in the directory that mirrors the one where the interface under test is found in an SDK. For example:
//sdk/cts/tests/tools//sdk/cts/tests/fidl. For example, tests for fuchsia.sysmem should go in //sdk/cts/tests/fidl/fuchsia.sysmem.//sdk/cts/tests/pkg. For example, tests for async-loop should go in //sdk/cts/tests/pkg/async-loop.If Fuchsia developers are not clear on where to put their tests, they should consult the OWNERS of the relevant directory.
CTS tests target API and ABI that are available through externally-available SDKs. Build support ensures that tests only depend on API elements that are either available via an SDK, or allowlisted for use within the CTS. All build targets that are not allowlisted must use the cts_ rule variants found in //sdk/cts/build instead of the standard fuchsia.git rules (i.e., use cts_fuchsia_component, cts_executable, and so on).
The allowlist for non-SDK code can be found in //sdk/cts/build/allowed_cts_deps.gni. Test authors who believe they need an additional inclusion should reach out to the OWNERS of this directory.
All API tests must be written in languages supported by the SDK they test. In most cases, this implies C++. ABI tests may be written in any language; in order to avoid having to build external support for languages we do not support via the SDK, if an ABI test needs to be in another language, we will include it as a prebuilt binary or package (whichever is more appropriate).
Tests for particular headers must be written in a language that supports that header. As of this writing, C headers target C11 and C++11 and above, and C++ headers target C++14 and above.
CTS tests may restrict themselves to a particular language version. For example, we may decide that particular tests are restricted to C++14 in order to guarantee that headers maintain C++14 compatibility.
Language restrictions for target-side tests are not applicable to host-side tests. The language for host-side tests is test-specific. If it will require the CTS to depend on a new toolchain, the decision should be made in consultation with the CTS team. For end-to-end tests and scripts that run on the host, as of this writing, we support the use of Dart (and, specifically sl4f). As supported languages change, documentation will be made available about which languages are supported for host-side testing.
Tests should contain a check for every documented assertion about a particular API or ABI. For example, if we have a class fit::basic_string_view, and it has a method size that is documented to return the size of the string_view, we would have a test that creates a string_view, calls the size method, and asserts that the return value is correct.
We recognize that it may be difficult to do this in some cases, and that some tests may require specific device setup that may be hard to replicate. We recommend that developers start working on testing early in the development cycle. The long-term goal is to make CTS testing a requirement for all changes to the platform surface area.
Tests should reflect best practices about the usage of a given API. Informally, if an end developer copies a test's usage of the API, the test author would believe that developer is using the API correctly. Tests should, to the extent possible, not depend on undocumented, application-specific invariants. In the future, in the case of widespread use of undocumented behaviors outside of the Fuchsia tree, we may need to add tests for use cases that do not follow recommended usages.
Wherever possible, tests should avoid creating test doubles (e.g., mocks and fakes) for the internal state of the target device. The intent of the CTS is to make sure the entire device behaves correctly, not to make sure that a particular component behaves correctly in isolation.
However, this does not mean that CTS tests cannot benefit from fakes in some environments. For example, for the purposes of using CTS tests to ensure platform stability, we may find it useful to exercise tests that require real hardware or manual input, such audio or connectivity tests, in an automated environment that does not have those features available. While a CTS test itself should avoid the use of test doubles, the device under test can use fake drivers that feed the test fake data. CTS tests can rely on such drivers in cases where using real hardware is not practical.
If necessary, tests may require manual intervention to pass. We recommend that developers thoroughly investigate the possibility of automation.
CTS artifacts will be generated alongside the SDK artifacts that contain the relevant platform surface elements. Because of the soft transition requirements of RFC-0002, we expect that every SDK build will successfully execute the CTS associated with the previous build of the same SDK. As a proof of concept, we will implement infrastructure to guarantee this.
CTS artifacts will contain a test harness and build rules for gn. They will not contain a build system or toolchain; this must be supplied in the test execution environment. We will document which toolchains are known to be compatible with a given CTS.
Test examples can be found in fuchsia.git at //sdk/cts/.
This change will have the following performance impact:
Because changes associated with this RFC are test-only, they are a low security risk. Tests are not expected to interact with untrusted data from external sources.
Because changes associated with this RFC are test-only, they are a low privacy risk. Tests are not expected to interact with user data.
This proposal will increase the testing matrix for the platform. For example, given the six-week ABI stability guarantee, all ABI tests from the CTS generated six weeks earlier than a given build should be run and complete successfully against that build.
The new requirements in this proposal will also increase the overall number of platform tests.
As many required properties of the test framework as is practical will be enforced automatically; for example, the framework will automatically check that only allowed dependencies are included.
Documentation on how to write CTS tests will be included in //docs. There will be updates to testability and API process documents to reflect new CTS test authorship requirements. The steps needed to run CTS out of tree will be documented, so that end developers and system integrators can do them independently.
The chief drawback of this proposal is that it creates a significant new testing requirement for all changes to the platform surface area.
It is not a goal of the CTS effort to provide a complete solution to evolution and backwards compatibility issues. APIs and ABIs will have to be designed carefully to ensure that developers can migrate their code at a reasonable cost. For example, the FIDL team evolves language bindings with extreme care: they have a clear specification for how bindings ought to work, and actively tracks how conformant the various bindings are.
The CTS approach is a standard industry approach to maintaining backwards compatibility. Other approaches include:
Android solves this problem by releasing a CTS with their product. Developers of new Android devices must ensure that their devices pass the CTS.
As part of its Windows Hardware Compatibility Program, Microsoft produces a Windows Hardware Lab Kit that they distribute to developers of new Windows hardware.