docs/contribute/governance/rfcs/0015_cts.md

{% set rfcid = "RFC-0015" %}
{% include "docs/contribute/governance/rfcs/_common/_rfc_header.md" %}
# {{ rfc.name }}: {{ rfc.title }}
<!-- SET the `rfcid` VAR ABOVE. DO NOT EDIT ANYTHING ELSE ABOVE THIS LINE. -->

## Summary

This document presents requirements, design, and implementation strategy for a
set of Compatibility Tests for Fuchsia (CTF).  The CTF will offer a way
of testing platform implementations to make sure that they behave according to
Fuchsia's specifications. Fuchsia developers will write tests that guarantee
compatibility across changes to both the source and behavior of the platform.
When these tests pass, they will guarantee that a particular release, when
run on a particular device, is compatible with a target API level and target ABI
revision, as defined in [RFC-0002].

For the purposes of this document, we refer to API and ABI jointly as the
_platform surface area_. A _platform surface area element_ is a named and
versioned identifier associated with the platform surface area (e.g., a method
name). Future RFCs may formalize these definitions.

## Motivation

All of the open-source tests for Fuchsia platform behavior are currently
(December 2020) built and run as part of the platform build. As the platform
evolves, we keep the tests passing at head. As a result, we have no tests that
guarantee backwards compatibility with older versions of the platform.

Currently, we use a number of product tests to ensure compatibility and
stability. These product tests are difficult to triage: because they rely on
the stability of the product, and target many different parts of the platform,
it is difficult for platform engineers to determine where the bug may be.

At the same time, developers are writing code that targets older versions of the
Fuchsia platform surface area. In this document, we refer to such developers as
_end developers_.

As we roll out API breaking changes, we have no safeguards in place that raise a
flag when we break compatibility with end developers' code. Over the course of
the project, undocumented API changes have frequently been released that cause
external code to stop building.

Furthermore, we are currently building out strong guarantees about backwards ABI
compatibility. As of 9 November 2020, we require a six-week ABI compatibility
window, but have no enforcement mechanism for it.

We need a set of tests that we can execute independently from the platform build
that identify clearly when we break our contracts with end developers. This
will help ensure that we maintain compatibility with externally developed code,
and provide more easily triaged, targeted test coverage for the parts of the
platform that are currently only exercised by product tests.

In the long term, we will also need a set of tests that system integrators can
execute to know if they are producing a compliant Fuchsia implementation.

Fuchsia's CTF will offer a way of testing platform implementations to make sure
that they are compatible with particular platform releases (as defined in
[RFC-0002]). We aspire to have a test for each documented behavior in the
platform surface area.

When we create a release, we can use the CTF to tell us about the compatibility
of its surface area with that of other release versions.

When someone is developing a device running Fuchsia, and wants to see if it is
compatible with a given SDK, they can take the CTF and the SDK with which it
wants to demonstrate compatibility, pass the tests, and have confidence that
their product correctly implements Fuchsia's semantics - it will be "Fuchsia
compatible".

When a developer wants to understand how to write code against a particular API,
or using a particular ABI, they will be able to use these tests as reference.

[RFC-0002] allows a platform implementation to provide partial support for
backwards compatibility. CTF will provide a way to test partial compatibility.

Note that the CTF is not intended as a complete solution for platform evolution
and backwards compatibility. It's not likely that CTF tests will cover every
last use case. API and ABI will still have to be designed with future use cases
in mind. See the section on [drawbacks and
alternatives](#drawbacks_alternatives_and-unknowns) for additional discussion.

## Design

The CTF design involves balancing ease of development with the need to build and
run the CTF itself outside of the Fuchsia repository. The requirements are as
follows:

1. There should be a CTF test for every documented behavior of every platform
   surface area element. Although we expect this to become a hard requirement
   eventually, this RFC does not specify such a requirement.

1. CTF tests may not rely on any internal details of a particular system image.
   To the extent that they rely on other platform code that is not itself
   subject to test, that code must be bundled as part of the CTF and also not
   rely on any internal details of a particular system image.

1. CTF tests must be updated by developers (that is, tests must be added or
   modified) when adding or changing elements of the platform surface area.

1. It must be possible to determine the API level and ABI revision of Fuchsia
   that a given CTF artifact targets.

1. CTF tests that are not included in the test suite as prebuilt artifacts must
   be written in languages supported by the SDK being used to test (see the
   [supported languages document] and the [Language](#Language) section below
   for more details).

### Authoring the Tests

We develop CTF tests alongside their corresponding SDK elements. Today, that
means we develop the tests in fuchsia.git. While it would be nice if CTF
developers had the same experience as the out of tree developers who use the
SDK, there are too many advantages to in-tree development to ignore:

1. Because feature development is done alongside test development, in-tree
   development of the tests will allow test authors to use a workflow with which
   they are familiar, as well as submit the test in the same CL as the feature.

1. Because the feature will be submitted at the same time as the test, there is
   no need for any machinery to align the CTF and the version that it qualifies.

CTF tests will use build-time enforcement to ensure that CTF tests can only
depend on SDK elements or other pre-approved CTF code. One of the dangers of
developing in-tree is that we may accidentally take on dependencies on platform
implementation details that are not exposed via the SDK. CTF tests must only
access publicly facing elements of the platform to prevent accidentally relying
on implementation details. CTF tests may use platform code that is appropriate
for writing tests (e.g., zxtest); such code will ship as part of the CTF
artifact.

CTF tests must not take dependencies on third party libraries that rely on the
SDK for their Fuchsia support. Third party libraries that require SDK elements
to support Fuchsia are going to be built against a particular SDK. We must make
sure that our tests are as decoupled as possible from anyone else's SDK
dependencies, as third party code may rely on platform features that we need to
exclude from the tests. For example, if we rely on a test suite that heavily
uses locking, it may be inappropriate for testing features of Zircon used to
implement locking. Because of this restriction, we will use zxtest rather than
gtest.

An artifact containing the CTF tests relevant to a given SDK will be published
alongside that SDK. This artifact will also contain build system support
sufficient to build and run the CTF tests outside of the tree. It will not
contain a toolchain.

The tests must exercise language support thoroughly. See the section on
[Language Support](#language) for more details.

## Implementation

### Coverage Requirements

All updates to Fuchsia platform surface area elements should include tests that
exercise the documented surface. This includes, but is not limited to, C/C++
headers, FIDL API, the FIDL wire format, and any surface described by the
[Fuchsia System Interface] document. If the surface area element can be
accessed by developers via an SDK, it must be tested.

We recognize that it may not be practical to require tests at this point. As
the CTF and platform grows, we expect this requirement will become more strict.

Almost all changes that require API review should have CTF tests, and API
reviewers should review with that in mind. The final review will be made by
testability reviewers, who should only approve platform surface area changes if
they are appropriately covered by CTF tests.

All tests are subject to the same review requirements as any other code
submitted to the tree. Note that this does not mean that tests must be run as
part of the commit queue, although we expect most will be. Examples of tests
that might not be run as part of the commit queue include manual tests and tests
that take longer than the commit queue allows.

#### Directory structure

The structure of the `//sdk/ctf/tests` directory mirrors the structure of
released SDKs. Tests go in the directory that mirrors the one where the
interface under test is found in an SDK. For example:

  * Tests for host tools should go in `//sdk/ctf/tests/tools`
  * Tests for FIDL interfaces should go in the appropriate
    subdirectory of `//sdk/ctf/tests/fidl`. For example, tests for
    `fuchsia.sysmem` should go in `//sdk/ctf/tests/fidl/fuchsia.sysmem`.
  * Tests for libraries should go in the appropriate subdirectory of
    `//sdk/ctf/tests/pkg`. For example, tests for `async-loop` should go in
    `//sdk/ctf/tests/pkg/async-loop`.

If Fuchsia developers are not clear on where to put their tests, they should
consult the OWNERS of the relevant directory.

#### Build support

CTF tests target API and ABI that are available through externally-available
SDKs. Build support ensures that tests only depend on API elements that are
either available via an SDK, or allowlisted for use within the CTF. All build
targets that are not allowlisted must use the `cts_` rule variants found in
`//sdk/ctf/build` instead of the standard fuchsia.git rules (i.e., use
`ctf_fuchsia_component`, `ctf_executable`, and so on).

The allowlist for non-SDK code can be found in
`//sdk/ctf/build/allowed_ctf_deps.gni`. Test authors who believe they need an
additional inclusion should reach out to the OWNERS of this directory.

#### Language

##### Target-side tests

All API tests must be written in languages supported by the SDK they test. In
most cases, this implies C++. ABI tests may be written in any language; in
order to avoid having to build external support for languages we do not support
via the SDK, if an ABI test needs to be in another language, we will include it
as a prebuilt binary or package (whichever is more appropriate).

Tests for particular headers must be written in a language that supports that
header. As of this writing, C headers target C11 and C++11 and above, and C++
headers target C++14 and above.

CTF tests may restrict themselves to a particular language version. For
example, we may decide that particular tests are restricted to C++14 in order to
guarantee that headers maintain C++14 compatibility.

##### Host-side tests

Language restrictions for target-side tests are not applicable to host-side
tests. The language for host-side tests is test-specific. If it will require
the CTF to depend on a new toolchain, the decision should be made in
consultation with the CTF team. For end-to-end tests and scripts that run on
the host, as of this writing, we support the use of Dart (and, specifically
`sl4f`). As supported languages change, documentation will be made available
about which languages are supported for host-side testing.

#### Test Requirements

Tests should contain a check for every documented assertion about a particular
API or ABI. For example, if we have a class `fit::basic_string_view`, and it
has a method `size` that is documented to return the size of the string_view, we
would have a test that creates a string_view, calls the `size` method, and
asserts that the return value is correct.

We recognize that it may be difficult to do this in some cases, and that some
tests may require specific device setup that may be hard to replicate. We
recommend that developers start working on testing early in the development
cycle. The long-term goal is to make CTF testing a requirement for all changes
to the platform surface area.

Tests should reflect best practices about the usage of a given API. Informally,
if an end developer copies a test's usage of the API, the test author would
believe that developer is using the API correctly. Tests should, to the extent
possible, not depend on undocumented, application-specific invariants. In the
future, in the case of widespread use of undocumented behaviors outside of the
Fuchsia tree, we may need to add tests for use cases that do not follow
recommended usages.

Wherever possible, tests should avoid creating test doubles (e.g., mocks and
fakes) for the internal state of the target device. The intent of the CTF is to
make sure the entire device behaves correctly, not to make sure that a
particular component behaves correctly in isolation.

However, this does not mean that CTF tests cannot benefit from fakes in some
environments.  For example, for the purposes of using CTF tests to ensure
platform stability, we may find it useful to exercise tests that require real
hardware or manual input, such audio or connectivity tests, in an automated
environment that does not have those features available.  While a CTF test
itself should avoid the use of test doubles, the device under test can use fake
drivers that feed the test fake data.  CTF tests can rely on such drivers in
cases where using real hardware is not practical.

In addition, CTF tests should maintain isolation from each other. If one test
fails, or if one aspect of the system under test misbehaves, then ideally this
failure should be localized rather than affect other tests. Lack of isolation
between tests is sometimes referred to as "test cross-talk". For instance
consider a test that changes global state in a device settings component. If
the test fails to restore the original state before it terminates, or if
another component on the system changes the global state during test execution,
then cross-talk may occur. For such a test to be isolated, the test author
might consider creating an affordance for locally-scoped settings rather than
mutating global state.

If necessary, tests may require manual intervention to pass. We recommend that
developers thoroughly investigate the possibility of automation.

### Deployment

CTF artifacts will be generated alongside the SDK artifacts that contain the
relevant platform surface elements. Because of the soft transition requirements
of RFC-0002, we expect that every SDK build will successfully execute the CTF
associated with the previous build of the same SDK. As a proof of concept, we
will implement infrastructure to guarantee this.

CTF artifacts will contain a test harness and build rules for gn. They will not
contain a build system or toolchain; this must be supplied in the test execution
environment. We will document which toolchains are known to be compatible with
a given CTF.

### Examples

Test examples can be found in fuchsia.git at `//sdk/ctf/`.

## Performance

This change will have the following performance impact:

 * An increase in time to run all platform tests, stemming from an increased
   number of tests.
 * No impact on production performance, because the changes are test-only.

## Security considerations

Because changes associated with this RFC are test-only, they are a low security
risk. Tests are not expected to interact with untrusted data from external
sources.

## Privacy considerations

Because changes associated with this RFC are test-only, they are a low privacy
risk. Tests are not expected to interact with user data.

## Testing

This proposal will increase the testing matrix for the platform. For example,
given the six-week ABI stability guarantee, all ABI tests from the CTF generated
six weeks earlier than a given build should be run and complete successfully
against that build.

The new requirements in this proposal will also increase the overall number of
platform tests.

As many required properties of the test framework as is practical will be
enforced automatically; for example, the framework will automatically check that
only allowed dependencies are included.

## Documentation

Documentation on how to write CTF tests will be included in `//docs`. There
will be updates to testability and API process documents to reflect new CTF test
authorship requirements. The steps needed to run CTF out of tree will be
documented, so that end developers and system integrators can do them
independently.

## Drawbacks, alternatives, and unknowns

The chief drawback of this proposal is that it creates a significant new testing
requirement for all changes to the platform surface area.

It is not a goal of the CTF effort to provide a complete solution to evolution
and backwards compatibility issues. APIs and ABIs will have to be designed
carefully to ensure that developers can migrate their code at a reasonable cost.
For example, the FIDL team evolves language bindings with extreme care: they
have a [clear specification for how bindings ought to
work](/docs/reference/fidl/language/bindings-spec.md), and actively [tracks how
conformant the various bindings
are](/docs/development/languages/fidl/guides/compatibility/README.md).

The CTF approach is a standard industry approach to maintaining backwards
compatibility. Other approaches include:

 * Simply being careful. We know empirically that this does not work by itself.
 * Not evolving the platform. Obviously, simply never making changes is not
   practical. Most scaled down versions of this (for example, shipping most of
   an application's dependencies with it, or providing a virtual environment for
   every application) are at odds with Fuchsia's design principles and product
   goals.
 * Formal verification. We do not consider formal verification to be a scalable
   alternative to testing.

## Prior art and references

Android solves this problem by releasing a
[CTF](https://source.android.com/compatibility/cts) with their product.
Developers of new Android devices must ensure that their devices pass the CTF.

As part of its [Windows Hardware Compatibility
Program](https://docs.microsoft.com/en-us/windows-hardware/design/compatibility/),
Microsoft produces a [Windows Hardware Lab
Kit](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/) that they
distribute to developers of new Windows hardware.

<!-- xrefs -->
[RFC-0002]: /docs/contribute/governance/rfcs/0002_platform_versioning.md
[supported languages document]: /docs/contribute/governance/policy/programming_languages.md
[Fuchsia System Interface]: /docs/concepts/packages/system.md
[Fuchsia language policy]: /docs/contribute/governance/policy/programming_languages.md