docs/development/testing/ctf/contributing_tests.md

# Contributing Tests to CTF

Note: Compatibility testing documents are being overhauled in Q2
2024.  Some information in this document is out of date. See
https://fxbug.dev/339893257 for details.

This guide explains how to [add](#add-a-test) and [remove](#remove-a-test) [CTF]
tests.

## How to add a test to CTF {#add-a-test}

Please read the CTF [test requirements](#test-requirements) before getting
started. The most notable requirement is that a test must use a
[test realm factory] before this guide can be followed.

A test is included in CTF if it is in the group at [//sdk/ctf/tests:tests].
Before adding the test to this group, it's build rules must be rewritten to use
CTF versions. This is an unfortunate but necessary step that will go away soon.

To add a test to this group, perform each of the steps below in the BUILD.gn
file containing the test's `fuchsia_test_component` target.

### 1. Import CTF build rules

```gn
import("//sdk/ctf/build/ctf.gni")
```

### 2. Use CTF GN templates {#the-ctf-templates}

Replace the test's `fuchsia_package` declaration with `ctf_fuchsia_package`:

* {Before}

  ```gn
  fuchsia_package("fuchsia-example-tests") {
    testonly = true
    package_name = "fuchsia-example-tests"
    deps = [ ":fuchsia-example-test-component" ]
  }
  ```

* {After}

  ```gn
  ctf_fuchsia_package("fuchsia-example-tests") {
    testonly = true
    package_name = "fuchsia-example-tests"
    deps = [ ":fuchsia-example-test-component" ]
  }
  ```

### 3. Add the test release archive to the build graph {#the-test-archive}

`ctf_fuchsia_package` generates a `${target_name}_archive` target that produces
a FAR archive of the test. This archive is what gets released in CTF. Add a
`group("tests")` target to the BUILD.gn file if it doesn't exist, then add the
archive as a dependency:

```gn
group("tests") {
  testonly = true
  deps = [
    ":fuchsia-example-tests_archive",
    ...
  ]
}
```

You successfully added a `group("tests")` to the `BUILD.gn` file. You are now
ready to add that group to [//sdk/ctf/tests:tests].

### 4. Add a GN template to build the prebuilt CTF test {#the-gn-template}

To teach CTF how to build the test's prebuilt package into a runnable target,
create a GN template that generates a package identical to the
`fuchsia_test_package` used to run the latest version of the test. For example,
if the test root component and test package look like this:

```gn
fuchsia_test_component("fuchsia-example-test-root") {
  testonly = true
  manifest = "meta/fuchsia-example-test-root.cml"
  test_type = "ctf"
}

fuchsia_test_package("fuchsia-example-tests-latest") {
  test_components = [ ":fuchsia-example-test-root" ]
  subpackages = [
    ":fuchsia-example-tests", # latest version of the test suite.
    ":fuchsia-example-test-realm-factory",
  ]
  deps = [ ":fuchsia-example-test-helper" ]
}
```

Then add this template to [//sdk/ctf/build/generate_ctf_tests.gni]

```gn
template("generate_fuchsia-example-tests") {
  forward_variables_from(invoker, [ "test_info" ])
  fuchsia_package_with_test(target_name) {
    test_component = "{{ '<var>' }}//path/to/test{{ '</var>' }}:fuchsia-example-test-root",
    test_component_name = "test-root.cm"
    subpackages = [
      test_info.target_label, # prebuilt version of the test suite.
      "{{ '<var>' }}//path/to/test{{ '</var>' }}:fuchsia-example-test-realm_factory"
    ]
    deps = [ "{{ '<var>' }}//path/to/test{{ '</var>' }}:fuchsia-example-test-helper" ]
  }
}
```

Note:

* For CTF to match this template with the prebuilt test package, this template
  _must_ be named as `generate_{package_name}` and `package_name` must match
  the test's original `ctf_fuchsia_package` name.
* This template subpackages `test_info.target_label` instead of
  `:fuchsia-example-tests` because the former points to the prebuilt version of
  the test from CIPD and the latter points to the latest version built from
  source.

### 5. Test the changes

To verify that these steps were completed correctly, run these commands:

```sh
fx set core.x64 --with //sdk/ctf:ctf_artifacts
fx build
```

The build should show an error prompt to run a command that updates
[//sdk/ctf/goldens/package_archives.json]. Run that command, then run
`fx build` again.

At this stage the build will print an error if the GN template defined in the
previous step is missing or if it cannot match the prebuilt version of the test
with the GN template. If everything succeeds, the test can be run using this
command:

```sh
fx test fuchsia-example-test_ctf<N>
```

Given that this is the version of the prebuilt test from the current build rather
than some version from a previous CTF release, `N` is the Fuchsia platform's
currently in-development API level.

### 6. Submit the changes

Send the changes for review. After submission the test will automatically be
included in the next CTF release when the next milestone branch is cut. For
additional review, please reach out to <fuchsia-ctf-team@google.com>.

## How to remove a test from CTF {#remove-a-test}

To remove a test from future CTF releases:

1. Remove the [test archive](#the-test-archive) from the build graph.
2. Delete the test's [gn template](#the-gn-template).
3. Remove the [ctf templates](#the-ctf-templates) from the test's build rules.

Submit these changes to the main branch. Over time all of the API levels
corresponding to each release will become unsupported and the last version of
the test will stop running in CQ.

If you must immediately remove a test and all of its prebuilt version from CQ,
as an additional step you should follow Fuchsia's change control process and
cherry pick a CL that removes the test from the each of the corresponding
release branches. For example, If you want to remove a test from the CTF release
for API level 20, your CL needs to be cherry picked onto the `releases/f20`
branch.

## Test requirements {#test-requirements}

Every test must meet these requirements before being added to CTF.

### CTF tests must use the test realm factory pattern

The [test realm factory] pattern allows the build to version the test suite
component in CTF without versioning the component(s) under test. Please read the
test realm factory guide and refactor the test if necessary before following
this guide.

### CTF tests must only depend on partner-facing ABIs

A CTF test must only depend on software in the partner [SDK category] at
runtime because the test will force its dependencies to remain stable. This is
not enforced by presubmit checks. Usually test authors just need to verify that
the FIDL protocols their test suite components use are all available in the
`partner` SDK category.

### CTF tests must be written in C, C++, or Rust

At the time of writing CTF only supports these languages.

<!-- Links. Please link source code to https://cs.opensource.google -->
[CTF]: /docs/development/testing/ctf/compatibility_testing.md
[SDK category]: /docs/contribute/sdk/categories.md
[test realm factory]: /docs/development/testing/components/test_realm_factory.md
[//sdk/ctf/tests:tests]: https://cs.opensource.google/fuchsia/fuchsia/+/main:sdk/ctf/tests/BUILD.gn
[//sdk/ctf/build/generate_ctf_tests.gni]: https://cs.opensource.google/fuchsia/fuchsia/+/main:sdk/ctf/build/generate_ctf_tests.gni
[//sdk/ctf/goldens/package_archives.json]: https://cs.opensource.google/fuchsia/fuchsia/+/main:sdk/ctf/goldens/package_archives.json