docs/development/testing/ctf/user_guide.md - fuchsia - Git at Google


            
            # User guide for CTF


            This user guide provides a comprehensive overview of Compatibility Testing

            
            for Fuchsia (CTF), a mechanism for freezing artifacts on release branches

            
            and loading them on the `main` branch for testing purposes.


            The most common use case of CTF is to freeze a test on a release branch and

            
            then run that test against components on the `main` branch in CI/CQ. This

            
            prevents backwards-incompatible changes that would break pre-built clients.


            To learn more about CTF as a whole, see the [overview] and [motivation]

            
            documents.


            ## Basic usage


            This section describes how to use CTF and how to handle common scenarios.


            ### Freezing a Fuchsia Package


            To freeze a Fuchsia package called {{ '<var>' }}my-package{{ '</var>' }}, do the following:


            1. Change the package from `fuchsia_package` to `ctf_fuchsia_package`.

            
            2. Include the `{PACKAGE_NAME}_archive` target in

            
            [//sdk/ctf/tests:tests](https://fuchsia.googlesource.com/fuchsia/+/HEAD/sdk/ctf/tests/BUILD.gn#7).


            Example:


            ```gn

            
            # Old src/{{ '<var>' }}my-package{{ '</var>' }}/BUILD.gn

            
            import("//build/components/fuchsia_package.gni")


            fuchsia_package("{{ '<var>' }}my-package{{ '</var>' }}") {

            
              # ...

            
            }

            
            ```


            Change your file to this:


            ```gn

            
            # New src/{{ '<var>' }}my-package{{ '</var>' }}/BUILD.gn

            
            import("//sdk/ctf/build/ctf.gni")


            ctf_fuchsia_package("{{ '<var>' }}my-package{{ '</var>' }}") {

            
              # ...

            
            }

            
            ```


            And update `//sdk/ctf/tests/BUILD.gn`:


            ```gn

            
            group("tests") {

            
              deps = [

            
                # ...

            
                "//src/{{ '<var>' }}my-package{{ '</var>' }}:{{ '<var>' }}my-package{{ '</var>' }}_archive",

            
              ]

            
            }

            
            ```


            You can also create a `group("ctf_tests")` in your local `BUILD.gn`,

            
            and reference that in `//sdk/ctf/tests/BUILD.gn`.


            This will result in your package being compiled and stored in

            
            `ctf-artifacts`

            
            when the next Fuchsia milestone is released. After completing this step, you

            
            **must** include instructions for loading your package (see next section).


            ### Loading a Fuchsia Package


            To load a frozen Fuchsia package you need to update

            
            [`generate_ctf_tests.gni`][generate_ctf_tests]

            
            to include instructions for loading your package.


            You will need to define a new template in the `generate_ctf_tests.gni`

            
            file that is named `generate_{{ '<var>' }}my-package{{ '</var>'

            
            }}`, and this template must expand to a Fuchsia test package.


            The template will receive a parameter called `test_info` which

            
            includes the downloaded package's label as `test_info.target_label`.


            For example:


            ```gn

            
            template("generate_{{ '<var>' }}my-package{{ '</var>' }}") {

            
              forward_variables_from(invoker, [ "test_info" ])

            
              fuchsia_package_with_test(target_name) {

            
                test_component = "//src/{{ '<var>' }}my-package{{ '</var>' }}:{{ '<var>' }}my-package{{ '</var>' }}-test-root",

            
                test_component_name = "test-root.cm"

            
                subpackages = [

            
                  test_info.target_label, # prebuilt version of the test suite.

            
                  # other packages needed to implement the test, such as a RealmFactory.

            
                ]

            
              }

            
            }

            
            ```


            ### Freezing a host test


            To freeze a host test do the following:


            1. Change the build rule from `host_test` to `ctf_host_test`.

            
            2. Include that target in

            
            [//sdk/ctf/tests:tests](https://fuchsia.googlesource.com/fuchsia/+/HEAD/sdk/ctf/tests/BUILD.gn#7).


            Example:


            ```gn

            
            # Old src/{{ '<var>' }}my-host-test{{ '</var>' }}/BUILD.gn

            
            import("//build/testing/host_test.gni")


            host_test("{{ '<var>' }}my-host-test{{ '</var>' }}") {

            
              # ...

            
            }

            
            ```


            Change your file to this:


            ```gn

            
            # New src/{{ '<var>' }}my-host-test{{ '</var>' }}/BUILD.gn

            
            import("//sdk/ctf/build/ctf.gni")


            ctf_host_test("{{ '<var>' }}my-host-test{{ '</var>' }}") {

            
              # ...

            
            }

            
            ```


            And update `//sdk/ctf/tests/BUILD.gn`:


            ```gn

            
            group("tests") {

            
              deps = [

            
                # ...

            
                "//src/{{ '<var>' }}my-host-test{{ '</var>' }}:{{ '<var>' }}my-host-test{{ '</var>' }}",

            
              ]

            
            }

            
            ```


            You can also create a `group("ctf_tests")` in your local `BUILD.gn`,

            
            and reference that in `//sdk/ctf/tests/BUILD.gn`.


            This will result in your host test and its dependencies being bundled

            
            in `ctf-artifacts` when the next Fuchsia milestone is released.

            
            After completing this step, your test will automatically run in CTF

            
            using any arguments you pass to `ctf_host_test`.


            ## Basic concepts


            CTF consists of two mechanisms:


            1. Build rules to freeze an artifact and upload it to [CIPD].

            
            1. [Build rules][generate_ctf_tests] to thaw that artifact and turn

            
               it into a test.


            The freezing and loading process is exercised at the tip of the `fuchsia.git`

            
            repository for local tests through the `//sdk/ctf` build target. Including

            
            `--with-test //sdk/ctf` in your `fx set` args results in the following

            
            build output:


            ```bash

            
            {{ '<var>' }}$FUCHSIA_DIR/out/default{{ '</var>' }}/cts/host_test_manifest.json

            
            {{ '<var>' }}$FUCHSIA_DIR/out/default{{ '</var>' }}/cts/package_archives.json

            
            {{ '<var>' }}$FUCHSIA_DIR/out/default{{ '</var>' }}/cts/*.far       # Packages referenced in manifest

            
            {{ '<var>' }}$FUCHSIA_DIR/out/default{{ '</var>' }}/cts/host_x64/*  # Host tests referenced in manifest

            
            ```


            This results in both the original tests and the thawed version of those tests

            
            being included in your build:


            ```bash

            
            $ fx test --dry

            
            ...

            
              fuchsia-pkg://fuchsia.com/fuchsia-diagnostics-tests-latest#meta/fuchsia-diagnostics-tests-root.cm

            
              fuchsia-pkg://fuchsia.com/fuchsia-driver-test_tests-package#meta/test-root.cm

            
            ...

            
              fuchsia-pkg://fuchsia.com/fuchsia-diagnostics-tests_ctf_in_development#meta/fuchsia-diagnostics-tests-root.cm

            
              fuchsia-pkg://fuchsia.com/fuchsia-driver-test_tests_ctf_in_development#meta/test-root.cm

            
            ```


            Rules defined in [`generate_ctf_tests.gni`][generate_ctf_tests] are

            
            responsible for thawing packages referenced in `package_archives.json`

            
            by turning them into normal Fuchsia test packages. Since these

            
            packages were thawed from the CTF build at the tip of tree, they

            
            all have a `ctf_in_development` suffix. All of these tests can be

            
            run using `fx test`.


            Each Fuchsia milestone release (for example, F16 and F17) is

            
            associated with a release branch in git (for example,

            
            [`refs/heads/releases/F16`][release-f16]. The process above happens

            
            for each change to a release branch, resulting in a new `ctf`

            
            directory containing packages and host tests referenced in manifests.

            
            On release branches, however, the resulting directory is zipped and

            
            uploaded to [CIPD] where it is downloaded as a prebuilt by

            
            `jiri run-hooks`. You can find these downloaded prebuilts in your

            
            `fuchsia.git` checkout at `prebuilt/ctf/f*`, where there is one

            
            directory per Fuchsia milestone release. The source of truth for

            
            which milestones' prebuilts are downloaded is

            
            [`version_history.json`][version-history], which lists `supported`

            
            API levels to test.


            The build target defined at `//sdk/ctf/release:tests` iterates over

            
            the downloaded prebuilts and applies the

            
            [`generate_ctf_tests.gni`][generate_ctf_tests] rules to thaw each

            
            downloaded package in the same way that the in-development packages

            
            were thawed. They can be included using `--with-test

            
            //sdk/ctf/release:tests` in your `fx set` args.


            This will add new tests for each supported level to your build:


            ```bash

            
            $ fx test --dry

            
            ...

            
              fuchsia-pkg://fuchsia.com/fuchsia-diagnostics-tests_ctf18#meta/fuchsia-diagnostics-tests-root.cm

            
              fuchsia-pkg://fuchsia.com/fuchsia-driver-test_tests_ctf18#meta/test-root.cm

            
            ...

            
              fuchsia-pkg://fuchsia.com/fuchsia-diagnostics-tests_ctf19#meta/fuchsia-diagnostics-tests-root.cm

            
              fuchsia-pkg://fuchsia.com/fuchsia-driver-test_tests_ctf19#meta/test-root.cm

            
            ```


            Each thawed test has a suffix indicating the release branch where

            
            the frozen package originated (e.g. ctf18, ctf19 for F18 and F19

            
            respectively). All of these tests can be run using `fx test`.


            ## Troubleshooting failures


            CTF tests are designed to fail when there is an incompatibility

            
            between the frozen package and packages produced at tip of

            
            `fuchsia.git`, and this can cause the `core.x64-debug-ctf` builder

            
            to fail during CL submission. See the dedicated

            
            [troubleshooting guide][troubleshooting] for detailed instructions

            
            on handling failure scenarios to unblock CL submission.


            [CIPD]: https://chrome-infra-packages.appspot.com/p/fuchsia/cts/linux-amd64

            
            [generate_ctf_tests]: https://fuchsia.googlesource.com/fuchsia/+/HEAD/sdk/ctf/build/generate_ctf_tests.gni

            
            [motivation]: /docs/development/testing/ctf/compatibility_testing.md#motivation

            
            [overview]: /docs/development/testing/ctf/overview.md

            
            [troubleshooting]: /docs/development/testing/ctf/troubleshooting.md

            
            [version-history]: https://fuchsia.googlesource.com/fuchsia/+/HEAD/sdk/version_history.json

            
            [release-f16]: https://fuchsia.googlesource.com/fuchsia/+/refs/heads/releases/f16
	# User guide for CTF

	This user guide provides a comprehensive overview of Compatibility Testing
	for Fuchsia (CTF), a mechanism for freezing artifacts on release branches
	and loading them on the `main` branch for testing purposes.

	The most common use case of CTF is to freeze a test on a release branch and
	then run that test against components on the `main` branch in CI/CQ. This
	prevents backwards-incompatible changes that would break pre-built clients.

	To learn more about CTF as a whole, see the [overview] and [motivation]
	documents.

	## Basic usage

	This section describes how to use CTF and how to handle common scenarios.

	### Freezing a Fuchsia Package

	To freeze a Fuchsia package called {{ '<var>' }}my-package{{ '</var>' }}, do the following:

	1. Change the package from `fuchsia_package` to `ctf_fuchsia_package`.
	2. Include the `{PACKAGE_NAME}_archive` target in
	[//sdk/ctf/tests:tests](https://fuchsia.googlesource.com/fuchsia/+/HEAD/sdk/ctf/tests/BUILD.gn#7).

	Example:

	```gn
	# Old src/{{ '<var>' }}my-package{{ '</var>' }}/BUILD.gn
	import("//build/components/fuchsia_package.gni")

	fuchsia_package("{{ '<var>' }}my-package{{ '</var>' }}") {
	# ...
	}
	```

	Change your file to this:

	```gn
	# New src/{{ '<var>' }}my-package{{ '</var>' }}/BUILD.gn
	import("//sdk/ctf/build/ctf.gni")

	ctf_fuchsia_package("{{ '<var>' }}my-package{{ '</var>' }}") {
	# ...
	}
	```

	And update `//sdk/ctf/tests/BUILD.gn`:

	```gn
	group("tests") {
	deps = [
	# ...
	"//src/{{ '<var>' }}my-package{{ '</var>' }}:{{ '<var>' }}my-package{{ '</var>' }}_archive",
	]
	}
	```

	You can also create a `group("ctf_tests")` in your local `BUILD.gn`,
	and reference that in `//sdk/ctf/tests/BUILD.gn`.

	This will result in your package being compiled and stored in
	`ctf-artifacts`
	when the next Fuchsia milestone is released. After completing this step, you
	must include instructions for loading your package (see next section).

	### Loading a Fuchsia Package

	To load a frozen Fuchsia package you need to update
	[`generate_ctf_tests.gni`][generate_ctf_tests]
	to include instructions for loading your package.

	You will need to define a new template in the `generate_ctf_tests.gni`
	file that is named `generate_{{ '<var>' }}my-package{{ '</var>'
	}}`, and this template must expand to a Fuchsia test package.

	The template will receive a parameter called `test_info` which
	includes the downloaded package's label as `test_info.target_label`.

	For example:

	```gn
	template("generate_{{ '<var>' }}my-package{{ '</var>' }}") {
	forward_variables_from(invoker, [ "test_info" ])
	fuchsia_package_with_test(target_name) {
	test_component = "//src/{{ '<var>' }}my-package{{ '</var>' }}:{{ '<var>' }}my-package{{ '</var>' }}-test-root",
	test_component_name = "test-root.cm"
	subpackages = [
	test_info.target_label, # prebuilt version of the test suite.
	# other packages needed to implement the test, such as a RealmFactory.
	]
	}
	}
	```

	### Freezing a host test

	To freeze a host test do the following:

	1. Change the build rule from `host_test` to `ctf_host_test`.
	2. Include that target in
	[//sdk/ctf/tests:tests](https://fuchsia.googlesource.com/fuchsia/+/HEAD/sdk/ctf/tests/BUILD.gn#7).

	Example:

	```gn
	# Old src/{{ '<var>' }}my-host-test{{ '</var>' }}/BUILD.gn
	import("//build/testing/host_test.gni")

	host_test("{{ '<var>' }}my-host-test{{ '</var>' }}") {
	# ...
	}
	```

	Change your file to this:

	```gn
	# New src/{{ '<var>' }}my-host-test{{ '</var>' }}/BUILD.gn
	import("//sdk/ctf/build/ctf.gni")

	ctf_host_test("{{ '<var>' }}my-host-test{{ '</var>' }}") {
	# ...
	}
	```

	And update `//sdk/ctf/tests/BUILD.gn`:

	```gn
	group("tests") {
	deps = [
	# ...
	"//src/{{ '<var>' }}my-host-test{{ '</var>' }}:{{ '<var>' }}my-host-test{{ '</var>' }}",
	]
	}
	```

	You can also create a `group("ctf_tests")` in your local `BUILD.gn`,
	and reference that in `//sdk/ctf/tests/BUILD.gn`.

	This will result in your host test and its dependencies being bundled
	in `ctf-artifacts` when the next Fuchsia milestone is released.
	After completing this step, your test will automatically run in CTF
	using any arguments you pass to `ctf_host_test`.

	## Basic concepts

	CTF consists of two mechanisms:

	1. Build rules to freeze an artifact and upload it to [CIPD].
	1. [Build rules][generate_ctf_tests] to thaw that artifact and turn
	it into a test.

	The freezing and loading process is exercised at the tip of the `fuchsia.git`
	repository for local tests through the `//sdk/ctf` build target. Including
	`--with-test //sdk/ctf` in your `fx set` args results in the following
	build output:

	```bash
	{{ '<var>' }}$FUCHSIA_DIR/out/default{{ '</var>' }}/cts/host_test_manifest.json
	{{ '<var>' }}$FUCHSIA_DIR/out/default{{ '</var>' }}/cts/package_archives.json
	{{ '<var>' }}$FUCHSIA_DIR/out/default{{ '</var>' }}/cts/*.far # Packages referenced in manifest
	{{ '<var>' }}$FUCHSIA_DIR/out/default{{ '</var>' }}/cts/host_x64/* # Host tests referenced in manifest
	```

	This results in both the original tests and the thawed version of those tests
	being included in your build:

	```bash
	$ fx test --dry
	...
	fuchsia-pkg://fuchsia.com/fuchsia-diagnostics-tests-latest#meta/fuchsia-diagnostics-tests-root.cm
	fuchsia-pkg://fuchsia.com/fuchsia-driver-test_tests-package#meta/test-root.cm
	...
	fuchsia-pkg://fuchsia.com/fuchsia-diagnostics-tests_ctf_in_development#meta/fuchsia-diagnostics-tests-root.cm
	fuchsia-pkg://fuchsia.com/fuchsia-driver-test_tests_ctf_in_development#meta/test-root.cm
	```

	Rules defined in [`generate_ctf_tests.gni`][generate_ctf_tests] are
	responsible for thawing packages referenced in `package_archives.json`
	by turning them into normal Fuchsia test packages. Since these
	packages were thawed from the CTF build at the tip of tree, they
	all have a `ctf_in_development` suffix. All of these tests can be
	run using `fx test`.

	Each Fuchsia milestone release (for example, F16 and F17) is
	associated with a release branch in git (for example,
	[`refs/heads/releases/F16`][release-f16]. The process above happens
	for each change to a release branch, resulting in a new `ctf`
	directory containing packages and host tests referenced in manifests.
	On release branches, however, the resulting directory is zipped and
	uploaded to [CIPD] where it is downloaded as a prebuilt by
	`jiri run-hooks`. You can find these downloaded prebuilts in your
	`fuchsia.git` checkout at `prebuilt/ctf/f*`, where there is one
	directory per Fuchsia milestone release. The source of truth for
	which milestones' prebuilts are downloaded is
	[`version_history.json`][version-history], which lists `supported`
	API levels to test.

	The build target defined at `//sdk/ctf/release:tests` iterates over
	the downloaded prebuilts and applies the
	[`generate_ctf_tests.gni`][generate_ctf_tests] rules to thaw each
	downloaded package in the same way that the in-development packages
	were thawed. They can be included using `--with-test
	//sdk/ctf/release:tests` in your `fx set` args.

	This will add new tests for each supported level to your build:

	```bash
	$ fx test --dry
	...
	fuchsia-pkg://fuchsia.com/fuchsia-diagnostics-tests_ctf18#meta/fuchsia-diagnostics-tests-root.cm
	fuchsia-pkg://fuchsia.com/fuchsia-driver-test_tests_ctf18#meta/test-root.cm
	...
	fuchsia-pkg://fuchsia.com/fuchsia-diagnostics-tests_ctf19#meta/fuchsia-diagnostics-tests-root.cm
	fuchsia-pkg://fuchsia.com/fuchsia-driver-test_tests_ctf19#meta/test-root.cm
	```

	Each thawed test has a suffix indicating the release branch where
	the frozen package originated (e.g. ctf18, ctf19 for F18 and F19
	respectively). All of these tests can be run using `fx test`.

	## Troubleshooting failures

	CTF tests are designed to fail when there is an incompatibility
	between the frozen package and packages produced at tip of
	`fuchsia.git`, and this can cause the `core.x64-debug-ctf` builder
	to fail during CL submission. See the dedicated
	[troubleshooting guide][troubleshooting] for detailed instructions
	on handling failure scenarios to unblock CL submission.

	[CIPD]: https://chrome-infra-packages.appspot.com/p/fuchsia/cts/linux-amd64
	[generate_ctf_tests]: https://fuchsia.googlesource.com/fuchsia/+/HEAD/sdk/ctf/build/generate_ctf_tests.gni
	[motivation]: /docs/development/testing/ctf/compatibility_testing.md#motivation
	[overview]: /docs/development/testing/ctf/overview.md
	[troubleshooting]: /docs/development/testing/ctf/troubleshooting.md
	[version-history]: https://fuchsia.googlesource.com/fuchsia/+/HEAD/sdk/version_history.json
	[release-f16]: https://fuchsia.googlesource.com/fuchsia/+/refs/heads/releases/f16