blob: 4494ef2c841e1b756fd3f4ea4ff2ae3210cf4564 [file] [log] [blame] [view]
# FIDL Cross-Petal Change Tests
These are tests to validate source-compatibility through various changes to FIDL
libraries. They do not test binary-compatibility. Tests for compiled languages
are only meant to be built, not executed, since source-compatibility issues show
up at compile time.
This directory consists of:
* Tools to generate source compatibility tests. The usage of these tools is
explained in the [Writing tests](#writing-tests) section.
* A command line tool, located under `gen/`.
* A gn template for building source compatibility tests, defined in
`fidl_source_compatibility.gni`.
* The source compatibility tests themselves. Each test is located in a separate
directory, which ideally follows the `[parent]-[target]-[change]` format
used in the FIDL ABI/API compatibility guide. These tests are generated using
the tool above, and also each contain a generated README.md file describing
the transition being tested.
## Transition terminology
Transitions that follow a specific pattern are sometimes given a specific name to make it
easier to refer to these kinds of transitions when discussing them.
One set of terms we use is "FIDL assisted" and "source assisted": when
transitioning a FIDL library involves an initial state (**before**), an
intermediate state (**during**), and a final state (**after**), depending on the
bindings used and the kind of change made to the FIDL library, the transition is
either [FIDL-assisted](#fidl-assisted) or [source-assisted](#source-assisted).
### FIDL-assisted {#fidl-assisted}
In a FIDL-assisted transition, you change source code while the FIDL library is
held in a transitional state (e.g., using the `Transitional` attribute). For
these transitions, we test four states:
| Time | 1 | 2 | 3 | 4 |
| ------ | ------ | ------ | ------ | ----- |
| FIDL | before | during | during | after |
| Source | before | before | after | after |
### Source-assisted {#source-assisted}
In a source-assisted transition, you change the FIDL library while source code
held is in a transitional state (e.g., using `default:` in switch statements).
This _would_ lead to testing four states:
| Time | 1 | 2 | 3 | 4 |
| ------ | ------ | ------ | ------ | ----- |
| FIDL | before | before | after | after |
| Source | before | during | during | after |
However, certain FIDL changes require a FIDL-assisted transition in some
bindings and a source-assisted transition in others. Suppose there is a change in
FIDL library _L_ requiring a FIDL-assisted transition in bindings _A_ and
source-assisted in bindings _B_. The following steps would be taken:
1. Initially, _L_, _A_, and _B_ are **before**.
2. Change _B_ to **during**.
3. Change _L_ to **during**.
4. Change _A_ to **after**.
5. Change _L_ to **after**.
6. Change _B_ to **after**.
All correct ways of interleaving the steps will have _L_ and _B_ both in the
**during** state at some point. Therefore, although a FIDL **during** state is
unnecessary for a pure source-assisted transition, we must include it in tests.
Thus, we actually test 5 states for source-assisted transitions:
| Time | 1 | 2 | 3 | 4 | 5 |
| ------ | ------ | ------ | ------ | ------ | ----- |
| FIDL | before | before | during | after | after |
| Source | before | during | during | during | after |
## Updating test documentation
The transition example docs are each generated from individual source
compatibility test cases. To update a file (e.g. `bits_member_add.md`), you need
to update the test that the file is generated from, and then regenerate the
documentation.
The tests are located in `//src/tests/fidl/source_compatibility`, and have a
name matching the generated doc's filename. For example, the test corresponding
to `bits_member_add.md` is `bits-member-add`.
* Make the desired change
* For code changes, find the test file containing the code you want to change
by identifying the step number and binding of the code in question, and
modify the source as desired
* For instruction changes, find the corresponding entry in the test's JSON
configuration file (`test.json`) using step number and binding
and modify the value under the `"instructions"` key. The value should be an
array of strings, which will get rendered as a bulleted list in markdown.
* Regen the documentation by running, e.g.
`src/tests/fidl/source_compatibility/gen/main.py regen bits-member-add`. The
argument of `regen` should be the name of the directory that the test is
contained in.
* Run `fx format-code` to get rid of formatting differences, and verify the
desired change
## Writing tests (#writing-tests)
A test is declared by defining a test JSON configuration and adding a
`source_compatibility_test` target to the build (for examples, see the
`test.json` files within each test directory). This is done by running the
source compatibility gen tool.
The rest of the commands in this README assume
that this tool is aliased, e.g. by adding the following to your rc file:
```
alias scompat=$FUCHSIA_DIR/src/tests/fidl/source_compatibility/gen/main.py
```
A high level overview of the flow for creating a test using the tools is
as follows:
* Run the test scaffolding tool, giving it the name of the directory it will
be written to, e.g. `scompat generate_test foo-rename-bar`.
* The first step ("Step 0") of the tool will ask you to define the test title
as well as the initial states of the FIDL library and for each binding. It
will ask you for the filename, write a stub file, then pause to let you fill
out the contents of the file.
* At this step (and after each subsequent step), the tool will output a
`test.json` file containing the current state of the test, and a `BUILD.gn`
file declaring a target for the test.
* You should add the target defined in the build file to the root source
compatibility test group, so that you can build the test at each step, as
you write it.
* It can also be helpful to keep the `test.json` file open as you use the
tool to verify the changes it makes. You can correct any typos or other
mistakes that were provided to the tool by editing the JSON file, then
running `scompat regen`.
* The tool will then alternate between asking you to define the next FIDL change
and the next bindings change. It will prefill the source files for the next
step with the contents of the file from the previous step.
* Since the tool output (e.g. `test.json`) is updated at each step, you can
simply quit the tool when the transition is complete. It can also be helpful
to build the test between adding steps to ensure that they are correct. Note
however that the tool does not save the state until you complete the current
step.
* Update documentation top level documentation when adding removing tests (the
`generate_test`, `generate_reverse`, and `regen` command automatically regen
the test-specific documentation).
* Update the table of contents in
`//docs/development/languages/fidl/guides/compatibility/_toc.yaml` to point to
the newly generated documentation. You can do this by running `scompat regen_toc`.
* Update the top level ABI/API compatibility doc to add or remove xrefs to
generated documentation.
### Modifying existing tests:
If you call the tool on an existing test (i.e. a directory containing a
`test.json` file), it will resume from where you last left off. In other words,
the tool will only append new steps to the test, and cannot insert or
otherwise edit existing steps. You must make edits manually by modifying the
source files and `test.json` file yourself, then run `scompat regen` to
regenerate the auxiliary files that are based on the `test.json`, such as the
README and GN sidecar file.
The test JSON structure is defined in `gen/types_.py`, which contains a number
of classes which correspond directly to the test JSON.
### Debugging tests:
When a test fails to compile, the failure output will contain a path to a place
somewhere in your out directory (what the path represents depends on the
binding, e.g. for C++ the path is the path to the `.o` file, whereas for Dart
it's the path that the source file is copied to before building), that will tell
you exactly which test/FIDL file/source file combination the error is coming
from. For example, if you see the directory
protocoleventremove_dart_step_03_after_step_02_during_dart_package
somewhere in the path, you can deduce the following based on the ordering:
* The test `protocoleventremove`.
* The language is `dart`.
* The FIDL step being used is `step_03_after`.
* The source file being used is `step_02_during`.
From this, you can deduce that the files in question are
`protocol-event-remove/fidl/step_03_after.test.fidl` and
`protocol-event-remove/fidl/step_02_during.dart`, and use that to debug the
compile error.
### "Reverse" tests:
Many tests are symmetrical: for example, removing a method consists of the
same steps as adding a method, but backwards. Currently, the source
compatibility tool provides a way to "bootstrap" a new test based on an
existing one, by running it in reverse. For example, if you have a test
`protocol-method-add` and want to generate the symmetric
`protocol-method-remove` test based on it, you can run:
scompat generate_reverse protocol-method-add protocol-method-remove
This will create a copy of the method add test but in reverse. After generating
the test, there are two additional manual steps:
* Rename the FIDL library used in the test. For example,
replace all instances of `protocolmethoadd` with `protocolmethodremove` inside
of the `protocol-method-remove/` test.
* Update the transition instructions for the test. Generally, this will involve
negating any existing instructions. For example, an instruction of
`"add method foo"`, would become `"remove method foo"` in the reversed test.
* Run the regen command (`scompat regen [test_name]`) to regenerate the
documentation based on any instruction changes.