docs/development/testing/faq.md

# Testing: Questions and Answers

You are encouraged to add your own questions (and answers) here!

[TOC]

## Q: How do I define a new unit test?

A: Use language-appropriate constructs, like GTest for C++. You can define a new
file if need be, such as:

(in a BUILD.gn file)

```code
executable("unittests") {
  output_name = "scenic_unittests"
  testonly = true
  sources = ["some_test.cc"],
  deps = [":some_dep"],
}
```

## Q: What ensures it is run?

A: Tests are represented as entries in the `FUCHSIA_BUILD_DIR/tests.json` file
generated by GN based on arguments of `fx set`. The step that creates or updates
`tests.json` is called the gen phase of the build, and it is triggered by
`fx set`, `fx build` or explicitly by `fx gen`.

If `fx test` doesn't find your test, first check `FUCHSIA_BUILD_DIR/tests.json`.
In most cases, the product configuration plus the `--with` arguments of your
`fx set` command are missing, directly or transitively, the test that you are
attempting to run. If that's the case, just add the test GN label or a GN
label that transitively has the test as an `--with` argument to the `fx set`
command, for example:

`fx set ... --with //src/sys:tests`

A test may also be disabled on certain product or variant configurations. If
`fx test` does not find the test after adding the label using `fx set`, verify
that the test is not excluded by a build rule. For example, a test might be
excluded in coverage variants. This could appear as follows in the test's
BUILD.gn file:

```
group("tests") {
  if (!is_coverage) {
    deps = [ ":my-test" ]
  }
}
```

If a newly added test is in the product configuration defined by `fx set` but
still doesn't show in `tests.json`, you may need to run `fx gen` or `fx build`
to update `tests.json` so that `fx test` knows how to run it.

## Q: How do I run this unit test on a QEMU instance?

There's the easy way if your QEMU has networking, and the hard way if it
doesn't.

A (with networking): In one terminal, start your QEMU instance with `fx qemu -N`.
Next, on another terminal, type in `fx test escher_tests`.

This invocation runs all the test executables in the `escher_tests` package.

A (no networking): Start a QEMU instance (`fx qemu`), and then *manually* invoke
the `run-test-suite` command.

In the QEMU shell, type in `run-test-suite <test_url>`.

Note Well! Without networking, the files are loaded into the QEMU instance at
startup. So after rebuilding a test, you'll need to shutdown and re-start the
QEMU instance to see the rebuilt test.

To exit QEMU, `dm shutdown`.

## Q: How do I run this unit test on my development device?

A: Either manual invocation, like in QEMU, **or** `fx test` to a running
device.

Note that the booted device may not contain your binary at startup, but `fx
test` will build the test binary, ship it over to the device, and run it,
while piping the output back to your workstation terminal. Slick!

Make sure your device is running (hit Ctrl-D to boot an existing image) and
connected to your workstation.

From your workstation, `fx test escher_tests` will serially run through all
test executables contained in the `escher_tests` package.

To run just one test executable, use the following command:

```bash
fx test <executable-name>
```

Note: To learn more about the new `fx test` command, see
[Run Fuchsia tests][run_fuchsia_tests].

You can automatically rebuild, install, and run your tests on every source file
change with `fx -i`. For instance: `fx -i test escher_tests`.

## Q: Where are the test results captured?

A: The output is directed to your terminal.

There does exist a way to write test output into files (including a summary JSON
file), which is how CQ bots collect the test output for automated runs.

## Q: How to disable a test? How to find and run disabled tests? {#disable-test}

A: There are several ways to do this. Whenever doing any of these, be sure to
file a bug and reference that bug in a comment in the code that disables the
test.

### Tag the test as flaky

You can do this by adding "flaky" to the `tags` field in the
[test environment](/docs/contribute/testing/environments.md). This operates
on the entire test target (which corresponds to an executable). It will prevent this target
from running on the builders in the commit queue, and enable the target on special flaky
builders that continue to run the test in CI. Be sure to note the bug in a
comment in the BUILD.gn file.
[Example change](https://fuchsia-review.googlesource.com/c/topaz/+/296629/3/bin/flutter_screencap_test/BUILD.gn).

If you want to disable only some tests that are part of a larger test target, you'll
need to split the target into two GN targets, and tag one as flaky.

### C++ googletest only: Prefix name with DISABLED

To disable a particular test inside of a larger test executable,
you can mark it as disabled. Disabled tests are defined by having their name
prefixed with `DISABLED_`. One way to find them is therefore simply `git grep
DISABLED_`.

To force-run disabled tests: `fx test escher_tests --also-run-disabled-tests`.

### Rust only: apply the `#[ignore]` attribute

To disable a particular test inside of a larger Rust test executable, you can
tag it with `#[ignore]`. It should be applied underneath the `#[test]` attribute.

Example:

```rust
#[test]
#[ignore] // TODO(https://fxbug.dev/NNNNN) re-enable this test when de-flaked
fn flaky_test_we_need_to_fix() { ... }
```

### Mark test disabled

Alternatively, you may also disable an entire test executable within a
package containing several test executables. To do this, edit the `BUILD.gn` as
follows: `tests = [ { name = "scenic_unittests", disabled = true } ]`. As a
result, `scenic_unittests` will be put in a `disabled` subdirectory of
`/pkgfs/packages/<package_name>/0/test`, and will not be run by the CQ system.

### Comment out the test

To disable a particular test inside of a larger test executable, you can comment
out the code that defines that test.

## Q: How do I run a bunch of tests automatically? How do I ensure all dependencies are tested?

A: The primary feature of `fx test` is batch execution. See
[Run Fuchsia tests][run_fuchsia_tests] for examples on how to run multiple
tests or test suites together.

Additionally, you can always upload your patch to Gerrit and do a CQ dry run.

## Q: How do I run this unit test in a CQ dry run?

A: Clicking on CQ dry run (aka +1) will take your change's properly defined unit
test and run it on multiple bots, one for each build target (*x86-64* versus
*arm64*, *release* versus *debug*). Each job will have an output page showing
all the tests that ran.

## Q: How do I use some build time artifacts in my unit test?

A: The simplest artifact is just a file that is in your source directory.  For
this you just need to add it to `resources` attribute of the package definition
of your unit test.  For example, you may do something like this in your
`BUILD.gn`:

```code
rustc_binary("my-great-app") {
  with_unit_tests = true

  ...
}

test_package("my-great-app-tests") {
  deps = [
    ":my-great-app_test",
  ]

  resources = [
    {
      path = "source.zip"
      dest = "testing.zip"
    }
  ]
```

The file will be available as `/pkg/data/testing.zip` inside the environment
where the test binary will be executed.

TODO: If you want an artifact that is generated as part of the build process,
you should probably add the rule that generates the artifact to the `data_deps`
array of the `test_package` rule.  But I have not tried it yet.  Update this
section when you will try it :)

[run_fuchsia_tests]: /docs/development/testing/run_fuchsia_tests.md