docs/reference/testing/test-output-format.md

# Fuchsia test output format

This document describes the directory format used to collect and store results
and outputs for tests run on Fuchsia. It is currently output by
[ffx test run][ffx-test] when the `--output-directory` option is specified.

## Background

A single execution of `ffx test` produces a *test run*. A *test run* is
comprised of any number of *suite runs*, and each suite run is comprised of
any number of *test cases*.

A *suite run* is an execution of a test component. The test component is most
commonly identified by its URL, for example,
`fuchsia-pkg://fuchsia.com/run_test_suite_integration_tests#meta/passing-test-example.cm`.

A *test case* is an execution of a single test case contained in a suite.

In addition, test runs, suites, or test cases may produce *artifacts*. These
include outputs from the test such as stdout, stderr, and syslog. *Artifacts*
may be either a file or a directory.

## Overview

Test output is stored as a directory. The directory contains the results for a
single test run and any suite runs and test cases it contains. The root of the
directory contains a single JSON file called `run_summary.json` and any number
of subdirectories. Each subdirectory contains artifacts for either the test
run, or a single suite run or test case.

## Directory layout

The JSON file is always called `run_summary.json` and is always located in the
root level of the directory. `run_summary.json` contains the complete set of
results for the overall test run, suite runs within the test run, and test
cases within each suite run. It also contains the name of each artifact
subdirectory and a list of artifacts within it. Artifacts are always located in
the top level of an artifact subdirectory.

The names of subdirectories and artifacts within the subdirectories are not
specified as part of the schema. Instead, the actual names of subdirectories
and artifacts are listed in `run_summary.json`.

## `run_summary.json` contents

The exact schema for `run_summary.json` is published on
[fuchsia.dev][schema-in-fuchsia-dev]. In Fuchsia source, it is available under
the [//sdk/schema][schema-in-source] directory.

## Artifacts

Artifacts may consist of either a single file or a directory. The following
artifacts exist today and are known to the scheme.

Note: the schema is extensible; artifact types not known to the schema may be
stored in the output format.

| Type           | Description                                                   |
| -------------- | ------------------------------------------------------------- |
| SYSLOG         | Syslog collected from all components running in a test suite. |
| RESTRICTED_LOG | Any high severity logs that caused a test to be failed.       |
| STDOUT         | stdout collected from a single component in a test.           |
| STDERR         | stderr collected from a single component in a test.           |
| REPORT         | A copy of the stdout output produced by ffx test.             |
| CUSTOM         | A set of arbitrary files produced by a component in a test.   |

## Parsing

Tools that need to process test results should begin by parsing
`run_summary.json`, as it is the only part of the output directory with a
location defined in the schema. The remaining contents of the directory are
described by `run_summary.json`. Tools should not assume any particular naming
convention for artifacts or subdirectories in the output.

## Example

This example output was generated by running `ffx test` against an
[example test][example-test].

```bash
$ ffx test run --output-directory /tmp/ffx-out fuchsia-pkg://fuchsia.com/stdout-test#meta/stdout-test.cm
```

The contents of the `/tmp/ffx-out` directory generated by `ffx test` is
available in [examples/testing][example-output-dir].

## Related documents

[RFC-0163][output-rfc]: RFC proposing this output.

[example-output-dir]: /examples/testing/tools/output-directory
[example-test]: /examples/testing/tools/src/stdout_test.rs
[ffx-test]: https://fuchsia.dev/reference/tools/sdk/ffx#run_3
[schema-in-source]: /sdk/schema/ffx_test
[schema-in-fuchsia-dev]: https://fuchsia.dev/schema/ffx_test/run_summary-8d1dd964.json
[output-rfc]: /docs/contribute/governance/rfcs/0163_test_output_format.md