src/sys/pkg/tests/system-tests/README.md

# System Tests

This is the integration tests for a number of system tests.

## Test Setup

In order to build the system tests, add this to your `fx set`:

```sh
% fx set ... --with //src/sys/pkg:e2e_tests

% fx build
```

Next, you need to authenticate against luci to be able to download build
artifacts. First, authenticate against cipd with:

```sh
% cipd auth-login
```

Next, install chromium's `depot_tools` by following
[these instructions](https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html).
Then, login to luci by running:

```sh
% cd depot_tools
% ./luci-auth login -scopes "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/devstorage.read_write"
...
```

Now you should be able to run the tests against any device that is discoverable
by `ffx target list`.

## Available Tests

### Upgrade Tests

This tests Over the Air (OTA) updates. At a high level, it:

* Downloads downgrade and upgrade build artifacts
* Tells the device to reboot into recovery.
* Paves the device with the downgrade build artifacts (known as version N-1).
* OTAs the to the upgrade build artifacts (known as version N).
* OTAs the device to a variant of the upgrade build artifacts (known as version
  N').
* OTAs back into upgrade build artifacts (version N).

Note that for all tests below, the `--device name` and `--device-hostname
host/ip_addr` can be looked up from the output of `ffx target list`.
Without specifying them, the tests **might not work, especially if multiple
targets are connected**.

The tests can be run with:

```sh
% cd $(fx get-build-dir) && host_x64/system_tests_upgrade \
  --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \
  --ffx-path $(fx get-build-dir)/host-tools/ffx \
  --builder-name fuchsia/global.ci/core.x64-release \
  --fuchsia-build-dir $(fx get-build-dir) \
  --device fuchsia-5254-475e-82ef \
  --device-hostname fe80::e493:5839:b86a:f7f1%qemu
```

This will run through the whole test paving the build to the latest version
available from the specified builder, then OTA-ing the device to the local build
directory.

The Upgrade Tests also support reproducing a specific build. To do this,
determine the build ids from the downgrade and upgrade builds, then run:

```sh
% cd $(fx get-build-dir) && host_x64/system_tests_upgrade \
  --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \
  --ffx-path $(fx get-build-dir)/host-tools/ffx \
  --build-id 123456789... \
  --build-id 987654321... \
  --device fuchsia-5254-475e-82ef \
  --device-hostname fe80::e493:5839:b86a:f7f1%qemu
```

Or you can combine these options:

```sh
% cd $(fx get-build-dir) && host_x64/system_tests_upgrade \
  --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \
  --ffx-path $(fx get-build-dir)/host-tools/ffx \
  --build-id 123456789... \
  --fuchsia-build-dir $(fx get-build-dir) \
  --device fuchsia-5254-475e-82ef \
  --device-hostname fe80::e493:5839:b86a:f7f1%qemu
```

### Reboot Testing

The system tests support running reboot tests, where a device is rebooted a
configurable number of times, or errs out if a problem occurs. This
can be done by running:

```sh
% cd $(fx get-build-dir) && host_x64/system_tests_reboot \
  --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \
  --ffx-path $(fx get-build-dir)/host-tools/ffx \
  --fuchsia-build-dir $(fx get-build-dir) \
  --device fuchsia-5254-475e-82ef \
  --device-hostname fe80::e493:5839:b86a:f7f1%qemu
```

Or if you want to test a build, you can use:

* `--builder-name fuchsia/global.ci/core.x64-release`, to test
  the latest build published by that builder.
* `--build-id 1234...` to test the specific build.

## Running the Tests

When running the system tests, it's helpful to capture the serial logs, and
system logs, and the test output to a file in order to triage any failures. This
is especially handy when cycle testing. To simplify the setup, the system-tests
come with a helper script `run-test` that can setup a `tmux` session
for you. You can run it like this:

```sh
% ${FUCHSIA_DIR}/src/sys/pkg/tests/system-tests/bin/run-test \
  -o ~/logs \
  --tty /dev/ttyUSB0 \
  $(fx get-build-dir)/host_x64/system_tests_upgrade \
  --ffx-path $(fx get-build-dir)/host-tools/ffx \
  --builder-name fuchsia/global.ci/core.x64-release \
  --fuchsia-build-dir $(fx get-build-dir) \
  --device fuchsia-5254-475e-82ef \
  --device-hostname fe80::e493:5839:b86a:f7f1%qemu
```

This will setup a `tmux` with 3 windows, one for the serial session on
`/dev/ttyUSB0`, one for the system logs, and one for the test. All output from
the `tmux` windows will be saved into `~/logs`.

See the `run-test --help` for more options.

## Running the tests locally in the Fuchsia Emulator

The Fuchsia emulator supports running the tests locally. This is fully supported
on x64 only at present. Support for arm64 is work in progress. Follow these
instructions:

Build a supported x64 product with enabled system tests:

```sh
% fx set core.x64 --with //src/sys/pkg:e2e_tests

% fx build
```

Start the emulator in `--uefi` mode, providing valid keys and metadata files for
signing the ZBI (needed for Fuchsia's verified boot). For testing purposes, the
test keys included in the Fuchsia source can be used:

```sh
% ffx emu start --net tap -H --uefi \
  --vbmeta-key \
    third_party/android/platform/external/avb/test/data/testkey_atx_psk.pem \
  --vbmeta-key-metadata \
    third_party/android/platform/external/avb/test/data/atx_metadata.bin
```

### N+1 OTA test

After the emulator has started up, a local "N+1 OTA test". can be performed.
This is helpful to verify that the build which is currently in the `out`
directory does not have a bug that prevents being OTA'ed in the future.
This test creates a no-op change (called OTA N-prime in the log later) and
performs an OTA to this new version:

```sh
% cd $(fx get-build-dir) && host_x64/system_tests_upgrade \
  --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \
  --ffx-path $(fx get-build-dir)/host-tools/ffx \
  --fuchsia-build-dir $(fx get-build-dir) \
  --device fuchsia-5254-475e-82ef \
  --device-hostname fe80::e493:5839:b86a:f7f1%qemu
```

### Last Known Good to local build OTA

Instead of the mentioned N+1 OTA, it is also possible to perform an OTA test
from the LKG (**L**ast **K**known **G**ood) build from the Fuchsia build
infrastructure (if you have successfully authenticated to cipd, see above at
the top of the document). This will verify that the build in the local `out`
directory will successfully boot coming from a previous version.

```sh
% $(fx get-build-dir)/host_x64/system_tests_upgrade \
  --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \
  --ffx-path $(fx get-build-dir)/host-tools/ffx \
  --builder-name fuchsia/global.ci/core.x64-release \
  --fuchsia-build-dir $(fx get-build-dir) \
  --device fuchsia-5254-475e-82ef \
  --device-hostname fe80::e493:5839:b86a:f7f1%qemu
```