System Tests

This is the integration tests for a number of system tests.

Test Setup

In order to build the system tests, add this to your fx set:

% fx set ... --with //src/sys/pkg:e2e_tests
% fx build

Next, you need to authenticate against luci to be able to download build artifacts. Install chromium's depot_tools by following these instructions. Then, login to luci by running:

% cd depot_tools
% ./luci-auth login -scopes "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/devstorage.read_write"
...

Now you should be able to run the tests against any device that is discoverable by fx device-finder.

Available Tests

Upgrade Tests

This tests Over the Air (OTA) updates. At a high level, it:

Downloads downgrade and upgrade build artifacts
Tells the device to reboot into recovery.
Paves the device with the downgrade build artifacts (known as version N-1).
OTAs the to the upgrade build artifacts (known as version N).
OTAs the device to a variant of the upgrade build artifacts (known as version N').
OTAs back into upgrade build artifacts (version N).

You should be able to run the tests with:

% $(fx get-build-dir)/host_x64/system_tests_upgrade \
  --ssh-private-key ~/.ssh/fuchsia_ed25519 \
  --downgrade-builder-name fuchsia/global.ci/fuchsia-x64-release-build_only \
  --upgrade-fuchsia-build-dir $FUCHSIA_BUILD_DIR

This will run through the whole test paving the build to the latest version available from the specified builder, then OTA-ing the device to the local build directory.

The Upgrade Tests also support reproducing a specific build. To do this, determine the build ids from the downgrade and upgrade builds, then run:

% $(fx get-build-dir)/host_x64/system_tests_upgrade \
  --ssh-private-key ~/.ssh/fuchsia_ed25519 \
  --downgrade-build-id 123456789... \
  --upgrade-build-id 987654321...

Or you can combine these options:

% $(fx get-build-dir)/host_x64/system_tests_upgrade \
  --ssh-private-key ~/.ssh/fuchsia_ed25519 \
  --downgrade-build-id 123456789... \
  --upgrade-fuchsia-build-dir $FUCHSIA_BUILD_DIR

There are more options to the test, to see them all run $(fx get-build-dir)/host_x64/system_tests_upgrade -- -h.

Reboot Testing

The system tests support running reboot tests, where a device is rebooted a configurable number of times, or errs out if a problem occurs. This can be done by running:

% $(fx get-build-dir)/host_x64/system_tests_reboot \
  --ssh-private-key ~/.ssh/fuchsia_ed25519 \
  --fuchsia-build-dir $FUCHSIA_BUILD_DIR

Or if you want to test a build, you can use:

--builder-name fuchsia/global.ci/fuchsia-x64-release-build_only, to test the latest build published by that builder.
--build-id 1234... to test the specific build.

Tracking Testing

The system tests support running tracking tests, where a device is continuously updated to the latest available version, or errs out if a problem occurs. This can be done by running:

% $(fx get-build-dir)/host_x64/system_tests_tracking \
  --ssh-private-key ~/.ssh/fuchsia_ed25519 \
  --downgrade-builder-name fuchsia/global.ci/fuchsia-x64-release-build_only \
  --upgrade-builder-name fuchsia/global.ci/fuchsia-x64-release-build_only

The --downgrade-build* argument is optional, and only necessary if you want to start the tracking test from a known zero state.

Note that at the moment the only supported upgrade mode is --upgrade-builder-name $BUILDER_NAME.

Running the Tests

When running the system tests, it's helpful to capture the serial logs, and system logs, and the test output to a file in order to triage any failures. This is especially handy when cycle testing. To simplify the setup, the system-tests come with a helper script run-test that can setup a tmux session for you. You can run it like this:

% ${FUCHSIA_DIR}/src/sys/pkg/tests/system-tests/bin/run-test \
  -o ~/logs \
  --tty /dev/ttyUSB0 \
  $(fx get-build-dir)/host_x64/system_tests_upgrade \
  --downgrade-builder-name fuchsia/global.ci/fuchsia-x64-release-build_only \
  --upgrade-fuchsia-build-dir $(fx get-build-dir)

This will setup a tmux with 3 windows, one for the serial session on /dev/ttyUSB0, one for the system logs, and one for the test. All output from the tmux windows will be saved into ~/logs.

See the run-test --help for more options.

Running the tests locally in the Fuchsia Emulator (experimental)

At the moment, the builtin fx emu does not bring up a configuration that can be OTA-ed. Until this is implemented, the bin/ directory contains some helper scripts that bring up an OTA-able Fuchsia emulator. Follow these instructions to it.

The run-emu script will create a Fuchsia EFI image, launch QEMU, then run the paver. To create the image the script wraps fx make-fuchsia-vol. To launch QEMU there is a modified version of run-zircon (also in the bin/ directory) that uses OVMF to allow us to run the bootloader. Any arguments you pass to the script will be passed through to QEMU.

One thing to note, by default, you won't see any terminal output in the emulator after an OTA. To restore this behavior, you need an extra build argument to send output to the terminal:

% fx set ... \
  --with //src/sys/pkg:tests \
  --args 'dev_bootfs_labels+=["//src/zircon/kernel/cmdline:serial-legacy"]'
% fx build

With all that setup, you now should be able to run the tests. First, launch the emulator:

% ./bin/run-emu

In a new terminal, run the tests. For example, to run the upgrade tests:

% $(fx get-build-dir)/host_x64/system_tests_upgrade \
  --ssh-private-key ~/.ssh/fuchsia_ed25519 \
  --downgrade-builder-name fuchsia/global.ci/fuchsia-x64-release-build_only \
  --upgrade-fuchsia-build-dir $(fx get-build-dir) \
  --device fuchsia-5254-0063-5e7a

Note that your QEMU device will always be named fuchsia-5254-0063-5e7a. Explicitly naming the device for the test will help prevent accidental upgrades to your other devices.