This is the integration tests for a number of system tests.
In order to build the system tests, add this to your fx set:
% fx set ... --with //src/sys/pkg:e2e_tests % fx build
Next, you need to authenticate against luci to be able to download build artifacts. First, authenticate against cipd with:
% cipd auth-login
Next, install chromium's depot_tools by following these instructions. Then, login to luci by running:
% cd depot_tools % ./luci-auth login -scopes "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/devstorage.read_write" ...
Now you should be able to run the tests against any device that is discoverable by ffx target list.
This tests Over the Air (OTA) updates. At a high level, it:
Note that for all tests below, the --device name and --device-hostname host/ip_addr can be looked up from the output of ffx target list. Without specifying them, the tests might not work, especially if multiple targets are connected.
The tests can be run with:
% cd $(fx get-build-dir) && host_x64/system_tests_upgrade \ --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \ --ffx-path $(fx get-build-dir)/host-tools/ffx \ --builder-name fuchsia/global.ci/core.x64-release \ --fuchsia-build-dir $(fx get-build-dir) \ --device fuchsia-5254-475e-82ef \ --device-hostname fe80::e493:5839:b86a:f7f1%qemu
This will run through the whole test paving the build to the latest version available from the specified builder, then OTA-ing the device to the local build directory.
The Upgrade Tests also support reproducing a specific build. To do this, determine the build ids from the downgrade and upgrade builds, then run:
% cd $(fx get-build-dir) && host_x64/system_tests_upgrade \ --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \ --ffx-path $(fx get-build-dir)/host-tools/ffx \ --build-id 123456789... \ --build-id 987654321... \ --device fuchsia-5254-475e-82ef \ --device-hostname fe80::e493:5839:b86a:f7f1%qemu
Or you can combine these options:
% cd $(fx get-build-dir) && host_x64/system_tests_upgrade \ --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \ --ffx-path $(fx get-build-dir)/host-tools/ffx \ --build-id 123456789... \ --fuchsia-build-dir $(fx get-build-dir) \ --device fuchsia-5254-475e-82ef \ --device-hostname fe80::e493:5839:b86a:f7f1%qemu
The system tests support running reboot tests, where a device is rebooted a configurable number of times, or errs out if a problem occurs. This can be done by running:
% cd $(fx get-build-dir) && host_x64/system_tests_reboot \ --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \ --ffx-path $(fx get-build-dir)/host-tools/ffx \ --fuchsia-build-dir $(fx get-build-dir) \ --device fuchsia-5254-475e-82ef \ --device-hostname fe80::e493:5839:b86a:f7f1%qemu
Or if you want to test a build, you can use:
--builder-name fuchsia/global.ci/core.x64-release, to test the latest build published by that builder.--build-id 1234... to test the specific build.When running the system tests, it's helpful to capture the serial logs, and system logs, and the test output to a file in order to triage any failures. This is especially handy when cycle testing. To simplify the setup, the system-tests come with a helper script run-test that can setup a tmux session for you. You can run it like this:
% ${FUCHSIA_DIR}/src/sys/pkg/tests/system-tests/bin/run-test \ -o ~/logs \ --tty /dev/ttyUSB0 \ $(fx get-build-dir)/host_x64/system_tests_upgrade \ --ffx-path $(fx get-build-dir)/host-tools/ffx \ --builder-name fuchsia/global.ci/core.x64-release \ --fuchsia-build-dir $(fx get-build-dir) \ --device fuchsia-5254-475e-82ef \ --device-hostname fe80::e493:5839:b86a:f7f1%qemu
This will setup a tmux with 3 windows, one for the serial session on /dev/ttyUSB0, one for the system logs, and one for the test. All output from the tmux windows will be saved into ~/logs.
See the run-test --help for more options.
The Fuchsia emulator supports running the tests locally. This is fully supported on x64 only at present. Support for arm64 is work in progress. Follow these instructions:
Build a supported x64 product with enabled system tests:
% fx set core.x64 --with //src/sys/pkg:e2e_tests % fx build
Start the emulator in --uefi mode, providing valid keys and metadata files for signing the ZBI (needed for Fuchsia's verified boot). For testing purposes, the test keys included in the Fuchsia source can be used:
% ffx emu start --net tap -H --uefi \ --vbmeta-key \ third_party/android/platform/external/avb/test/data/testkey_atx_psk.pem \ --vbmeta-key-metadata \ third_party/android/platform/external/avb/test/data/atx_metadata.bin
After the emulator has started up, a local “N+1 OTA test”. can be performed. This is helpful to verify that the build which is currently in the out directory does not have a bug that prevents being OTA'ed in the future. This test creates a no-op change (called OTA N-prime in the log later) and performs an OTA to this new version:
% cd $(fx get-build-dir) && host_x64/system_tests_upgrade \ --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \ --ffx-path $(fx get-build-dir)/host-tools/ffx \ --fuchsia-build-dir $(fx get-build-dir) \ --device fuchsia-5254-475e-82ef \ --device-hostname fe80::e493:5839:b86a:f7f1%qemu
Instead of the mentioned N+1 OTA, it is also possible to perform an OTA test from the LKG (Last Kknown Good) build from the Fuchsia build infrastructure (if you have successfully authenticated to cipd, see above at the top of the document). This will verify that the build in the local out directory will successfully boot coming from a previous version.
% $(fx get-build-dir)/host_x64/system_tests_upgrade \ --ssh-private-key $(ffx config get ssh.priv | tr -d '"') \ --ffx-path $(fx get-build-dir)/host-tools/ffx \ --builder-name fuchsia/global.ci/core.x64-release \ --fuchsia-build-dir $(fx get-build-dir) \ --device fuchsia-5254-475e-82ef \ --device-hostname fe80::e493:5839:b86a:f7f1%qemu