blob: 0e9c5528fb8a6103972c3bcaaee7ee4fea92d3cb [file] [log] [blame] [view]
# Run an end-to-end test
This guide provides instructions on how to run an end-to-end test for testing a
Fuchsia product.
This guide uses the Fuchsia emulator ([FEMU](/docs/get-started/set_up_femu.md)) to
emulate a device that runs Fuchsia. As for the end-to-end test, the guide uses
the
[`screen_is_not_black`](/src/tests/end_to_end/screen_is_not_black/)
end-to-end test.
The `screen_is_not_black` end-to-end test reboots a device under test
(DUT), waits 100 seconds, and takes a snapshot of the devices screen. If the
snapshot image is not a black screen, the test concludes that Fuchsia is
successfully up and running on the device after reboot.
To run this end-to-end test, the steps are:
1. [Prerequisites](#prerequisites).
1. [Build a Fuchsia image to include the end-to-end test](#build-a-fuchsia-image-to-include-the-end-to-end-test).
1. [Start the emulator with the Fuchsia image](#start-the-emulator-with-the-fuchsia-image).
1. [Run the end-to-end test](#run-the-end-to-end-test).
Also, to run any end-to-end test, see the [Appendices](#appendices) section.
## 1. Prerequisites {#prerequisites}
This guide requires that you've completed the following guides:
* [Download the Fuchsia source code](/docs/get-started/get_fuchsia_source.md).
* [Start the Fuchsia emulator](/docs/get-started/set_up_femu.md).
## 2. Build a Fuchsia image to include the end-to-end test {#build-a-fuchsia-image-to-include-the-end-to-end-test}
Before you can run the `screen_is_not_black` end-to-end test, you first
need to build your Fuchsia image to include the test in the build artifacts:
Note: The examples in this guide use the `workstation_eng` product. End-to-end tests work with most
products except `core`.
1. To add the end-to-end test, run the `fx set` command with the following
`--with` option:
```posix-terminal
fx set workstation_eng.qemu-x64 --with //src/tests/end_to_end/screen_is_not_black
```
`//src/tests/end_to_end/screen_is_not_black` is a test directory in the
Fuchsia source tree. The
<code>[BUILD.gn](/src/tests/end_to_end/screen_is_not_black/BUILD.gn)</code>
file in this directory defines the <code>screen_is_not_black</code> target
to include the <code>screen_is_not_black</code> end-to-end test in the build
artifacts.
1. Build your Fuchsia image:
```posix-terminal
fx build
```
When the `fx build` command completes, the build artifacts now include the
`screen_is_not_black` end-to-end test, which you can run from your host
machine.
## 3. Start the emulator with the Fuchsia image {#start-the-emulator-with-the-fuchsia-image}
Start the emulator with your Fuchsia image and run a
[package repository server](/docs/development/build/fx.md#serve-a-build):
Note: The steps in this section assume that you don't have any terminals
currently running FEMU or the `fx serve` command.
1. Configure an IPv6 network for the emulator:
Note: This has to be completed once per machine.
```posix-terminal
sudo ip tuntap add dev qemu mode tap user $USER && sudo ip link set qemu up
```
1. Configure the upscript:
Note: If your machine is behind a firewall, you may need to apply some additional
configuration to allow the emulator to access the network. This is typically
accomplished by running an "upscript", which sets up the interfaces and firewall
access rules for the current process. If you're on a corporate network, check
with your internal networking team to see if they have an existing upscript
for you to use.
If you're not behind a firewall, there's still some configuration needed to
enable tun/tap networking. The example upscript
at <code>{{ '<var>' }}FUCHSIA_ROOT{{ '</var>' }}/scripts/start-unsecure-internet.sh</code>
should work for the majority of non-corporate users.
```posix-terminal
ffx config set emu.upscript {{ '<var>' }}PATH_TO_UPSCRIPT{{ '</var>' }}
```
Replace the following:
* `PATH_TO_UPSCRIPT`: The path to a FEMU network setup script; for example,
`~/fuchsia/scripts/start-unsecure-internet.sh`.
1. Start the package server
```posix-terminal
fx serve
```
1. Start the emulator:
```posix-terminal
ffx emu start --net tap
```
When startup is complete, the emulator prints the following message and opens
a shell prompt:
```none {:.devsite-disable-click-to-copy}
Logging to "{{ '<var>' }}$HOME{{ '</var>' }}/.local/share/Fuchsia/ffx/emu/instances/fuchsia-emulator/emulator.log"
Waiting for Fuchsia to start (up to 60 seconds)........Emulator is ready.
```
1. The `--net` flag requires a value to indicate which kind of
networking to implement. `--net` has the following possible values:
- `tap`: Attaches a Tun/Tap interface.
- `user`: Sets up mapped ports through SLiRP.
- `none`: Disables networking.
- `auto`: Checks the host system's capabilities and selects `tap` if it is
available or `user` if a tap interface is unavailable.
`auto` is the default.
`auto` is the default if the flag is not specified on the command line.
The upscript is automatically executed only if the user selects `tap`
mode.
If `auto` is used, the launcher checks for a tap interface on the
device. If it finds a tap interface, it uses `tap` mode; otherwise it
uses `user` mode.
1. Run the `fx set-device` command and select `fuchsia-emulator` (the
emulator's default device name) to be your device, for example:
<pre>
$ fx set-device
ERROR: Multiple devices found, please pick one from the list:
1) fuchsia-4407-0bb4-d0eb
2) fuchsia-emulator
#? <b>2</b>
New default device: fuchsia-emulator
</pre>
## 4. Run the end-to-end test {#run-the-end-to-end-test}
Run the `screen_is_not_black` end-to-end test:
```posix-terminal
fx test --e2e screen_is_not_black
```
When the test passes, this command prints output similar to the
following:
```none {:.devsite-disable-click-to-copy}
Saw a screen that is not black; waiting for 0:00:59.924543 now.
...
[FINE]: Running over ssh: killall sl4f.cmx
01:46 +1: All tests passed!
```
## Appendices {#appendices}
### Run any end-to-end test {#run-any-end-to-end-test}
Use the `fx test --e2e` command to run an end-to-end test from your host
machine:
```posix-terminal
fx test --e2e <TEST_NAME>
```
Some product configurations may include a set of end-to-end tests by default
(see [Examine product configuration files](#examine-product-configuration-files)).
However, if you want to run an end-to-end test that is not part of your
product configuration, configure your Fuchsia image to include the specific
test:
```posix-terminal
fx set <PRODUCT>.<BOARD> --with <TEST_DIRECTORY>:<TARGET>
```
For example, the following commands configure and build your Fuchsia image
with all the end-to-end tests in the
<code>[//src/tests/end_to_end/perf](/src/tests/end_to_end/perf/)</code> test
directory:
```none {:.devsite-disable-click-to-copy}
$ fx set workstation_eng.qemu-x64 --with //src/tests/end_to_end/perf:test
$ fx build
```
Note: Some end-to-end tests are designed to run only on specific product
configurations.
For the list of all available end-to-end tests in the Fuchsia repository, see
the [//src/tests/end\_to\_end](/src/tests/end_to_end/) directory.
### Examine product configuration files {#examine-product-configuration-files}
To find out which end-to-end tests are included in a
specific product configuration, examine product configuration files (`.gni`) in
the Fuchsia repository's <code>[//products][products-dir]</code> directory.
The following example shows the product configurations files in the
`//products` directory:
```none {:.devsite-disable-click-to-copy}
~/fuchsia/products$ ls *.gni
bringup.gni core.gni terminal.gni workstation_eng.gni
```
To see the list of all available product configurations, you can run the
following command:
```posix-terminal
fx list-products
```
Among these product configurations, <code>[terminal][terminal-gni]</code> and
<code>[workstation_eng][workstation-gni]</code> include end-to-end tests by
default. The following example shows the end-to-end tests included
in `terminal.gni`:
```none {:.devsite-disable-click-to-copy}
cache_package_labels += [
...
"//src/tests/end_to_end/bundles:end_to_end_deps",
"//src/tests/end_to_end/bundles:terminal_end_to_end_deps",
]
...
universe_package_labels += [
"//src/tests/end_to_end/screen_is_not_black",
"//src/tests/end_to_end/sl4f:test",
"//src/tests/end_to_end/perf:test",
...
]
```
<!-- Reference links -->
[products-dir]: /products/
[terminal-gni]: /products/terminal.gni
[workstation-gni]: /products/workstation_eng.gni