Test Runner

test_runner is a TCP daemon that runs on Fuchsia to accept connections, read test commands and execute them. It provides a way to run tests from the host and get back results.

On Fuchsia, each test is run in a new application environment, which exposes a service that the tests can use to report completion (particularly useful for integration tests).

Instructions

Prerequisites

• An instance of zircon running (on QEMU or real device), configured with networking. For example, see networking configuration doc. For QEMU, see also the warning below.

• A build configuration that runs test_runner at startup. For example, ./package/gn/gen.py -p boot_test_modular. Alternatively, you can run test_runner from the shell.

Running the tests

The script will automatically search for a Zircon device on the local subnet and use it. This discovery is performed using ipv6. This process works for both QEMU and for real hardware, but not for the Fuchsia test infrastructure.

Run a test using //garnet/bin/test_runner/run_test. For example:

$ fx exec garnet/bin/test_runner/run_test "device_runner --ledger_repository_for_testing --device_shell=dummy_device_shell --user_shell=dev_user_shell --user_shell_args=--root_module=/system/test/modular_tests/agent_trigger_test"

This will return when it has completed (either by succeeding or crashing). You can watch the QEMU console to see any console output written by test. In case of a crash, this console output will also be dumped by run_test.

You can also run a series of tests by supplying a JSON file describing the tests to run:

$ fx exec garnet/bin/test_runner/run_test --test_file=peridot/tests/modular_tests.json

Selecting between multiple devices

If you have more than one device, you can select a particular device with run_test's --server flag. The device name is displayed on the boot screen, or you can view all connected devices with the netls command, like this:

fx netls

Pass the device name to the run_test tool:

fx exec garnet/bin/test_runner/run_test --server rain-detour-glaze-donut ...

Fuchsia test infrastructure

For the Fuchsia automated test infrastructure, QEMU is configured to transparently route 127.0.0.1:8342 to a known IP address. ipv6 is not supported. run_test supports this environment by allowing you to specify an ipv4 address for the --server parameter. However, there are caveats:

• loglistener does not work because it requires ipv6. Therefore, no logging data will be captured if a test has errors.
• The --sync parameter is not supported.

Test Config Description

You can control the testing configuration by passing run_test the name of a JSON file with the --test_file parameter. For example:

fx exec garnet/bin/test_runner/run_test --test_file=peridot/tests/modular_tests.json

You can select a particular test to run by passing run_test the --test_name parameter in addition to the name of a JSON file with the --test_file parameter. For example:

fx exec garnet/bin/test_runner/run_test --test_file=peridot/tests/modular_tests.json --test_name=parent_child

The JSON file looks similar to this:

{
  "tests":[
    {
      "name":"dummy_user_shell",
      "exec":"device_runner --ledger_repository_for_testing --device_shell=dummy_device_shell --user_shell=dummy_user_shell"
    },
    {
      "name":"parent_child",
      "exec":"device_runner --ledger_repository_for_testing --device_shell=dummy_device_shell --user_shell=dev_user_shell --user_shell_args=--root_module=/system/test/modular_tests/parent_module",
      "copy":{
        "/system/test/modular_tests":[
          "parent_module",
          "child_module"
        ]
      }
    }
}

The top-level tests field is a list of tests to run, sequentially. Each test is an object with the following fields:

• name
  • A string field identifying the test. Required.
• exec
  • A string with the command representing the test to run. This will be run in a new application environment with a TestRunner service, which some part of the test is expected to use to report completion.
• copy
  • A map field where each key is the name of a directory on the Fuchsia system and each name value of the array is the name of a file on the host to be copied to that directory. The copy field is optional. If absent, the files currently on the device will be used. You must pass the --sync flag to run_test to cause the files to be copied. This does not work with QEMU devices accessed with ipv4.