blob: a321ae927935d32a1735dbb84a9c21a7d03fd1a1 [file] [log] [blame] [view]
# Testing
The test harness which runs on our bots (called "runtests") picks up all
executables in the "/boot/test" and "/system/test" directories and runs them.
If you provide a command-line argument, such as `runtests -S -m widget_test`,
runtests will only run the single test requested -- in this case, `widget_test`.
"runtests" takes command-line arguments to toggle classes of tests to execute.
These classes are the following:
* **Small**: Isolated tests for functions and classes. These must be totally
synchronous and single-threaded. These tests should be parallelizable; there
shouldn't be any shared resources between them.
* **Medium**: Single-process integration tests. Ideally these are also synchronous
and single-threaded but they might run through a large chunk of code in each
test case, or they might use disk, making them a bit slower.
* **Large**: Slow, multi-process, or particularly incomprehensible single-process
integration tests. These tests are often too slow / flaky to run in a CQ, and
we should try to limit how many we have.
* **Performance**: Tests which are expected to pass, but which are measured
using other metrics (thresholds, statistical techniques) to identify
Since runtests doesn't really know what "class" is executing when it launches a
test, it encodes this information in the environment variable
`RUNTESTS_TEST_CLASS`, which is detailed in [the unittest
header][unittest-header] , and lets the executable itself decide what to run /
not run. This environment variable is a bitmask indicating which tests to run.
For example, if a a test executable is run with "small" and "medium" tests,
it will be executed ONCE with `RUNTESTS_TEST_CLASS` set to 00000003 (the
hex bitwise OR of "TEST_SMALL" and "TEST_MEDIUM" -- though this information
should be parsed using the [unittest header][unittest-header], as it may be
updated in the future).
## Magenta Tests (ulib/test, and/or using ulib/unittest)
The following macros can be used to filter tests into these categories:
The legacy `RUN_TEST(widget_test)` is aliased to mean the same thing as
## Fuchsia Tests (not using ulib/unittest)
The environment variable `RUNTESTS_TEST_CLASS` will still be available to all
executables launched by runtests. The [unittest header][unittest-header] can be
used to parse different categories of tests which the runtests harness attempted
to run.
## Runtests CLI
By default, runtests will run both small and medium tests.
To determine how to run a custom set of test categories, run `runtests -h`,
which includes usage information.
[unittest-header]: ../system/ulib/unittest/include/unittest/unittest.h "Unittest Header"