| # CLI tests | |
| The CLI tests are focused on testing the zstd CLI. | |
| They are intended to be simple tests that the CLI and arguments work as advertised. | |
| They are not intended to test the library, only the code in `programs/`. | |
| The library will get incidental coverage, but if you find yourself trying to trigger a specific condition in the library, this is the wrong tool. | |
| ## Test runner usage | |
| The test runner `run.py` will run tests against the in-tree build of `zstd` and `datagen` by default. Which means that `zstd` and `datagen` must be built. | |
| The `zstd` binary used can be passed with `--zstd /path/to/zstd`. | |
| Additionally, to run `zstd` through a tool like `valgrind` or `qemu`, set the `--exec-prefix 'valgrind -q'` flag. | |
| Similarly, the `--datagen`, and `--zstdgrep` flags can be set to specify | |
| the paths to their respective binaries. However, these tools do not use | |
| the `EXEC_PREFIX`. | |
| Each test executes in its own scratch directory under `scratch/test/name`. E.g. `scratch/basic/help.sh/`. Normally these directories are removed after the test executes. However, the `--preserve` flag will preserve these directories after execution, and save the tests exit code, stdout, and stderr in the scratch directory to `exit`, `stderr`, and `stdout` respectively. This can be useful for debugging/editing a test and updating the expected output. | |
| ### Running all the tests | |
| By default the test runner `run.py` will run all the tests, and report the results. | |
| Examples: | |
| ``` | |
| ./run.py | |
| ./run.py --preserve | |
| ./run.py --zstd ../../build/programs/zstd --datagen ../../build/tests/datagen | |
| ``` | |
| ### Running specific tests | |
| A set of test names can be passed to the test runner `run.py` to only execute those tests. | |
| This can be useful for writing or debugging a test, especially with `--preserve`. | |
| The test name can either be the path to the test file, or the test name, which is the path relative to the test directory. | |
| Examples: | |
| ``` | |
| ./run.py basic/help.sh | |
| ./run.py --preserve basic/help.sh basic/version.sh | |
| ./run.py --preserve --verbose basic/help.sh | |
| ``` | |
| ### Updating exact output | |
| If a test is failing because a `.stderr.exact` or `.stdout.exact` no longer matches, you can re-run the tests with `--set-exact-output` and the correct output will be written. | |
| Example: | |
| ``` | |
| ./run.py --set-exact-output | |
| ./run.py basic/help.sh --set-exact-output | |
| ``` | |
| ## Writing a test | |
| Test cases are arbitrary executables, and can be written in any language, but are generally shell scripts. | |
| After the script executes, the exit code, stderr, and stdout are compared against the expectations. | |
| Each test is run in a clean directory that the test can use for intermediate files. This directory will be cleaned up at the end of the test, unless `--preserve` is passed to the test runner. Additionally, the `setup` script can prepare the directory before the test runs. | |
| ### Calling zstd, utilities, and environment variables | |
| The `$PATH` for tests is prepended with the `bin/` sub-directory, which contains helper scripts for ease of testing. | |
| The `zstd` binary will call the zstd binary specified by `run.py` with the correct `$EXEC_PREFIX`. | |
| Similarly, `datagen`, `unzstd`, `zstdgrep`, `zstdcat`, etc, are provided. | |
| Helper utilities like `cmp_size`, `println`, and `die` are provided here too. See their scripts for details. | |
| Common shell script libraries are provided under `common/`, with helper variables and functions. They can be sourced with `source "$COMMON/library.sh`. | |
| Lastly, environment variables are provided for testing, which can be listed when calling `run.py` with `--verbose`. | |
| They are generally used by the helper scripts in `bin/` to coordinate everything. | |
| ### Basic test case | |
| When executing your `$TEST` executable, by default the exit code is expected to be `0`. However, you can provide an alternate expected exit code in a `$TEST.exit` file. | |
| When executing your `$TEST` executable, by default the expected stderr and stdout are empty. However, you can override the default by providing one of three files: | |
| * `$TEST.{stdout,stderr}.exact` | |
| * `$TEST.{stdout,stderr}.glob` | |
| * `$TEST.{stdout,stderr}.ignore` | |
| If you provide a `.exact` file, the output is expected to exactly match, byte-for-byte. | |
| If you provide a `.glob` file, the output is expected to match the expected file, where each line is interpreted as a glob syntax. Additionally, a line containing only `...` matches all lines until the next expected line matches. | |
| If you provide a `.ignore` file, the output is ignored. | |
| #### Passing examples | |
| All these examples pass. | |
| Exit 1, and change the expectation to be 1. | |
| ``` | |
| exit-1.sh | |
| --- | |
| #!/bin/sh | |
| exit 1 | |
| --- | |
| exit-1.sh.exit | |
| --- | |
| 1 | |
| --- | |
| ``` | |
| Check the stdout output exactly matches. | |
| ``` | |
| echo.sh | |
| --- | |
| #!/bin/sh | |
| echo "hello world" | |
| --- | |
| echo.sh.stdout.exact | |
| --- | |
| hello world | |
| --- | |
| ``` | |
| Check the stderr output using a glob. | |
| ``` | |
| random.sh | |
| --- | |
| #!/bin/sh | |
| head -c 10 < /dev/urandom | xxd >&2 | |
| --- | |
| random.sh.stderr.glob | |
| --- | |
| 00000000: * * * * * * | |
| ``` | |
| Multiple lines can be matched with ... | |
| ``` | |
| random-num-lines.sh | |
| --- | |
| #!/bin/sh | |
| echo hello | |
| seq 0 $RANDOM | |
| echo world | |
| --- | |
| random-num-lines.sh.stdout.glob | |
| --- | |
| hello | |
| 0 | |
| ... | |
| world | |
| --- | |
| ``` | |
| #### Failing examples | |
| Exit code is expected to be 0, but is 1. | |
| ``` | |
| exit-1.sh | |
| --- | |
| #!/bin/sh | |
| exit 1 | |
| --- | |
| ``` | |
| Stdout is expected to be empty, but isn't. | |
| ``` | |
| echo.sh | |
| --- | |
| #!/bin/sh | |
| echo hello world | |
| ``` | |
| Stderr is expected to be hello but is world. | |
| ``` | |
| hello.sh | |
| --- | |
| #!/bin/sh | |
| echo world >&2 | |
| --- | |
| hello.sh.stderr.exact | |
| --- | |
| hello | |
| --- | |
| ``` | |
| ### Setup & teardown scripts | |
| Finally, test writing can be eased with setup and teardown scripts. | |
| Each directory in the test directory is a test-suite consisting of all tests within that directory (but not sub-directories). | |
| This test suite can come with 4 scripts to help test writing: | |
| * `setup_once` | |
| * `teardown_once` | |
| * `setup` | |
| * `teardown` | |
| The `setup_once` and `teardown_once` are run once before and after all the tests in the suite respectively. | |
| They operate in the scratch directory for the test suite, which is the parent directory of each scratch directory for each test case. | |
| They can do work that is shared between tests to improve test efficiency. | |
| For example, the `dictionaries/setup_once` script builds several dictionaries, for use in the `dictionaries` tests. | |
| The `setup` and `teardown` scripts run before and after each test case respectively, in the test case's scratch directory. | |
| These scripts can do work that is shared between test cases to make tests more succinct. | |
| For example, the `dictionaries/setup` script copies the dictionaries built by the `dictionaries/setup_once` script into the test's scratch directory, to make them easier to use, and make sure they aren't accidentally modified. | |
| #### Examples | |
| ``` | |
| basic/setup | |
| --- | |
| #!/bin/sh | |
| # Create some files for testing with | |
| datagen > file | |
| datagen > file0 | |
| datagen > file1 | |
| --- | |
| basic/test.sh | |
| --- | |
| #!/bin/sh | |
| zstd file file0 file1 | |
| --- | |
| dictionaries/setup_once | |
| --- | |
| #!/bin/sh | |
| set -e | |
| mkdir files/ dicts/ | |
| for i in $(seq 10); do | |
| datagen -g1000 > files/$i | |
| done | |
| zstd --train -r files/ -o dicts/0 | |
| --- | |
| dictionaries/setup | |
| --- | |
| #!/bin/sh | |
| # Runs in the test case's scratch directory. | |
| # The test suite's scratch directory that | |
| # `setup_once` operates in is the parent directory. | |
| cp -r ../files ../dicts . | |
| --- | |
| ``` |