# perfcompare: Performance comparison tool

## Running the unit tests

The unit tests can be run using Fuchsia's GN plumbing for Python code,
which is how the tests are run on Fuchsia Infra.  Example:

```sh
fx set core.x64 --with-host src/testing/perfcompare:tests
fx test perfcompare_test
```

Alternatively, they can be run directly using `fuchsia-vendored-python` or
`python3`:

```sh
fuchsia-vendored-python src/testing/perfcompare/perfcompare_test.py
```

This case allows the golden files to be updated when running the test:

```sh
fuchsia-vendored-python src/testing/perfcompare/perfcompare_test.py --generate
```

## Running stats_test.py

perfcompare.py used to depend on SciPy, but that dependency was difficult
to provide in the Fuchsia GN build.  That dependency was removed: the
relevant function has been reimplemented in stats.py.  However,
stats_test.py can test stats.py against SciPy, and hence depends on SciPy.
There are two ways that stats_test.py can be run:

* Via `vpython3`:

  ```sh
  ./prebuilt/third_party/vpython/vpython3 src/testing/perfcompare/stats_test.py
  ```

  This will automatically download prebuilt, hermetic versions of
  dependencies.  Note that `vpython3` is used for running perfcompare.py on
  the Fuchsia Infra builders.

* On Linux, when using Debian/Ubuntu, the dependencies can be
  installed using APT:

  ```sh
  sudo apt-get install python3-scipy
  python3 src/testing/perfcompare/stats_test.py
  ```

## Example: Running perf tests locally and comparing results

The perfcompare tool can be used to run perf tests locally and to
compare the results.

As an example, suppose you want to compare the results from
`rust_inspect_benchmarks_test` on two Git commits,
`BEFORE_VERSION` and `AFTER_VERSION`.

This test case is tested in the `terminal` product and may not work in
a different product.

The following commands would gather a dataset of perf test results for
`BEFORE_VERSION` and save them in the directory `perf_results_before`:

```sh
git checkout BEFORE_VERSION
# Covers dependencies of rust_inspect_benchmarks_test.
fx set terminal.x64 --with //bundles/buildbot/terminal
fx build
fx update
python3 src/testing/perfcompare/perfcompare.py run_local \
  --boots=5 \
  --iter_cmd='fx test --e2e rust_inspect_benchmarks_test' \
  --iter_file='out/test_out/*/*.fuchsiaperf.json' \
  --dest=perf_results_before
```

These commands would do the same, but for `AFTER_VERSION`, saving the
results dataset in a different directory, `perf_results_after`:

```sh
git checkout AFTER_VERSION
fx build
fx update
python3 src/testing/perfcompare/perfcompare.py run_local \
  --boots=5 \
  --iter_cmd='fx test --e2e rust_inspect_benchmarks_test' \
  --iter_file='out/test_out/*/*.fuchsiaperf.json' \
  --dest=perf_results_after
```

Note that the `run_local` commands will reboot Fuchsia.

The two datasets can then be compared with the following command,
which prints a table showing the "before" and "after" results side by
side:

```sh
python3 src/testing/perfcompare/perfcompare.py compare_perf perf_results_before perf_results_after
```