The perfcompare tool requires Python 3.
There are two ways to run the perfcompare tool with the required dependencies (currently the scipy Python library, which depends on the numpy Python library):
Via vpython3:
./prebuilt/third_party/vpython/vpython3 src/testing/perfcompare/perfcompare.py
This will automatically download prebuilt, hermetic versions of dependencies. vpython3 is used for running perfcompare on the Infra builders.
On Linux, when using Debian/Ubuntu, the dependencies can be installed using APT:
sudo apt-get install python3-scipy
This command should install numpy as well.
The unit tests used to be run on CI/CQ, but that regressed. Running them on CI/CQ is somewhat difficult now because of the dependency on scipy, so for now the tests must be run manually, with:
python3 src/testing/perfcompare/perfcompare_test.py
The perfcompare tool can be used to run perf tests locally and to compare the results.
As an example, suppose you want to compare the results from rust_inspect_benchmarks_test on two Git commits, BEFORE_VERSION and AFTER_VERSION.
This test case is tested in the terminal product and may not work in a different product.
The following commands would gather a dataset of perf test results for BEFORE_VERSION and save them in the directory perf_results_before:
git checkout BEFORE_VERSION # Covers dependencies of rust_inspect_benchmarks_test. fx set terminal.x64 --with //bundles/buildbot/terminal fx build fx update python3 src/testing/perfcompare/perfcompare.py run_local \ --boots=5 \ --iter_cmd='fx test --e2e rust_inspect_benchmarks_test' \ --iter_file='out/test_out/*/*.fuchsiaperf.json' \ --dest=perf_results_before
These commands would do the same, but for AFTER_VERSION, saving the results dataset in a different directory, perf_results_after:
git checkout AFTER_VERSION fx build fx update python3 src/testing/perfcompare/perfcompare.py run_local \ --boots=5 \ --iter_cmd='fx test --e2e rust_inspect_benchmarks_test' \ --iter_file='out/test_out/*/*.fuchsiaperf.json' \ --dest=perf_results_after
Note that the run_local commands will reboot Fuchsia.
The two datasets can then be compared with the following command, which prints a table showing the “before” and “after” results side by side:
python3 src/testing/perfcompare/perfcompare.py compare_perf perf_results_before perf_results_after