The “perfcompare” try builder is an optional CQ try builder for measuring the performance impact of a change without landing it (i.e. for pre-submit performance testing). It runs performance tests both with and without a CL applied and compares their results to see if there were any performance regressions or improvements.
Googlers can refer to the Google-internal perfcompare docs for some additional documentation.
To run perfcompare on a Gerrit CL, do the following:
These perfcompare builders are currently available and supported for running fuchsia.git
's performance tests:
terminal.x64-release-perfcompare
(recent builds): This runs fuchsia.git
's performance tests on Intel NUCs (x64). This is the perfcompare version of the terminal.x64-release
builder (i.e. it runs the same set of performance tests as that builder).
terminal.vim3-release-perfcompare
(recent builds): This runs fuchsia.git
's performance tests on VIM3s (ARM64). This is the perfcompare version of the terminal.vim3-release
builder. Note that terminal.vim3-release
is not run by the CQ by default, so it is more likely to be broken or have higher flake rates than other builders.
Perfcompare is not supported yet for integration.git
CLs.
Specifically, CLs that change dependencies in Jiri manifest files or jiri.lock
files or that use patches.json
are not yet supported by perfcompare. This includes CLs that change prebuilt packages, such as toolchain roll CLs.
Perfcompare does not know how to check out the source and prebuilt binaries before and after the CL in these cases, so it will give wrong results in these cases. It will produce a finding that there is no change in performance, even if the CL does change performance.
Here is part of the output from a perfcompare run on a simple test CL:
Summary counts: 2939 test cases in total 2938 test cases had no significant difference (no_sig_diff) 1 test case got faster 0 test cases got slower 0 test cases added 0 test cases removed Results from test cases with differences: Test case Improve/regress? Factor change Mean before Mean after ---------------------------------------- ---------------- ------------- ------------------ ----------------- fuchsia.microbenchmarks: ExampleNoOpLoop faster 0.143-0.145 405.36 +/- 0.39 ns 58.49 +/- 0.30 ns Results from all test cases: Test case Improve/regress? Factor change Mean before Mean after --------------------------------------------- ---------------- ------------- ----------------- ----------------- ... fuchsia.microbenchmarks: Syscall/ManyArgs no_sig_diff 0.986-1.008 92.94 +/- 0.66 ns 92.65 +/- 0.40 ns fuchsia.microbenchmarks: Syscall/Null no_sig_diff 0.993-1.007 84.33 +/- 0.40 ns 84.31 +/- 0.19 ns fuchsia.microbenchmarks: Thread/CreateAndJoin no_sig_diff 0.950-1.034 34229 +/- 711 ns 33935 +/- 739 ns fuchsia.microbenchmarks: TicksGet no_sig_diff 0.981-1.022 19.77 +/- 0.19 ns 19.81 +/- 0.21 ns ...
The perfcompare builder measures the performance impact of individual CLs, not stacks of CLs.
As an example, suppose you have a series of CLs: P1, P2, P3, P4, P5, where P1 is the oldest (that is, all the other CLs depend on it). If you run perfcompare on P3, the “with CL” build will include P1+P2+P3, while the “without CL” build will include just P1+P2.
git merge --squash
), upload that to Gerrit, and run perfcompare on that.The perfcompare builder applies the following steps sequentially to produce the “with CL” and “without CL” builds:
integration.git
.jiri patch
, which uses git rebase
.git checkout HEAD^
in the Git repo where the CL series was applied.Steps 1-3 are the same as for non-perfcompare Fuchsia try builders.
patches.json
or that change dependencies in Jiri manifest files are not supported yet, as mentioned above.The perfcompare builders use perfcompare.py
to compare performance results. It is possible to use perfcompare.py
to run performance tests locally (that is, not using Fuchsia Infra) and compare their results. See the documentation.
It is possible to download the raw performance test results produced by a perfcompare try builder run. This is useful if you want to modify the analysis that perfcompare.py
performs. To do that, use the following steps:
Find the values of the cas_instance
and perfcompare_dataset_digest
fields from the output properties of the perfcompare build. These can be found on the build page for the build (which is reachable from the “Checks” tab in the Gerrit code review). Examples of typical values are:
cas_instance="projects/chromium-swarm/instances/default_instance"
perfcompare_dataset_digest="3ff389154e02490f29e379564f7e70b3df66f74c3116ed50172cceec1e9d9888/165"
For downloading results data from non-perfcompare builds, the field name to use is perf_dataset_digest
rather than perfcompare_dataset_digest
.
Download the dataset by running the following command (using the prebuilt cas
tool from the Fuchsia checkout):
./prebuilt/tools/cas/cas download -cas-instance $CAS_INSTANCE -digest $DIGEST -dir $DEST_DIR
Run perfcompare.py
on the downloaded dataset:
python3 src/testing/perfcompare/perfcompare.py compare_perf $DEST_DIR/without_cl/ $DEST_DIR/with_cl/
Note that the RBE-CAS system keeps the data for only about 2-3 months, so the download command will fail if the build was not run recently. (The current default for the time-to-live (TTL) in RBE-CAS is 90 days.)