| # Perfcompare: Performance comparison try builder | |
| The "perfcompare" try builder is an optional CQ try builder for | |
| measuring the performance impact of a change without landing it | |
| (i.e. for pre-submit performance testing). It runs performance tests | |
| both with and without a CL applied and compares their results to see | |
| if there were any performance regressions or improvements. | |
| Googlers can refer to the [Google-internal perfcompare | |
| docs][internal-doc] for some additional documentation. | |
| ## How to use it | |
| ### For fuchsia.git CLs | |
| To run perfcompare on a Gerrit CL, do the following: | |
| * **Start a build:** Select "Choose tryjobs" in the Gerrit Web UI, | |
| and select one or more of the perfcompare builders from the list | |
| of builders. A quick way to do that is to type "perfcompare" into | |
| the search field, which will filter the list to display the | |
| available perfcompare builders. | |
| * **Get the results:** A link to the try builder's results page will | |
| appear on the CL in Gerrit. When the builder run is finished, the | |
| results will be under **"compare perf test results without and | |
| with CL" -> "stdout" (or "raw")** on the build page. | |
| These perfcompare builders are currently available and supported for | |
| running `fuchsia.git`'s performance tests: | |
| * `terminal.x64-release-perfcompare` | |
| ([recent builds](https://ci.chromium.org/p/fuchsia/builders/try/terminal.x64-release-perfcompare)): | |
| This runs `fuchsia.git`'s performance tests on Intel NUCs | |
| (x64). This is the perfcompare version of the | |
| `terminal.x64-release` builder (i.e. it runs the same set of | |
| performance tests as that builder). | |
| * `terminal.vim3-release-perfcompare` | |
| ([recent builds](https://ci.chromium.org/p/fuchsia/builders/try/terminal.vim3-release-perfcompare)): | |
| This runs `fuchsia.git`'s performance tests on VIM3s (ARM64). This | |
| is the perfcompare version of the `terminal.vim3-release` | |
| builder. Note that `terminal.vim3-release` is not run by the CQ by | |
| default, so it is more likely to be broken or have higher flake | |
| rates than other builders. | |
| ### For integration.git CLs | |
| **Perfcompare is not supported yet for `integration.git` CLs.** | |
| Specifically, CLs that change dependencies in Jiri manifest files or | |
| `jiri.lock` files or that use `patches.json` are not yet supported by | |
| perfcompare. This includes CLs that change prebuilt packages, such as | |
| toolchain roll CLs. | |
| Perfcompare does not know how to check out the source and prebuilt | |
| binaries before and after the CL in these cases, so it will give wrong | |
| results in these cases. It will produce a finding that there is no | |
| change in performance, even if the CL does change performance. | |
| ## Example output | |
| Here is part of the output from a perfcompare run on a simple [test | |
| CL]: | |
| [test CL]: <https://fuchsia-review.googlesource.com/c/fuchsia/+/482567> | |
| ```none | |
| Summary counts: | |
| 2939 test cases in total | |
| 2938 test cases had no significant difference (no_sig_diff) | |
| 1 test case got faster | |
| 0 test cases got slower | |
| 0 test cases added | |
| 0 test cases removed | |
| Results from test cases with differences: | |
| Test case Improve/regress? Factor change Mean before Mean after | |
| ---------------------------------------- ---------------- ------------- ------------------ ----------------- | |
| fuchsia.microbenchmarks: ExampleNoOpLoop faster 0.143-0.145 405.36 +/- 0.39 ns 58.49 +/- 0.30 ns | |
| Results from all test cases: | |
| Test case Improve/regress? Factor change Mean before Mean after | |
| --------------------------------------------- ---------------- ------------- ----------------- ----------------- | |
| ... | |
| fuchsia.microbenchmarks: Syscall/ManyArgs no_sig_diff 0.986-1.008 92.94 +/- 0.66 ns 92.65 +/- 0.40 ns | |
| fuchsia.microbenchmarks: Syscall/Null no_sig_diff 0.993-1.007 84.33 +/- 0.40 ns 84.31 +/- 0.19 ns | |
| fuchsia.microbenchmarks: Thread/CreateAndJoin no_sig_diff 0.950-1.034 34229 +/- 711 ns 33935 +/- 739 ns | |
| fuchsia.microbenchmarks: TicksGet no_sig_diff 0.981-1.022 19.77 +/- 0.19 ns 19.81 +/- 0.21 ns | |
| ... | |
| ``` | |
| ## Interpreting the results | |
| * `no_sig_diff` means that no [statistically | |
| significant](https://en.wikipedia.org/wiki/Statistical_significance) | |
| difference was found in the metric. It does not mean that there | |
| was no difference, just that any difference was too small | |
| (relative to the amount of variation in the metric) to be | |
| detected. | |
| * `ci_too_wide` is shown in the "Factor change" column if the | |
| confidence intervals in "Mean before" and "Mean after" are so wide | |
| that the lower bound went negative. This will happen for a metric | |
| if it has a large amount of variation. | |
| ## Testing CL stacks versus individual CLs | |
| The perfcompare builder measures the performance impact of individual | |
| CLs, **not** stacks of CLs. | |
| As an example, suppose you have a series of CLs: P1, P2, P3, P4, P5, | |
| where P1 is the oldest (that is, all the other CLs depend on it). If | |
| you run perfcompare on P3, the "with CL" build will include P1+P2+P3, | |
| while the "without CL" build will include just P1+P2. | |
| * This provides a way to measure effects on test cases that haven't | |
| been landed yet. You can have one CL that adds a new performance | |
| test, and a follow-on CL that changes the | |
| software-under-test. Running perfcompare on the second CL will | |
| show how that CL affects the new test. | |
| * If you do want to measure the overall effect of a patch stack, one | |
| way to do that is to squash the changes into a single Git commit | |
| (such as with `git merge --squash`), upload that to Gerrit, and | |
| run perfcompare on that. | |
| ### The "with CL" and "without CL" builds | |
| The perfcompare builder applies the following steps sequentially to | |
| produce the "with CL" and "without CL" builds: | |
| 1. Check out Fuchsia from the current tip-of-tree revision of | |
| `integration.git`. | |
| 2. Apply the CL series to the checkout, up to and including the CL | |
| being tested. This uses `jiri patch`, which uses `git rebase`. | |
| 3. Build Fuchsia. This gives the "with CL" build. | |
| 4. Unapply the topmost CL from the checkout (leaving earlier CLs in | |
| the CL series, if any, applied). This works by running `git | |
| checkout HEAD^` in the Git repo where the CL series was applied. | |
| 5. Build Fuchsia again, doing an incremental build. This gives the | |
| "without CL" build. | |
| Steps 1-3 are the same as for non-perfcompare Fuchsia try builders. | |
| ## Limitations | |
| * CLs that use `patches.json` or that change dependencies in Jiri | |
| manifest files are not supported yet, as mentioned above. | |
| ## How to run performance comparisons locally | |
| The perfcompare builders use | |
| [`perfcompare.py`](/src/testing/perfcompare/perfcompare.py) to compare | |
| performance results. It is possible to use `perfcompare.py` to run | |
| performance tests locally (that is, not using Fuchsia Infra) and | |
| compare their results. See the | |
| [documentation](/src/testing/perfcompare/README.md). | |
| ## How to download the raw performance results | |
| <!-- Allow lines to be wrapped in the code blocks below. For single | |
| shell commands, wrapping can be more convenient for | |
| copy-and-pasting than using backslashes for splitting up | |
| lines. --> | |
| <style> | |
| pre.wrapped { | |
| white-space: pre-wrap; | |
| } | |
| </style> | |
| It is possible to download the raw performance test results produced | |
| by a perfcompare try builder run. This is useful if you want to modify | |
| the analysis that `perfcompare.py` performs. To do that, use the | |
| following steps: | |
| 1. Find the values of the `cas_instance` and | |
| `perfcompare_dataset_digest` fields from the output properties of | |
| the perfcompare build. These can be found on the build page for | |
| the build (which is reachable from the "Checks" tab in the Gerrit | |
| code review). Examples of typical values are: | |
| * `cas_instance="projects/chromium-swarm/instances/default_instance"` | |
| * `perfcompare_dataset_digest="3ff389154e02490f29e379564f7e70b3df66f74c3116ed50172cceec1e9d9888/165"` | |
| For downloading results data from non-perfcompare builds, the | |
| field name to use is `perf_dataset_digest` rather than | |
| `perfcompare_dataset_digest`. | |
| 2. Download the dataset by running the following command (using the | |
| prebuilt `cas` tool from the Fuchsia checkout): | |
| ```shell {:.wrapped} | |
| ./prebuilt/tools/cas/cas download -cas-instance $CAS_INSTANCE -digest $DIGEST -dir $DEST_DIR | |
| ``` | |
| 3. Run `perfcompare.py` on the downloaded dataset: | |
| ```shell {:.wrapped} | |
| python3 src/testing/perfcompare/perfcompare.py compare_perf $DEST_DIR/without_cl/ $DEST_DIR/with_cl/ | |
| ``` | |
| Note that the RBE-CAS system keeps the data for only about 2-3 months, | |
| so the download command will fail if the build was not run | |
| recently. (The current default for the time-to-live (TTL) in | |
| RBE-CAS is 90 days.) | |
| [internal-doc]: <https://goto.google.com/fuchsia-perfcompare-internal> |