tree bde47d4982ee235256b2f54b3fb6a9ac27619015
parent 25b167abbb661c57289d0a6904c0df45626a73d5
author Mark Seaborn <mseaborn@google.com> 1555565665 +0000
committer CQ bot account: commit-bot@chromium.org <commit-bot@chromium.org> 1555565665 +0000

[perfcompare] Do multiple launches of each test process instead of just one

We've found that the mean running times of performance tests tend to
vary between launches of the same process and between reboots.

We want to calculate the mean across process launches and reboots, so
we need to sample accordingly.  This change implements sampling across
process relaunches.  Sampling across reboots will be done later.

 * Before: We launch the perftest process once, and (for each test
   case) do 1000 test runs in that process.  We treat this as a sample
   of size 1000 when calculating confidence intervals.

 * After: We launch the perftest process 30 times, and (for each test
   case) do 100 test runs in each process.  We treat this as a sample
   of size 30 when calculating confidence intervals.

Bug: IN-646
Test: "python perfcompare_test.py" + perfcompare trybot
Change-Id: Ic0176dcc394e87015302bbe3d8588a68408fca05