garnet/bin/perfcompare/perfcompare_test_output.txt

### adding_test
Summary counts:
  2 test cases in total
  1 test case had no significant difference (no_sig_diff)
  0 test cases got faster
  0 test cases got slower
  1 test case added
  0 test cases removed


Results from test cases with differences:

Test case      Improve/regress?  Factor change  Mean before  Mean after
-------------  ----------------  -------------  -----------  -----------
SecondExample  added             -              -            1975 +/- 78


Results from all test cases:

Test case            Improve/regress?  Factor change  Mean before  Mean after
-------------------  ----------------  -------------  -----------  -----------
ClockGetTimeExample  no_sig_diff       0.949-1.054    991 +/- 26   991 +/- 26
SecondExample        added             -              -            1975 +/- 78


### comparison_improvement
Summary counts:
  2 test cases in total
  1 test case had no significant difference (no_sig_diff)
  1 test case got faster
  0 test cases got slower
  0 test cases added
  0 test cases removed


Results from test cases with differences:

Test case            Improve/regress?  Factor change  Mean before  Mean after
-------------------  ----------------  -------------  -----------  -----------
ClockGetTimeExample  faster            0.900-0.967    1491 +/- 26  1391 +/- 26


Results from all test cases:

Test case            Improve/regress?  Factor change  Mean before  Mean after
-------------------  ----------------  -------------  -----------  -----------
ClockGetTimeExample  faster            0.900-0.967    1491 +/- 26  1391 +/- 26
SecondExample        no_sig_diff       0.924-1.083    1975 +/- 78  1975 +/- 78


### comparison_improvement_small
Summary counts:
  2 test cases in total
  2 test cases had no significant difference (no_sig_diff)
  0 test cases got faster
  0 test cases got slower
  0 test cases added
  0 test cases removed


Results from all test cases:

Test case            Improve/regress?  Factor change  Mean before  Mean after
-------------------  ----------------  -------------  -----------  -----------
ClockGetTimeExample  no_sig_diff       0.933-1.002    1491 +/- 26  1441 +/- 26
SecondExample        no_sig_diff       0.924-1.083    1975 +/- 78  1975 +/- 78


### comparison_no_change
Summary counts:
  2 test cases in total
  2 test cases had no significant difference (no_sig_diff)
  0 test cases got faster
  0 test cases got slower
  0 test cases added
  0 test cases removed


Results from all test cases:

Test case            Improve/regress?  Factor change  Mean before  Mean after
-------------------  ----------------  -------------  -----------  -----------
ClockGetTimeExample  no_sig_diff       0.949-1.054    991 +/- 26   991 +/- 26
SecondExample        no_sig_diff       0.924-1.083    1975 +/- 78  1975 +/- 78


### comparison_regression
Summary counts:
  2 test cases in total
  1 test case had no significant difference (no_sig_diff)
  0 test cases got faster
  1 test case got slower
  0 test cases added
  0 test cases removed


Results from test cases with differences:

Test case            Improve/regress?  Factor change  Mean before  Mean after
-------------------  ----------------  -------------  -----------  -----------
ClockGetTimeExample  slower            1.031-1.104    1491 +/- 26  1591 +/- 26


Results from all test cases:

Test case            Improve/regress?  Factor change  Mean before  Mean after
-------------------  ----------------  -------------  -----------  -----------
ClockGetTimeExample  slower            1.031-1.104    1491 +/- 26  1591 +/- 26
SecondExample        no_sig_diff       0.924-1.083    1975 +/- 78  1975 +/- 78


### removing_test
Summary counts:
  2 test cases in total
  1 test case had no significant difference (no_sig_diff)
  0 test cases got faster
  0 test cases got slower
  0 test cases added
  1 test case removed


Results from test cases with differences:

Test case      Improve/regress?  Factor change  Mean before  Mean after
-------------  ----------------  -------------  -----------  ----------
SecondExample  removed           -              1975 +/- 78  -


Results from all test cases:

Test case            Improve/regress?  Factor change  Mean before  Mean after
-------------------  ----------------  -------------  -----------  ----------
ClockGetTimeExample  no_sig_diff       0.949-1.054    991 +/- 26   991 +/- 26
SecondExample        removed           -              1975 +/- 78  -


### validate_perfcompare
Rate of mismatches (non-intersections) of confidence intervals for each test:
0.512821 ClockGetTimeExample
0.000000 SecondExample

Mean mismatch rate: 0.256410
Mean relative confidence interval width: 0.432586
Number of test cases: 2
Number of result sets: 40 groups of 5 boots each (ignoring 0 leftover boots)
Expected number of test cases with mismatches: 0.512821