| |
| |
| ### adding_test |
| Summary counts: |
| 2 test cases in total |
| 1 test case had no significant difference (no_sig_diff) |
| 0 test cases got faster |
| 0 test cases got slower |
| 1 test case added |
| 0 test cases removed |
| |
| |
| Results from test cases with differences: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------ ---------------- ------------- ----------- -------------- |
| fuchsia.example: SecondExample added - - 1975 +/- 78 ns |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- ------------- -------------- |
| fuchsia.example: ClockGetTimeExample no_sig_diff 0.949-1.054 992 +/- 26 ns 992 +/- 26 ns |
| fuchsia.example: SecondExample added - - 1975 +/- 78 ns |
| |
| |
| ### comparison_improvement |
| Summary counts: |
| 2 test cases in total |
| 1 test case had no significant difference (no_sig_diff) |
| 1 test case got faster |
| 0 test cases got slower |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from test cases with differences: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample faster 0.900-0.967 1492 +/- 26 ns 1392 +/- 26 ns |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample faster 0.900-0.967 1492 +/- 26 ns 1392 +/- 26 ns |
| fuchsia.example: SecondExample no_sig_diff 0.924-1.083 1975 +/- 78 ns 1975 +/- 78 ns |
| |
| |
| ### comparison_improvement_small |
| Summary counts: |
| 2 test cases in total |
| 2 test cases had no significant difference (no_sig_diff) |
| 0 test cases got faster |
| 0 test cases got slower |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample no_sig_diff 0.933-1.002 1492 +/- 26 ns 1442 +/- 26 ns |
| fuchsia.example: SecondExample no_sig_diff 0.924-1.083 1975 +/- 78 ns 1975 +/- 78 ns |
| |
| |
| ### comparison_no_change |
| Summary counts: |
| 2 test cases in total |
| 2 test cases had no significant difference (no_sig_diff) |
| 0 test cases got faster |
| 0 test cases got slower |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample no_sig_diff 0.949-1.054 992 +/- 26 ns 992 +/- 26 ns |
| fuchsia.example: SecondExample no_sig_diff 0.924-1.083 1975 +/- 78 ns 1975 +/- 78 ns |
| |
| |
| ### comparison_no_change_zero_width_ci |
| Summary counts: |
| 1 test case in total |
| 1 test case had no significant difference (no_sig_diff) |
| 0 test cases got faster |
| 0 test cases got slower |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| -------------------------- ---------------- ------------- ------------ ------------ |
| example_suite: ExampleTest no_sig_diff 1.000-1.000 200 +/- 0 ns 200 +/- 0 ns |
| |
| |
| ### comparison_regression |
| Summary counts: |
| 2 test cases in total |
| 1 test case had no significant difference (no_sig_diff) |
| 0 test cases got faster |
| 1 test case got slower |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from test cases with differences: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample slower 1.031-1.104 1492 +/- 26 ns 1592 +/- 26 ns |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample slower 1.031-1.104 1492 +/- 26 ns 1592 +/- 26 ns |
| fuchsia.example: SecondExample no_sig_diff 0.924-1.083 1975 +/- 78 ns 1975 +/- 78 ns |
| |
| |
| ### removing_test |
| Summary counts: |
| 2 test cases in total |
| 1 test case had no significant difference (no_sig_diff) |
| 0 test cases got faster |
| 0 test cases got slower |
| 0 test cases added |
| 1 test case removed |
| |
| |
| Results from test cases with differences: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------ ---------------- ------------- -------------- ---------- |
| fuchsia.example: SecondExample removed - 1975 +/- 78 ns - |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- ------------- |
| fuchsia.example: ClockGetTimeExample no_sig_diff 0.949-1.054 992 +/- 26 ns 992 +/- 26 ns |
| fuchsia.example: SecondExample removed - 1975 +/- 78 ns - |
| |
| |
| ### run_local |
| |
| Results after 2 boots: |
| |
| Test case Mean |
| --------------------- -------------- |
| example_suite: MyTest 1010 +/- 82 ns |
| |
| |
| Results after 3 boots: |
| |
| Test case Mean |
| --------------------- -------------- |
| example_suite: MyTest 1009 +/- 10 ns |
| |
| |
| Results after 4 boots: |
| |
| Test case Mean |
| --------------------- ------------- |
| example_suite: MyTest 999 +/- 60 ns |
| |
| |
| |
| ### validate_perfcompare |
| Rate of mismatches (non-intersections) of confidence intervals for each test: |
| 0.512821 fuchsia.example: ClockGetTimeExample |
| 0.000000 fuchsia.example: SecondExample |
| |
| Mean mismatch rate: 0.256410 |
| Mean relative confidence interval width: 0.432586 |
| Number of test cases: 2 |
| Number of result sets: 40 groups of 5 boots each (ignoring 0 leftover boots) |
| Expected number of test cases with mismatches: 0.512821 |
