| |
| |
| ### adding_test |
| Summary counts: |
| 2 test cases in total |
| 1 test case had no significant difference (no_sig_diff) |
| 0 test cases improved |
| 0 test cases regressed |
| 1 test case added |
| 0 test cases removed |
| |
| |
| Results from test cases with differences: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------ ---------------- ------------- ----------- -------------- |
| fuchsia.example: SecondExample added - - 1975 +/- 78 ns |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- ------------- -------------- |
| fuchsia.example: ClockGetTimeExample no_sig_diff 0.949-1.054 992 +/- 26 ns 992 +/- 26 ns |
| fuchsia.example: SecondExample added - - 1975 +/- 78 ns |
| |
| |
| ### comparison_improvement |
| Summary counts: |
| 2 test cases in total |
| 1 test case had no significant difference (no_sig_diff) |
| 1 test case improved |
| 0 test cases regressed |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from test cases with differences: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample improved 0.900-0.967 1492 +/- 26 ns 1392 +/- 26 ns |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample improved 0.900-0.967 1492 +/- 26 ns 1392 +/- 26 ns |
| fuchsia.example: SecondExample no_sig_diff 0.924-1.083 1975 +/- 78 ns 1975 +/- 78 ns |
| |
| |
| ### comparison_improvement_biggerisbetter |
| Summary counts: |
| 2 test cases in total |
| 1 test case had no significant difference (no_sig_diff) |
| 1 test case improved |
| 0 test cases regressed |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from test cases with differences: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- ------------------------ ------------------------ |
| fuchsia.example: ClockGetTimeExample improved 1.034-1.112 1392 +/- 26 bytes/second 1492 +/- 26 bytes/second |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- ------------------------ ------------------------ |
| fuchsia.example: ClockGetTimeExample improved 1.034-1.112 1392 +/- 26 bytes/second 1492 +/- 26 bytes/second |
| fuchsia.example: SecondExample no_sig_diff 0.924-1.083 1975 +/- 78 bytes/second 1975 +/- 78 bytes/second |
| |
| |
| ### comparison_improvement_small |
| Summary counts: |
| 2 test cases in total |
| 2 test cases had no significant difference (no_sig_diff) |
| 0 test cases improved |
| 0 test cases regressed |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample no_sig_diff 0.933-1.002 1492 +/- 26 ns 1442 +/- 26 ns |
| fuchsia.example: SecondExample no_sig_diff 0.924-1.083 1975 +/- 78 ns 1975 +/- 78 ns |
| |
| |
| ### comparison_no_change |
| Summary counts: |
| 2 test cases in total |
| 2 test cases had no significant difference (no_sig_diff) |
| 0 test cases improved |
| 0 test cases regressed |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample no_sig_diff 0.949-1.054 992 +/- 26 ns 992 +/- 26 ns |
| fuchsia.example: SecondExample no_sig_diff 0.924-1.083 1975 +/- 78 ns 1975 +/- 78 ns |
| |
| |
| ### comparison_no_change_zero_width_ci |
| Summary counts: |
| 1 test case in total |
| 1 test case had no significant difference (no_sig_diff) |
| 0 test cases improved |
| 0 test cases regressed |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| -------------------------- ---------------- ------------- ------------ ------------ |
| example_suite: ExampleTest no_sig_diff 1.000-1.000 200 +/- 0 ns 200 +/- 0 ns |
| |
| |
| ### comparison_regression |
| Summary counts: |
| 2 test cases in total |
| 1 test case had no significant difference (no_sig_diff) |
| 0 test cases improved |
| 1 test case regressed |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from test cases with differences: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample regressed 1.031-1.104 1492 +/- 26 ns 1592 +/- 26 ns |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample regressed 1.031-1.104 1492 +/- 26 ns 1592 +/- 26 ns |
| fuchsia.example: SecondExample no_sig_diff 0.924-1.083 1975 +/- 78 ns 1975 +/- 78 ns |
| |
| |
| ### comparison_regression_biggerisbetter |
| Summary counts: |
| 2 test cases in total |
| 1 test case had no significant difference (no_sig_diff) |
| 0 test cases improved |
| 1 test case regressed |
| 0 test cases added |
| 0 test cases removed |
| |
| |
| Results from test cases with differences: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- ------------------------ ------------------------ |
| fuchsia.example: ClockGetTimeExample regressed 0.900-0.967 1492 +/- 26 bytes/second 1392 +/- 26 bytes/second |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- ------------------------ ------------------------ |
| fuchsia.example: ClockGetTimeExample regressed 0.900-0.967 1492 +/- 26 bytes/second 1392 +/- 26 bytes/second |
| fuchsia.example: SecondExample no_sig_diff 0.924-1.083 1975 +/- 78 bytes/second 1975 +/- 78 bytes/second |
| |
| |
| ### display_single_boot_single_dataset |
| Test case Mean |
| ------------------------------------ ------------ |
| fuchsia.example: ClockGetTimeExample 991.7777 ns |
| fuchsia.example: SecondExample 1975.3331 ns |
| |
| |
| ### display_single_boot_two_datasets |
| Summary counts: |
| 2 test cases in total |
| 0 test cases had no significant difference (no_sig_diff) |
| 1 test case cannot be compared because we have point estimates only |
| 0 test cases improved |
| 0 test cases regressed |
| 0 test cases added |
| 1 test case removed |
| |
| |
| Results from test cases with differences: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------ ---------------- ------------- ------------ ---------- |
| fuchsia.example: SecondExample removed - 1975.3331 ns - |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- ------------ ------------ |
| fuchsia.example: ClockGetTimeExample point_estimate - 991.7777 ns 1991.7777 ns |
| fuchsia.example: SecondExample removed - 1975.3331 ns - |
| |
| |
| ### display_single_dataset |
| Test case Mean |
| ------------------------------------ -------------- |
| fuchsia.example: ClockGetTimeExample 992 +/- 26 ns |
| fuchsia.example: SecondExample 1975 +/- 78 ns |
| |
| |
| ### display_three_datasets |
| Test case Mean 1 Mean 2 Mean 3 |
| ------------------------------------ -------------- -------------- -------------- |
| fuchsia.example: ClockGetTimeExample 992 +/- 26 ns 1992 +/- 26 ns 2992 +/- 26 ns |
| fuchsia.example: SecondExample 1975 +/- 78 ns - 1975 +/- 78 ns |
| |
| |
| ### removing_test |
| Summary counts: |
| 2 test cases in total |
| 1 test case had no significant difference (no_sig_diff) |
| 0 test cases improved |
| 0 test cases regressed |
| 0 test cases added |
| 1 test case removed |
| |
| |
| Results from test cases with differences: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------ ---------------- ------------- -------------- ---------- |
| fuchsia.example: SecondExample removed - 1975 +/- 78 ns - |
| |
| |
| Results from all test cases: |
| |
| Test case Improve/regress? Factor change Mean before Mean after |
| ------------------------------------ ---------------- ------------- -------------- ------------- |
| fuchsia.example: ClockGetTimeExample no_sig_diff 0.949-1.054 992 +/- 26 ns 992 +/- 26 ns |
| fuchsia.example: SecondExample removed - 1975 +/- 78 ns - |
| |
| |
| ### run_local |
| |
| Results after 2 boots: |
| |
| Test case Mean |
| --------------------- -------------- |
| example_suite: MyTest 1010 +/- 82 ns |
| |
| |
| Results after 3 boots: |
| |
| Test case Mean |
| --------------------- -------------- |
| example_suite: MyTest 1009 +/- 10 ns |
| |
| |
| Results after 4 boots: |
| |
| Test case Mean |
| --------------------- ------------- |
| example_suite: MyTest 999 +/- 60 ns |
| |
| |
| |
| ### validate_perfcompare |
| Rate of mismatches (non-intersections) of confidence intervals for each test: |
| 0.512821 fuchsia.example: ClockGetTimeExample |
| 0.000000 fuchsia.example: SecondExample |
| |
| Mean mismatch rate: 0.256410 |
| Mean relative confidence interval width: 0.432586 |
| Number of test cases: 2 |
| Number of result sets: 40 groups of 5 boots each (ignoring 0 leftover boots) |
| Expected number of test cases with mismatches: 0.512821 |
