Benchmark_O
TestsThe Benchmark_O
binary is used directly from command line as well as a subcomponent invoked from higher-level scripts (eg. Benchmark_Driver
). These script therefore depend on the supported command line options and the format of its console output. The following lit
tests also serve as a verification of this public API to prevent its accidental breakage.
Note: Following tests use Ackermann as an example of a benchmark that is excluded from the default “pre-commit” list because it is marked unstable
and the default skip-tags (unstable,skip
) will exclude it. It's also alphabetically the first benchmark in the test suite (used to verify running by index). If these assumptions change, the test must be adapted.
RUN: %Benchmark_O --list | %FileCheck %s \ RUN: --check-prefix LISTPRECOMMIT \ RUN: --check-prefix LISTTAGS LISTPRECOMMIT: #,Test,[Tags] LISTPRECOMMIT-NOT: Ackermann LISTPRECOMMIT: {{[0-9]+}},AngryPhonebook LISTTAGS-SAME: ,[ LISTTAGS-NOT: TestsUtils.BenchmarkCategory. LISTTAGS-SAME: String, api, validation LISTTAGS-SAME: ]
Verify Ackermann is listed when skip-tags are explicitly empty and that it is marked unstable:
RUN: %Benchmark_O --list --skip-tags= | %FileCheck %s --check-prefix LISTALL LISTALL: Ackermann LISTALL-SAME: unstable LISTALL: AngryPhonebook
The logic for filtering tests based on specified names, indices and tags is shared between the default “run” and --list
commands. It is tested on the list command, which is much faster, because it runs no benchmarks. It provides us with ability to do a “dry run”.
Run benchmark by name (even if its tags match the skip-tags) or test number:
RUN: %Benchmark_O Ackermann --list | %FileCheck %s --check-prefix NAMEDSKIP NAMEDSKIP: Ackermann RUN: %Benchmark_O 1 --list | %FileCheck %s --check-prefix RUNBYNUMBER RUNBYNUMBER: Ackermann
Composition of tags
and skip-tags
:
RUN: %Benchmark_O --list --tags=Dictionary,Array \ RUN: | %FileCheck %s --check-prefix ANDTAGS ANDTAGS: TwoSum ANDTAGS-NOT: Array2D ANDTAGS-NOT: DictionarySwap RUN: %Benchmark_O --list --tags=algorithm --skip-tags=validation \ RUN: | %FileCheck %s --check-prefix TAGSANDSKIPTAGS TAGSANDSKIPTAGS: Ackermann TAGSANDSKIPTAGS: DictOfArraysToArrayOfDicts TAGSANDSKIPTAGS: Fibonacci TAGSANDSKIPTAGS: RomanNumbers RUN: %Benchmark_O --list --tags=algorithm \ RUN: --skip-tags=validation,Dictionary,String \ RUN: | %FileCheck %s --check-prefix ORSKIPTAGS ORSKIPTAGS: Ackermann ORSKIPTAGS-NOT: DictOfArraysToArrayOfDicts ORSKIPTAGS: Fibonacci ORSKIPTAGS-NOT: RomanNumbers
Alphabetic sorting of tests
RUN: %Benchmark_O --list \ RUN: | %FileCheck %s --check-prefix ALPHASORT ALPHASORT: COWArrayGuaranteedParameterOverhead ALPHASORT: COWTree ALPHASORT: ChainedFilterMap ALPHASORT: Chars ALPHASORT: FatCompactMap
Each real benchmark execution takes about a second per sample. If possible, multiple checks are combined into one run to minimize the test time.
RUN: %Benchmark_O AngryPhonebook --num-iters=1 \ RUN: | %FileCheck %s --check-prefix NUMITERS1 \ RUN: --check-prefix LOGHEADER \ RUN: --check-prefix LOGBENCH LOGHEADER-LABEL: #,TEST,SAMPLES,MIN(us),MAX(us),MEAN(us),SD(us),MEDIAN(us) LOGBENCH: {{[0-9]+}}, NUMITERS1: AngryPhonebook,1 NUMITERS1-NOT: 0,0,0,0,0 LOGBENCH-SAME: ,{{[0-9]+}},{{[0-9]+}},{{[0-9]+}},{{[0-9]+}},{{[0-9]+}}
RUN: %Benchmark_O 1 Ackermann 1 AngryPhonebook \ RUN: --verbose --num-samples=2 --memory \ RUN: | %FileCheck %s --check-prefix RUNJUSTONCE \ RUN: --check-prefix CONFIG \ RUN: --check-prefix LOGVERBOSE \ RUN: --check-prefix MEASUREENV \ RUN: --check-prefix LOGMEMORY CONFIG: NumSamples: 2 CONFIG: Tests Filter: ["1", "Ackermann", "1", "AngryPhonebook"] CONFIG: Tests to run: Ackermann, AngryPhonebook LOGMEMORY: #,TEST,SAMPLES,MIN(us),MAX(us),MEAN(us),SD(us),MEDIAN(us),MAX_RSS(B) LOGVERBOSE-LABEL: Running Ackermann for 2 samples. LOGVERBOSE: Measuring with scale {{[0-9]+}}. LOGVERBOSE-NEXT: Sample 0,{{[0-9]+}} LOGVERBOSE-NEXT: Measuring with scale {{[0-9]+}}. LOGVERBOSE-NEXT: Sample 1,{{[0-9]+}} MEASUREENV: MAX_RSS {{[0-9]+}} - {{[0-9]+}} = {{[0-9]+}} ({{[0-9]+}} pages) MEASUREENV: ICS {{[0-9]+}} - {{[0-9]+}} = {{[0-9]+}} MEASUREENV: VCS {{[0-9]+}} - {{[0-9]+}} = {{[0-9]+}} RUNJUSTONCE-LABEL: 1,Ackermann RUNJUSTONCE-NOT: 1,Ackermann LOGMEMORY: ,{{[0-9]+}},{{[0-9]+}},{{[0-9]+}},{{[0-9]+}},{{[0-9]+}},{{[0-9]+}} LOGVERBOSE-LABEL: Running AngryPhonebook for 2 samples.
RUN: not %Benchmark_O --bogus 2>&1 \ RUN: | %FileCheck %s --check-prefix ARGPARSE ARGPARSE: error: unsupported argument '--bogus' RUN: not %Benchmark_O --sample-time \ RUN: 2>&1 | %FileCheck %s --check-prefix NOVALUE NOVALUE: error: missing value for '--sample-time' RUN: not %Benchmark_O --sample-time= \ RUN: 2>&1 | %FileCheck %s --check-prefix EMPTYVAL EMPTYVAL: error: missing value for '--sample-time' RUN: not %Benchmark_O --sample-time=NaN \ RUN: 2>&1 | %FileCheck %s --check-prefix NANVALUE NANVALUE: error: 'NaN' is not a valid 'Double' for '--sample-time' RUN: not %Benchmark_O --num-iters \ RUN: 2>&1 | %FileCheck %s --check-prefix NUMITERS NUMITERS: error: missing value for '--num-iters' RUN: not %Benchmark_O --num-samples \ RUN: 2>&1 | %FileCheck %s --check-prefix NUMSAMPLES NUMSAMPLES: error: missing value for '--num-samples' RUN: not %Benchmark_O --sleep \ RUN: 2>&1 | %FileCheck %s --check-prefix SLEEP SLEEP: error: missing value for '--sleep' RUN: not %Benchmark_O --delim \ RUN: 2>&1 | %FileCheck %s --check-prefix DELIM DELIM: error: missing value for '--delim' RUN: not %Benchmark_O --tags=bogus \ RUN: 2>&1 | %FileCheck %s --check-prefix BADTAG BADTAG: error: 'bogus' is not a valid 'BenchmarkCategory' RUN: not %Benchmark_O --skip-tags=bogus \ RUN: 2>&1 | %FileCheck %s --check-prefix BADSKIPTAG BADSKIPTAG: error: 'bogus' is not a valid 'BenchmarkCategory'
Measuring memory use of a test with our method is valid only for single test.
RUN: %Benchmark_O 1 2 --memory --list \ RUN: 2>&1 | %FileCheck %s --check-prefix WARNMEMORY WARNMEMORY: warning:
RUN: %Benchmark_O --help | %FileCheck %s --check-prefix OPTIONS OPTIONS: usage: Benchmark_O [--argument=VALUE] [TEST [TEST ...]] OPTIONS: optional arguments: OPTIONS: --help OPTIONS-SAME: show this help message and exit OPTIONS: --verbose OPTIONS: --delim OPTIONS: --tags OPTIONS: --list