syz-testbed
is the tool that simplifies the process of evaluating the performance of different syzkaller versions (or configurations) against each other. The tool automates checking out syzkaller repos, building them, running syz-manager
s and collecting/summarizing their results.
syz-testbed
requires a JSON config file. An example of such a file:
{ "workdir": "/tmp/syz-testbed-workdir/", "corpus": "/tmp/corpus.db", "target": "syz-manager", "max_instances": 5, "run_time": "24h", "http": "0.0.0.0:50000", "checkouts": [ { "name": "first", "repo": "https://github.com/google/syzkaller.git", }, { "name": "second", "repo": "https://github.com/google/syzkaller.git", "branch": "some-dev-branch", } ], "manager_config": { "target": "linux/amd64", "kernel_obj": "/tmp/linux-stable", "image": "/tmp/kernel-image/bullseye.img", "sshkey": "/tmp/kernel-image/bullseye.id_rsa", "procs": 8, "type": "qemu", "vm": { "count": 2, "kernel": "/tmp/linux-stable/arch/x86/boot/bzImage", "cpu": 2, "mem": 2048 } } }
Given such a configuration file, syz-testbed
will do the following:
master
branch of https://github.com/google/syzkaller.git
into /tmp/syz-testbed-workdir/checkouts/first/
and build it.some-dev-branch
of https://github.com/google/syzkaller.git
into /tmp/syz-testbed-workdir/checkouts/second/
and build it.first
and 2 instances of second
(max_instances = 5
).The directory structure looks as follows:
/tmp/syz-testbed-workdir/ └── checkouts ├── first │ ├── run-first-0 │ │ ├── log.txt │ │ ├── manager.cfg │ │ └── workdir │ ├── run-first-1 │ │ ├── log.txt │ │ ├── manager.cfg │ │ └── workdir │ └── run-first-4 │ │ ├── log.txt │ │ ├── manager.cfg │ │ └── workdir └── second ├── run-second-2 │ ├── log.txt │ ├── manager.cfg │ └── workdir └── run-second-3 ├── log.txt ├── manager.cfg └── workdir
run_hours
is 24), stop those 5 instances.first
and 3 instances of second
.The tool stops after receiving a SIGINT (e.g. after Ctrl+C) or a SIGTERM signal. Also, if one of the instances has exited due to some error, this also stops the whole experiment.
The tool has a simple web interface that displays the current information about the experiment (the number of active and finished instances, the time until instances are stopped, etc.) and the latest statistics collected from the syz-manager
s.
If the benchmp
parameter points to the syz-benchcmp
executable, then the web interface can also generate graphs of various parameters over time or the number of executions.
In order to enable the interface, set the http
parameter to the IP address and port to which syz-testbed
should bind. E.g. "http": "0.0.0.0:50000"
.
syz-testbed
provides two “views” of the statistics:
complete
- only includes data from the finished instances (i.e. those that have been running for run_hours
).all
- also includes the data from the currently active instances. The statistics from the finished instances is winded back to match the current uptime of the active instances.Therefore, the statistics is laid out the following way.
$ tree -L 2 /tmp/syz-testbed-workdir/ /tmp/syz-testbed-workdir/ ├── stats_all │ ├── benches │ │ ├── avg_first.txt │ │ ├── avg_second.txt │ ├── bugs.csv │ ├── checkout_stats.csv │ └── instance_stats.csv ├── stats_completed │ ├── benches │ │ ├── avg_first.txt │ │ ├── avg_second.txt │ ├── bugs.csv │ ├── checkout_stats.csv │ └── instance_stats.csv └── testbed.csv
bugs.csv
contains all the bugs found by the running instances. If a single checkout has several instances (i.e. count
> 1), syz-testbed
takes a union of bugs found by them. The purpose is ultimately to collect all bugs that could be found by that version of syzkaller.syz-manager
s is saved into instance_stats.csv
. The same data is also averaged among instances that belong to the same checkouts and saved into the checkout_stats.csv
file.tools/syz-benchcmp
) of all syz-manager
s belonging to a single checkout are averaged and saved into the corresponding files in the benches
folder.The statics is updated once every 90 seconds.
First, checkout the most recent version of syzkaller itself:
$ git clone https://github.com/google/syzkaller.git
Then, build syz-testbed
:
$ cd syzkaller/tools/syz-testbed/ $ go build
Write and save the configuration file (e.g. into the config.json
file). Then, syz-testbed
can be run using the following command:
$ ./syz-testbed -config config.json
Stopping the syz-testbed
process results in stopping all the syzkaller instances.
syz-testbed
can also be used to test syzkaller's ability to reproduce bugs. To do so, set the target
property in the syz-testbed
's config file to syz-repro
.
One can also specify the source of the crash log files. This is either just a folder, whose files will be treated accordingly or it can be a syzkaller's workdir. input_logs
must point to the folde with crash logs - syz-testbed
will traverse it and treat each file as an input. Otherwise, input_workdir
must be used.
For example:
"repro_config": { "input_workdir": "/tmp/some-syzkaller-workdir", "crashes_per_bug": 2, "skip_bugs": ["SYZFAIL", "no output", "corrupted", "lost connection"] },
In this case, syz-testbed
will traverse all bugs found by the syzkaller, skip those that match “SYZFAIL”, “no output”, “corrupted” or “lost connection”, then pick 2 random crash logs for each such bug for later processing.
syz-testbed
will check out and compile the specified syzkaller instances and will go on executing their syz-repro
s on each picked up crash log file, as long as the tool is not stopped.