syzbot
system continuously fuzzes main Linux kernel branches and automatically reports found bugs to kernel mailing lists. syzbot dashboard shows current statuses of bugs. All syzbot
-reported bugs are also CCed to syzkaller-bugs mailing list. Direct all questions to syzkaller@googlegroups.com
.
syzbot
needs to know when a bug is fixed in order to (1) verify that it is in fact fixed and (2) be able to report other similarly-looking crashes (while a bug is considered open all similarly-looking crashes are merged into the existing bug). To understand when a bug is fixed syzbot
needs to know what commit fixes the bug; once syzbot
knows the commit it will track when the commit reaches all kernel builds on all tracked branches. Only when the commit reaches all builds, the bug is considered closed (new similarly-looking crashes create a new bug).
If you fix a bug reported by syzbot
, please add the provided Reported-by
tag to the commit. You can also communicate with syzbot
by replying to its emails. The commands are:
Reported-by
tag):#syz fix: exact-commit-title
It‘s enough that the commit is merged into any tree or you are reasonably sure about its final title, in particular, you don’t need to wait for the commit to be merged into upstream tree. syzbot
only needs to know the title by which it will appear in tested trees. In case of an error or a title change, you can override the commit simply by sending another #syz fix
command.
#syz unfix
syzbot
bug:#syz dup: exact-subject-of-another-report
#syz undup
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug report.
Note: all commands must start from beginning of the line.
Note: please keep at least syzkaller-bugs@googlegroups.com
mailing list in CC. It serves as a history of what happened with each bug report. Keepint the main kernel mailing list (e.g. linux-kernel@vger.kernel.org
) in CC is useful as well so that it's searchable in those archives as well.
Note: syzbot
identifies bugs by the HASH
in the syzbot+HASH@
receiver email address. So, strictly saying, you don't need to reply to emails (e.g. if you did not receive them), you can send a new email to the syzbot+HASH@
email address, which you can find as Sender
in email archives or as Reported-by
email on the dashboard page for each bug.
syzbot
can test patches for bugs with reproducers. This can be used for testing of fix patches, or just for debugging (i.e. adding additional checks to code and testing with them), or to check if the bug still happens. To test on a particular git tree and branch reply with:
#syz test: git://repo/address.git branch
or alternatively, to test on exact commit reply with:
#syz test: git://repo/address.git commit-hash
You can also completely omit these parameters:
#syz test
In this case, syzbot will check out the latest commit from the branch where the issue was detected.
If you also provide a patch with the email, syzbot
will apply it on top of the tree before testing. The patch can be provided inline in email text or as a text attachment (which is more reliable if your email client messes with whitespaces).
If you want to include the patch directly in the email body, just paste the diff somewhere under the #syz test
command line, e.g.
#syz test: git://repo/address.git branch --- a/mm/kasan/kasan.c +++ b/mm/kasan/kasan.c - current->kasan_depth++; + current->kasan_depth--;
Here are some real examples of #syz test
commands for syzbot-reported bugs.
If you don't provide a patch, syzbot
will test the tree as is. This is useful if this is your own tree which already contains the patch, or to check if the bug is already fixed by some recent commit.
After sending an email you should typically get a reply email with results within an hour.
Note: you may send the request only to syzbot
email address, as patches sent to some mailing lists (e.g. netdev, netfilter-devel) will trigger patchwork.
Note: when testing a patch, syzbot uses the newest reproducer and the matching kernel config that are listed on the dashboard for this bug. As a result, specifying a repo, branch or commit id that are different from the ones that were used for reproducing, can result in false-positive Tested-by responses. For example this happens, when the bug is not reproducible on a specified kernel tree, with or without the supplied patch.
Note: see below for KMSAN
bugs testing.
Note: see below for USB
bugs testing.
For all its bugs, syzbot
automatically assigns kernel subsystems tags. For Linux, the predefined list of kernel subsystems can be found at https://syzkaller.appspot.com/upstream/subsystems.
By clicking on a name in the subsystem list or by following a tag after a bug's title, you can get the full list of bugs belonging to the subsystem. For example, all nfc
bugs are listed here: https://syzkaller.appspot.com/upstream/s/nfc.
syzbot
includes subsystem tags into email subject as well, with ?
indicating that it's an automatic guess: [syzbot] [ntfs?] kernel BUG in ntfs_iget
.
Over time, as we improve the classification rules or as syzbot obtains more information about the bug (e.g. finds a reproducer), syzbot
will update tags.
You can also manually override the automatic guess by replying to the syzbot
email:
#syz set subsystems: net, mm
Names of subsystems must be taken from the subsystem list page on the syzbot web dashboard.
It is possible to assign labels to syzkaller-reported bugs. These labels are displayed near bug titles on the bug lists and on individual bug pages.
There are two types of labels:
#syz set <label>
to set the flag.#syz unset <label>
to drop the flag.#syz set <label>: <value>[, <value2> ...]
to overwrite the value list.#syz unset <label>
would drop the label with all its values.There‘s a fixed list of supported labels (*). Most important ones: | Key | Description | Values | Example | | - | - | - | - | | subsystems
| A comma-separated list of subsystems of the bug | See the list on the syzbot dashboard | #syz set subsystems: mm, reiserfs
| | prio
| The priority of the bug | One of low
, normal
, high
| #syz set prio: low
#syz unset prio
| | no-reminders
| Don’t include the bug into monthly reminders | No values, just a flag | #syz set no-reminders
#syz unset no-reminders
|
It is also possible to set and unset labels for individual bugs from a monthly subsystem report. Let's consider an example.
Some of the still happening issues: Ref Crashes Repro Title <1> 10 No WARNING in __brelse (2) https://syzkaller.appspot.com/bug?id=cd9cb7a620dcfbf46a5eaf201031acaf3aeda28e <2> 3 No WARNING in kernfs_get (4) https://syzkaller.appspot.com/bug?id=b4278401038872458a20f08210e46f3ab519b786
One can send the following email to disable reminders for the first bug and to change the subsystem of the second bug:
#syz set <1> no-reminders #syz set <2> subsystems: kernfs
(*) If you need other labels to facilitate your work, feel free to contact us.
There are several additional aspects if the tree is rebuilt/rebased or contains amended/folded patches (namely, linux-next
).
First, adding Reported-by
tags to amended commits may be misleading. A Reported-by
tag suggests that the commit fixes a bug in some previous commit, but here it‘s not the case (it only fixes a bug in a previous version of itself which is not in the tree). In such case it’s recommended to include a Tested-by
or a Reviewed-by
tag with the hash instead. Technically, syzbot
accepts any tag, so With-inputs-from
would work too.
Then, if the guilty commit is completely dropped (actually removed from the tree during rebuild), then there is effectively no fixing commit. There is no good way to handle such cases at the moment. One possibility is to mark them with #syz invalid
. However this needs to be done only when syzbot
picks up the new tree in all builds (current kernel commits can be seen on dashboard). Otherwise, the occurrences of the crash that are still happening in the current build will immediately create a new bug report. Another possibility is to mark it as fixed with some unrelated later commit using #syz fix: some-later-commit
. This way syzbot
will wait until the commit propagates to all tested trees automatically.
In any case the relation between the report and the fix can later be fixed up using #syz fix: commit-title
commands.
syzbot
bisects bugs with reproducers to find commit that introduced the bug. syzbot
starts with the commit on which the bug was discovered, ensures that it can reproduce the bug and then goes back release-by-release to find the first release where kernel does not crash. Once such release is found, syzbot
starts bisection on that range. syzbot
has limitation of how far back in time it can go (currently v4.19
), going back in time is very hard because of incompatible compiler/linker/asm/perl/make/libc/etc, kernel build/boot breakages and large amounts of bugs.
The predicate for bisection is binary (crash/doesn't crash), syzbot
does not look at the exact crash and does not try to differentiate them. This is intentional because lots of bugs can manifest in different ways (sometimes 50+ different ways). For each revision syzbot
repeats testing 10 times and a single crash marks revision as bad (lots of bugs are due to races and are hard to trigger).
During bisection syzbot
uses different compilers depending on kernel revision (a single compiler can't build all revisions). These compilers are available here. Exact compiler used to test a particular revision is specified in the bisection log.
Bisection is best-effort and may not find the right commit for multiple reasons, including:
A single incorrect decision during bisection leads to an incorrect result, so please treat the results with understanding. You may consult the provided bisection log
to see how/why syzbot
has arrived to a particular commit. Suggestions and patches that improve bisection quality for common cases are welcome.
syzbot
supports cause bisection (find the commit that introduces a bug) and fix bisection (find the commit that fixes a bug).
The web UI for a specific kernel (say upstream linux) shows the Bisected
status for all bugs.
syzbot
will perform fix bisection on bugs that meet the following criterion:
If you receive an email with fix bisection results you think is correct, please reply with a #syz fix: commit-title
so that syzbot can close the bug report.
syzbot
aims at providing stand-alone C reproducers for all reported bugs. However, sometimes it can't extract a reproducer at all, or can only extract a syzkaller reproducer. syzkaller reproducers are programs in a special syzkaller notation and they can be executed on the target system with a little bit more effort. See this for instructions.
A syskaller program can also give you an idea as to what syscalls with what arguments were executed (note that some calls can actually be executed in parallel).
A syzkaller program can be converted to an almost equivalent C source using syz-prog2c
utility. syz-prog2c
has lots of flags in common with syz-execprog, e.g. -threaded
which controls if the syscalls are executed sequentially or in parallel. An example invocation:
syz-prog2c -prog repro.syz.txt -enable=all -threaded -repeat -procs=8 -sandbox=namespace -segv -tmpdir -waitrepeat
However, note that if syzbot
did not provide a C reproducer, it wasn't able to trigger the bug using the C program (though, it can be just because the bug is triggered by a subtle race condition).
Syzbot shares links to the bootable disk image, kernel object file and other relevant files that can be used to reproduce and debug the issue locally.
See this tutorial on how to use them.
If the provided reproducer does not work for you, most likely it is related to the fact that you have slightly different setup than syzbot
. syzbot
has obtained the provided crash report on the provided reproducer on a freshly-booted machine, so the reproducer worked for it somehow.
Note: if the report contains userspace arch: i386
, then the program needs to be built with -m32
flag.
syzbot
uses GCE VMs for testing, but usually it is not important.
If the reproducer exits quickly, try to run it several times, or in a loop. There can be some races involved.
Latest compiler used by syzbot is contained in gcr.io/syzkaller/syzbot:gcc-10.2.1
docker image. For in-tree kernel build in current directory it can be used as follows:
docker pull gcr.io/syzkaller/syzbot:gcc-10.2.1 docker run -it --user $(id -u ${USER}):$(id -g ${USER}) \ --volume "$PWD:/syzkaller/pwd" --workdir /syzkaller/pwd \ gcr.io/syzkaller/syzbot:gcc-10.2.1 make
For an out-of-tree build one needs to proxy more host directories using --volume
flag.
Older compilers used by syzbot
can be found here:
A QEMU-suitable Debian Stretch image can be found here (241 MB). A reference qemu
command line to run it is as follows:
qemu-system-x86_64 -smp 2 -m 4G -enable-kvm -cpu host \ -net nic -net user,hostfwd=tcp::10022-:22 \ -kernel arch/x86/boot/bzImage -nographic \ -device virtio-scsi-pci,id=scsi \ -device scsi-hd,bus=scsi.0,drive=d0 \ -drive file=stretch-amd64.img,format=raw,if=none,id=d0 \ -append "root=/dev/sda1 console=ttyS0 earlyprintk=serial"
And then you can ssh into it using:
ssh -p 10022 root@localhost
Note: before March 25th 2020 Debian Wheezy image was used for testing, so some of the bugs reported before that date might only be reproducible on Wheezy. That image is here and the key for it is here.
Reproducers are best-effort. syzbot
always tries to create reproducers, and once it has one it adds it to the bug. If there is no reproducer referenced in a bug, a reproducer does not exist. There are multiple reasons why syzbot
can fail to create a reproducer: some crashes are caused by subtle races and are very hard to reproduce in general; some crashes are caused by global accumulated state in kernel (e.g. lockdep reports); some crashes are caused by non-reproducible coincidences (e.g. an integer 0x12345
happened to reference an existing IPC object) and there is long tail of other reasons.
Bugs with reproducers are automatically reported to kernel mailing lists. Bugs without reproducers are first staged in moderation queue to filter out invalid, unactionable or duplicate reports. Staged bugs are shown on dashboard in moderation section and mailed to syzkaller-upstream-moderation mailing list. Staged bugs accept all commands supported for reported bugs (fix
, dup
, invalid
) with a restriction that bugs reported upstream can't be dup
-ed onto bugs in moderation queue. Additionally, staged bugs accept upstream command:
#syz upstream
which sends the bug to kernel mailing lists.
KMSAN
is a dynamic, compiler-based tool (similar to KASAN
) that detects uses of uninitialized values. As compared to (now deleted) KMEMCHECK
which simply detected loads of non-stored-to memory, KMSAN
tracks precise propagation of uninitialized values through memory and registers and only flags actual eventual uses of uninitialized values. For example, KMSAN
will detect a branch on or a copy_to_user()
of values that transitively come from uninitialized memory created by heap/stack allocations. This ensures /theoretical/ absence of both false positives and false negatives (with some implementation limitations of course). Note that KMSAN
requires clang
compiler.
KMSAN
is not upstream yet, though, we want to upstream it later. For now, it lives in github.com/google/kmsan and is based on a reasonably fresh upstream tree. As the result, any patch testing requests for KMSAN
bugs need to go to KMSAN
tree (https://github.com/google/kmsan.git
repo, master
branch). A standard way for triggering the test with KMSAN
tree is to send an email to syzbot+HASH
address containing the following line:
#syz test: https://github.com/google/kmsan.git master
and attach/inline your test patch in the same email.
Report explanation. The first call trace points to the use
of the uninit value (which is usually a branching or copying it to userspace). Then there are 0 or more “Uninit was stored to memory at:” stacks which denote how the unint value traveled through memory. Finally there is a “Uninit was created at:” section which points either to a heap allocation or a stack variable which is the original source of uninitialized-ness.
syzkaller has an ability to perform fuzzing of the Linux kernel USB stack, see the details here. As of now all kernel changes required for USB fuzzing have been merged into the mainline (the last one during the 5.8-rc1 merge window), so the USB fuzzing instance has been switched to target the usb-testing tree.
Testing kernel patches on the USB instance follows the same principle as on the mainline instances, with a few caveats:
You may specify any kernel tree for syz test
as long as it includes all mainline patches up to 5.8-rc1.
Some of the bugs have reproducers generated on kernel versions with custom kernel (when fuzzing was performed with non-yet-mainlined kernel patches), thus those reproducers might no longer work. The recommended workflow is to: first, execute a syz test
command on a target tree to make sure that the bug reproduces, and then execute a syz test
command with a fix/debug patch.
If the bug was triggered on the KMSAN
tree, follow the instructions above, with the exception that you must also use commit-hash
instead of the master
branch when testing patches.
syzbot
uses KMEMLEAK
to find memory leaks in the Linux kernel. KMEMLEAK
kernel config is stored here. See KMEMLEAK
docs for general info, algorithm overview and usage instructions.
Memory leaks may be tricky to debug because we have only the allocation stack, but we don‘t see where kernel code forgot to free the object or drop a reference. KMEMLEAK
can have some false positives on tricky kernel code that hides pointers to live objects and due to memory scanning non-atomicity. But don’t write off everything to false positives right away, the rate of false positives is observed to be very low. In particular, KMEMLEAK
is not confused by pointers stored in a middle of another object; and it's not confused if several pointer low bits are used as flags because a pointer into a middle of an object also marks the target as reachable.
A useful litmus test is to remove KMEMLEAK
code from the reproducer and run it for longer and/or multiple times. If memory consumption and number of live objects in /proc/slabinfo
steadily grow, most likely the leak is real.
The Kernel Concurrency Sanitizer (KCSAN) is a dynamic data-race detector. Reproduction of data-races is currently unsupported, and syzbot is unable to test patches.
While syzbot
can test patches that fix bugs, it does not support applying custom patches during fuzzing. It always tests vanilla unmodified git trees. There are several reasons for this:
We've experimented with application of custom patches in the past and it lead to unrecoverable mess. If you want syzbot
to pick up patches sooner, ask tree maintainers for priority handling.
However, syzbot kernel config always includes CONFIG_DEBUG_AID_FOR_SYZBOT=y
setting, which is not normally present in kernel. What was used for particularly elusive bugs in the past is temporary merging some additional debugging code into linux-next
under this config setting (e.g. more debug checks and/or debug output) and waiting for new crash reports from syzbot.
One can also always run syzkaller locally on any kernel for better stress testing of a particular subsystem and/or patch.
Kernel configs, sysctls and command line arguments that syzbot
uses are available in /dashboard/config.
Yes, it is here.