As a part of every bug report, syzbot shares downloadable assets -- that is, disk images and kernel binaries on which the bug was originally found.
This document serves as a guide on how to use those assets to reproce such bugs locally.
To be more specific, let's take this syzbot report: [syzbot] [hfs?] kernel BUG in hfsplus_bnode_put.
syzbot has found a reproducer for the following issue on: HEAD commit: 40f71e7cd3c6 Merge tag 'net-6.4-rc7' of git://git.kernel.o.. git tree: upstream console+strace: https://syzkaller.appspot.com/x/log.txt?x=10482ae3280000 kernel config: https://syzkaller.appspot.com/x/.config?x=7ff8f87c7ab0e04e dashboard link: https://syzkaller.appspot.com/bug?extid=005d2a9ecd9fbf525f6a compiler: Debian clang version 15.0.7, GNU ld (GNU Binutils for Debian) 2.35.2 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=142e7287280000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=13fd185b280000 Downloadable assets: disk image: https://storage.googleapis.com/syzbot-assets/073eea957569/disk-40f71e7c.raw.xz vmlinux: https://storage.googleapis.com/syzbot-assets/c8a97aaa4cdc/vmlinux-40f71e7c.xz kernel image: https://storage.googleapis.com/syzbot-assets/f536015eacbd/bzImage-40f71e7c.xz mounted in repro: https://storage.googleapis.com/syzbot-assets/b5f1764cd64d/mount_0.gz
There are 4 linked assets:
https://storage.googleapis.com/syzbot-assets/073eea957569/disk-40f71e7c.raw.xzvmlinux file that can be used e.g. for report symbolization or for gdb-based debugging: https://storage.googleapis.com/syzbot-assets/c8a97aaa4cdc/vmlinux-40f71e7c.xzbzImage file (it is already included in the disk image): https://storage.googleapis.com/syzbot-assets/f536015eacbd/bzImage-40f71e7c.xzhttps://storage.googleapis.com/syzbot-assets/b5f1764cd64d/mount_0.gzAll these links are also reachable from the web dashboard.
Boot a VM:
$ wget 'https://storage.googleapis.com/syzbot-assets/073eea957569/disk-40f71e7c.raw.xz' $ unxz disk-40f71e7c.raw.xz $ qemu-system-x86_64 -m 2G -smp 2,sockets=2,cores=1 -drive file=./disk-40f71e7c.raw,format=raw -net nic,model=e1000 -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -enable-kvm -nographic -snapshot -machine pc-q35-7.1
Build and run the C reproducer:
$ wget -O 'repro.c' 'https://syzkaller.appspot.com/x/repro.c?x=13fd185b280000' $ gcc repro.c -lpthread -static -o repro $ scp -P 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes ./repro root@127.0.0.1:/root/ $ ssh -p 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes root@127.0.0.1 'chmod +x ./repro && ./repro'
Wait a minute and notice a crash report in the qemu's serial output:
[ 91.956238][ T81] ------------[ cut here ]------------ [ 91.957508][ T81] kernel BUG at fs/hfsplus/bnode.c:618! [ 91.958645][ T81] invalid opcode: 0000 [#1] PREEMPT SMP KASAN [ 91.959861][ T81] CPU: 0 PID: 81 Comm: kworker/u5:3 Not tainted 6.4.0-rc6-syzkaller-00195-g40f71e7cd3c6 #0
For some bugs, there‘s either no C reproducer or it’s not reliable enough. In that case, syz reproducers might be useful.
You'll need to check out and build syzkaller first. The fastest way to do it is as follows (assuming Docker is installed and configured on your machine):
$ git clone https://github.com/google/syzkaller.git $ cd syzkaller $ ./tools/syz-env make
Then boot a VM exactly like in the previous section.
Download and run the syz reproducer:
$ wget -O 'repro.syz' 'https://syzkaller.appspot.com/x/repro.syz?x=142e7287280000' $ scp -P 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes ./bin/linux_amd64/* ./repro.syz root@127.0.0.1:/root/ $ ssh -p 10022 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes root@127.0.0.1 './syz-execprog -enable=all -repeat=0 -procs=6 ./repro.syz'
In some time, you‘ll see the same bug report in the VM’s serial output.
The commands above execute the ./syz-execprog -enable=all -repeat=0 -procs=6 ./repro.syz command inside the VM. For more details see this document.
tools/syz-crush toolThe syz-crush automatizes the steps above: it sets up and boots a pool of VMs and runs the given C or syz reproducer in them.
First, download the disk image and reproducers (see instructions above).
Then, go to the syzkaller checkout and build the syz-crush tool:
$ make crush
Prepare a config file (let it be config.json):
{
"name": "test",
"http": "0.0.0.0:0",
"target": "linux/amd64",
"image": "/tmp/disk-40f71e7c.raw",
"syzkaller": "/tmp/syzkaller",
"workdir": "/tmp/syzkaller/workdir",
"type": "qemu",
"procs": 6,
"vm": {
"count": 5,
"cmdline": "root=/dev/sda1",
"cpu": 2,
"mem": 2048,
"qemu_args": "-machine pc-q35-7.1 -enable-kvm"
}
}
You need to replace /tmp/syzkaller with the location of your syzkaller checkout and /tmp/disk-40f71e7c.raw with the location of the bootable disk image.
Run the tool:
$ mkdir workdir $ ./bin/syz-crush -config config.json repro.syz
If the C reproder did not work, try to run the syz reproducer.
If there's still no success, it might be that relatively rare case when the execution environment becomes important. Syzbot fuzzes kernels on GCE VMs, which might have a different instruction set / execution speed than locally run qemu VMs. These changes might be critical for the generated reproducer.
There's unfortunately no universal solution.
Note that you can always ask syzbot to apply your git patch and re-run the reproducer. It will be run in the same GCE environment where the bug was originally found.
See also this document.
The downloadable assets are not stored infinitely. Syzbot keeps them until the bug is fixed or marked as invalid + 30 days after that.
So if you cannot download the assets using the links from the email, this might be a sign that the bug is actually no longer worth looking at.
A recent qemu problem may prevent it from booting large kernel images. Add -machine pc-q35-7.1 to the qemu args to make it work.