link: stream the binary when setting content build IDs (#4764)
**What type of PR is this?**
Other
**What does this PR do? Why is it needed?**
Follow-up to #4728. `setContentBuildID` read the whole linked binary
into
memory, which can be prohibitive for large Go binaries. It now streams:
- The binary is hashed with `io.Copy` into sha256.
- ELF binaries are parsed with `elf.NewFile` over the open file. The
placeholder
check is a `ReadAt` of the build ID note, and a read error is returned
instead
of being treated as "no placeholder".
- Mach-O binaries are parsed with `macho.NewFile` over the open file.
The Go
build ID is found by a chunked search (`indexReader`, 1 MiB buffer that
keeps
the last `len(pattern)-1` bytes between reads) instead of `bytes.Index`
over
the whole file.
Peak memory no longer scales with the binary size. On a ~500 MB padded
ELF, peak
RSS of `setContentBuildID` went from 508 MB to 4.8 MB. The output is
unchanged:
ELF (linux/amd64, linux/arm64) and Mach-O (darwin/arm64) binaries are
byte-identical to those produced before this change.
`indexReader` is covered by a new unit test that places the match around
the
buffer boundary and reads in full and half chunks. The ELF path is
covered by
the existing `//tests/core/go_binary:buildid_test`, and the Mach-O path
by
`buildid_darwin_test`.
**Which issues(s) does this PR fix?**
N/A, follow-up to #4728.
**Other notes for review**
- Mach-O binaries are now read three times: once to hash, once to search
for the
build ID, and once in `machoCodeSign`. Memory use is constant, and the
extra
passes hit the OS file cache. Searching while hashing would make it one
pass,
but it is not worth the added complexity.
- `buildid_test` in `tests/core/go_binary` is Linux-only, so I could
only check
the ELF path locally by comparing the output of the old and new
`setContentBuildID` on the same binaries.
- The two commits are independent: ELF and hashing first, Mach-O search
and its
unit test second.
1 file changed