fix: support strings with NUL bytes (#15)
* fix: support strings with NUL bytes
Lean strings can contain NUL (U+0000): the literal "a\x00b" is a
three-character string, and the .olean stores its bytes followed by the
usual NUL terminator. The STRING opcode writes a string's bytes up to
and including the first NUL, so it cannot represent such a string, and
the compressor asserted that the first NUL was the terminator. Packing
any module with such a literal panicked at that assertion, which makes
`lake build -o` fail for the module.
A new opcode SIZED_STRING (0xb0, free in normal mode since pack_ctor
never yields a zero low nibble) is followed by the byte length, without
the terminator, and then the bytes. The compressor emits it only for
strings with a NUL before the terminator and keeps STRING for all
others, so archives of other modules are unchanged. The decompressor
counts the characters for the length field as it does for STRING.
A leantar without this change cannot unpack an archive containing
SIZED_STRING. Master panics with "unknown opcode 0xb0". Releases up to
0.1.21 decode 0xb0 as a constructor without fields and lose their place
in the stream: for an archive of a module with the literal above, 0.1.20
and 0.1.21 panic further on, and in principle such a misreading could
also end in a wrong .olean. No older leantar could have written such an
archive.
The approach follows #9 by ericrbg-harmonic, which used 0xa0, since
taken by SCALAR_ARRAY.
* read SIZED_STRING payload with read_exact like MPZ
---------
Co-authored-by: Matthias Görgens <matthias@metisr.ai>
Co-authored-by: Mario Carneiro <di.gama@gmail.com>
2 files changed