tree e4dcc1edf9f2b01cc1ca911a6d3e5a793bb928b4
parent 4a94a4b2dc7ec0119164ed69649bc96bc2b1025d
author Bill Neubauer <wcn@google.com> 1642011337 -0800
committer Bill Neubauer <wcn@google.com> 1647470242 -0700

Fixes handling of newline characters so that Normalize preserves the newline
characters of the original input. Includes fixes for OOB array conditions
detected by fuzzing and the crash logs from the service.

The code doing the tokenization of the newlines had some minor bugs that
resulted in spurious newlines being introduced into the token stream. This
wasn't a problem before since they were only used inside the tokenizer to
detect header constructs and de-hyphenate words, and were always removed from
the token stream passed to calling functions. This meant that the token stream
Normalize was reassembling had no newlines so it just produced a big line of
text. Fixing Normalize to preserve the original newlines required fixing these
glitches.

Added more test cases to cover the different newline-related scenarios and new
scenario files based on fuzzer findings.
***

PiperOrigin-RevId: 421331357