tree cc8f0c6559142a443e4c95c53bbe2d4d2453b07e
parent 9fc6ed6d865bf3329927b98d0da5591381969260
author Google Open Source <noreply+opensource@google.com> 1639439460 -0800
committer Bill Neubauer <wcn@google.com> 1647470214 -0700

Automated g4 rollback of changelist 415285962.

*** Reason for rollback ***

Crashing services with index out of bound (https://coroner.corp.google.com/#/crashes?user=compliance-presubmit3p)

*** Original change description ***

Fixes handling of newline characters so that Normalize preserves the newline
characters of the original input.

The code doing the tokenization of the newlines had some minor bugs that
resulted in spurious newlines being introduced into the token stream. This
wasn't a problem before since they were only used inside the tokenizer to
detect header constructs and de-hyphenate words, and were always removed from
the token stream passed to calling functions. This meant that the token stream
Normalize...

***

PiperOrigin-RevId: 416154051