Merge #10995
10995: internal: switch from trait-based TokenSource to simple struct of arrays r=matklad a=matklad
cc #10765
The idea here is to try to simplify the interface as best as we can. The original trait-based approach is a bit over-engineered and hard to debug. Here, we replace callback with just data. The next PR in series will replace the output `TreeSink` trait with data as well.
The biggest drawback here is that we now require to materialize all parser's input up-front. This is a bad fit for macro by example: when you parse `$e:expr`, you might consume only part of the input. However, today's trait-based solution doesn't really help -- we were already materializing the whole thing! So, let's keep it simple!
Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
