Commit graph

1 commit

Author SHA1 Message Date
Eyal Leshem
5458a2b21d Implement zero-copy tokenization for Word, SingleQuotedString, and Whitespace
Convert token string fields to use Cow<'a, str> to enable zero-copy tokenization
  for commonly used tokens:
  - Word.value: Regular identifiers and keywords now borrow from source
  - SingleQuotedString: String literals borrow when no escape processing needed
  - Whitespace: Single-line and multi-line comments borrow from source

Also add benchmark for measuring tokenization performance
2025-12-21 16:39:10 +02:00