Skip to content
#

parallel-lexing

Here is 1 public repository matching this topic...

Runtime lexers for real languages, and exact decisions about where a scan may be cut: certified boundaries, exact parallel tokenization for BPE, WordPiece and Unigram, and an auditor for flex, re2c, ANTLR and logos scanners.

  • Updated Nov 22, 2026
  • C++

Add this topic to your repo

To associate your repository with the parallel-lexing topic, visit your repo's landing page and select "manage topics."

Learn more