Runtime lexers for real languages, and exact decisions about where a scan may be cut: certified boundaries, exact parallel tokenization for BPE, WordPiece and Unigram, and an auditor for flex, re2c, ANTLR and logos scanners.
unicode regex tokenizer parallel-computing lexer lexical-analysis nfa dfa error-recovery compiler-frontend tokenization combinators automata-theory dfa-minimization finite-automata bpe cpp23 parallel-lexing lexer-audit certified-splitting
-
Updated
Nov 22, 2026 - C++