Skip to content
#

certified-splitting

Here are 2 public repositories matching this topic...

Runtime lexers for real languages, and exact decisions about where a scan may be cut: certified boundaries, exact parallel tokenization for BPE, WordPiece and Unigram, and an auditor for flex, re2c, ANTLR and logos scanners.

  • Updated Nov 22, 2026
  • C++

Exact parallel and streaming tokenization for LLM tokenizers: decides from a tokenizer's definition where its input can be cut, then encodes in parallel or as a stream with token ids byte-identical to sequential encoding. GPT-2, cl100k, o200k and Llama 3, in C++23.

  • Updated Nov 22, 2026
  • C++

Add this topic to your repo

To associate your repository with the certified-splitting topic, visit your repo's landing page and select "manage topics."

Learn more