You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Runtime lexers for real languages, and exact decisions about where a scan may be cut: certified boundaries, exact parallel tokenization for BPE, WordPiece and Unigram, and an auditor for flex, re2c, ANTLR and logos scanners.
Exact parallel and streaming tokenization for LLM tokenizers: decides from a tokenizer's definition where its input can be cut, then encodes in parallel or as a stream with token ids byte-identical to sequential encoding. GPT-2, cl100k, o200k and Llama 3, in C++23.