#
o200k
Here are 7 public repositories matching this topic...
Byte pair encoding tokenizer for OpenAI models that encodes text into token IDs, decodes them back, and counts tokens.
nlp encoding machine-learning natural-language-processing tokenizer openai text-processing gpt language-model tokenization text-encoding bpe byte-pair-encoding ai-tools subword-tokenization tiktoken token-counting cl100k o200k text-to-tokens
-
Updated
Sep 28, 2026 - TypeScript
Conversation compression gradients: Zipilogue / TesseraTalk / ShorthandLab / ConvoZip / MorphoZip
multilingual python nlp compression tokenizer data-compression text-compression llm cost-reduction tiktoken token-reduction context-window prompt-optimization llm-optimization prompt-compression context-compression token-optimization token-efficiency conversation-compression o200k
-
Updated
Nov 22, 2026 - Python
An attempt at making a tokenizer!
tokenizer gpt mlx huggingface huggingface-transformer huggingface-transformers tinystories token-optimization token-efficiency o200k-base o200k
-
Updated
Oct 12, 2026 - Python
Measured chars/4 against cl100k on 10,000 prompts: 40.5% of rows off by more than 20%, and the error flips sign by content type.
-
Updated
Nov 3, 2026 - Python
Add this topic to your repo
To associate your repository with the o200k topic, visit your repo's landing page and select "manage topics."