A real-time tokenization visualizer powered by a completely custom Byte-Pair Encoding (BPE) Tokenizer built from scratch in Python!
This project is a full-stack web application designed to visualize how Large Language Models (like GPT-2) break down raw text into subword tokens.
Under the hood, the tokenization is performed by a custom Python BPE Tokenizer built from the ground up (no third-party tokenizer libraries used). It implements the exact BPE training and encoding algorithms used by GPT-2, including handling overlapping special tokens, UTF-8 byte chunking, and iterative BPE rank merging.
Details:
- Vocabulary Size: 10,000
- Dataset: Trained on the TinyStories dataset
The web app consists of a sleek Next.js React frontend that communicates with the Python FastAPI backend in real-time to visually color-code the tokens as you type.
- Custom BPE Implementation: Pure Python implementation of GPT-2's BPE encoding.
- Real-Time Visualization: As you type, the frontend sends debounced requests to the FastAPI backend and instantly color-codes the returning token subwords.
- Serverless Architecture: Built to run entirely on Vercel's serverless functions seamlessly integrating the React frontend and Python API.
Below are some screenshots showing the underlying tokenizer training process on the TinyStories dataset and the rigorous test suite verifying exact matches with OpenAI's tiktoken!
