BPE Tokenizer Training

Trains a Byte-Pair Encoding tokenizer on the agentlans/high-quality-english-sentences dataset.

Setup

pip install tokenizers datasets