When training SuperBPE on Sinhala data, both Stage 1 (BPE) and Stage 2 (SuperBPE) produce garbage characters in the vocabulary and tokenizer output.
Tokens look like:
instead of valid Sinhala characters.
The issue is present in:
tokenizer.jsonvocab.json- Tokenization output during inference
Training procedure
I followed the repository instructions exactly:
Stage 1: Learn subwords with whitespace pretokenization (regular BPE)
scripts/train_tokenizer.sh
Stage 2: Resume training without whitespace pretokenization (SuperBPE)
scripts/extend_tokenizer.sh
No modifications were made to the scripts.
Dataset
- Dataset: Helsinki-NLP/opus-100 (en-si)
- Used Sinhala side (
si)
- Dataset is UTF-8
- Same issue appears in both Stage 1 and Stage 2 outputs
Observation
This does not appear to be a decoding issue at inference time, because the garbage characters are already present inside:
tokenizer.jsonvocab.json
This suggests the corruption happens during training.
Example
Original:
ඒ කාරණාවෙන් මාව අර්ථවත් උනා , ඒකෙන් මට ශක්තිය ලැබුනා∙ .
BPE tokenized:
['à¶Ĵ', 'Ġà¶ļ', 'à·ı', 'රණ', 'à·ı', 'à·Ģ', 'à·Ļ', 'න', 'à·Ĭ', 'Ġම', 'à·ı', 'à·Ģ', 'Ġà¶ħර', 'à·Ĭ', 'ථ', 'à·Ģà¶Ń', 'à·Ĭ', 'Ġà¶ĭන', 'à·ı', 'Ġ,', 'Ġà¶Ĵà¶ļ', 'à·Ļ', 'න', 'à·Ĭ', 'Ġමට', 'Ġà·ģ', 'à¶ļ', 'à·Ĭ', 'à¶Ń', 'à·Ĵ', 'ය', 'Ġල', 'à·IJ', 'බ', 'à·Ķ', 'න', 'à·ı', 'âĪĻ', 'Ġ.']
SuperBPE tokenized:
['à¶Ĵ', 'Ġà¶ļ', 'à·ı', 'රණ', 'à·ı', 'à·Ģ', 'à·Ļ', 'න', 'à·Ĭ', 'Ġම', 'à·ı', 'à·Ģ', 'Ġà¶ħර', 'à·Ĭ', 'ථ', 'à·Ģà¶Ń', 'à·Ĭ', 'Ġà¶ĭන', 'à·ı', 'Ġ,', 'Ġà¶Ĵà¶ļ', 'à·Ļ', 'න', 'à·Ĭ', 'Ġමට', 'Ġà·ģ', 'à¶ļ', 'à·Ĭ', 'à¶Ń', 'à·Ĵ', 'ය', 'Ġල', 'à·IJ', 'බ', 'à·Ķ', 'න', 'à·ı', 'âĪĻ', 'Ġ.']
Both stages produce identical corrupted tokens.
Expected behavior
Vocabulary and tokens should contain valid Sinhala Unicode characters.
Environment
- Python 3.11
- Windows 11 / WSL Ubuntu
When training SuperBPE on Sinhala data, both Stage 1 (BPE) and Stage 2 (SuperBPE) produce garbage characters in the vocabulary and tokenizer output.
Tokens look like:
instead of valid Sinhala characters.
The issue is present in:
tokenizer.jsonvocab.jsonTraining procedure
I followed the repository instructions exactly:
Stage 1: Learn subwords with whitespace pretokenization (regular BPE)
scripts/train_tokenizer.shStage 2: Resume training without whitespace pretokenization (SuperBPE)
scripts/extend_tokenizer.shNo modifications were made to the scripts.
Dataset
si)Observation
This does not appear to be a decoding issue at inference time, because the garbage characters are already present inside:
tokenizer.jsonvocab.jsonThis suggests the corruption happens during training.
Example
Original:
BPE tokenized:
SuperBPE tokenized:
Both stages produce identical corrupted tokens.
Expected behavior
Vocabulary and tokens should contain valid Sinhala Unicode characters.
Environment