Search results

Proper Method to Represent Special Characters?

Status: Open.

#41 In alasdairforsythe/tokenmonster;

· xTimeCrystal opened on Oct 26, 2025

Expose spans for mapping tokens to input characters

Status: Open.

#39 In alasdairforsythe/tokenmonster;

· iiLaurens opened on Jan 28, 2025

Preferred citation bibtex

Status: Open.

#35 In alasdairforsythe/tokenmonster;

· konstantinjdobler opened on May 8, 2024

Hangs with PyTorch data loaders when num_workers > 0

Status: Open.

#34 In alasdairforsythe/tokenmonster;

· ntoxeg opened on Mar 22, 2024

Update on multilingual

Status: Open.

#32 In alasdairforsythe/tokenmonster;

· kerighan opened on Feb 27, 2024

Inquiry on Extending Algorithm to Other Languages

Status: Open.

#30 In alasdairforsythe/tokenmonster;

· dsdanielpark opened on Dec 12, 2023

Special tokens not showing up correctly when tokenized.

Status: Open.

#29 In alasdairforsythe/tokenmonster;

· amazingvince opened on Nov 5, 2023

HUggingface tokenizer coming soon?

Status: Open.

#4 In alasdairforsythe/tokenmonster;

· kyegomez opened on Jun 4, 2023