Skip to content

fix: correct Tamil superscript tokenization and subscript indexing (#57) - #59

Merged
vm75 merged 2 commits into
mainfrom
dev
Oct 11, 2026
Merged

vm75 merged 2 commits into
mainfrom
dev

Conversation

@vm75

@vm75 vm75 commented Oct 11, 2026

Copy link
Copy Markdown
Owner

Description

Resolves #57.

  • Fix isolated superscript/subscript token absorption by allowing partial string lookahead consumption in InputReader.
  • Implement matchTamilMarker helper to decode canonical phoneme offsets and marker byte lengths for superscripts and subscripts.
  • Correct subscript UTF-8 indexing from dividing by 2 bytes to exact 3-byte offsets.
  • Add bounded phoneme offset indexing to prevent out-of-bounds mutation on eligible stops and Ja.
  • Ensure subsequent preserved characters (whitespace, punctuation, text) are retained intact.
  • Separate marker prefix matching from full-span marker verification in isTamilMarkerSpan / isTamilEndOfWord to prevent word-boundary regressions in TamilWriterHandler.
  • Add comprehensive component tests in test-handlers.cpp and end-to-end fixtures in test-cases.json.
  • Synchronize and regenerate WASM and JS distribution artifacts.

vm75 added 2 commits October 11, 2026 13:50
- Fix isolated superscript/subscript token absorption by allowing partial string lookahead consumption in InputReader
- Implement matchTamilMarker helper to decode canonical phoneme offsets and marker byte lengths for superscripts and subscripts
- Correct subscript UTF-8 indexing from dividing by 2 bytes to exact 3-byte offsets
- Add bounded phoneme offset indexing to prevent out-of-bounds mutation on eligible stops and Ja
- Ensure subsequent preserved characters (whitespace, punctuation, text) are retained intact
- Add comprehensive component tests in test-handlers.cpp and end-to-end fixtures in test-cases.json
- Rebuild WASM and JS distribution artifacts
@vm75
vm75 merged commit 30b7d9b into main Oct 11, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

defects noticed after tests

1 participant