perf: avoid quadratic byte-pair merging for long pieces

[rksharma-owg]

Summary

Fixes #151 by replacing the repeated full scan and vector removal in _byte_pair_merge with a priority queue over a linked list of active parts. Equal-rank merges retain the existing left-to-right tie-breaking behavior.

Validation

• Added a merge-priority regression test.
• Added a long-piece performance regression test; the pre-fix implementation took 3.74s for a 32 KiB repeated piece and the patched implementation completes in milliseconds.
• cargo test --all-targets --all-features (32 passed)
• git diff --check

cargo clippy --all-targets --all-features -- -D warnings remains blocked by two pre-existing warnings in src/encoding.rs and src/tiktoken.rs; pytest -q could not run because pytest is not installed in the environment.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-08T03:51:58.941114Z	62ef0b4	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.