Skip to content

Track what is left in SentenceBreaks::size_hint - #182

Open
youdie006 wants to merge 1 commit into
unicode-rs:masterfrom
youdie006:fix-sentence-breaks-size-hint
Open

Track what is left in SentenceBreaks::size_hint#182
youdie006 wants to merge 1 commit into
unicode-rs:masterfrom
youdie006:fix-sentence-breaks-size-hint

Conversation

@youdie006

Copy link
Copy Markdown

The inconsistency

SentenceBreaks::size_hint measures self.string.len(), the whole original
input, but next never shortens string - it only advances pos
(src/sentence.rs:193). So the hint never moves.

Your own Graphemes::size_hint at src/grapheme.rs:109 already measures the
span that is actually left:

let slen = self.cursor_back.cur_cursor() - self.cursor.cur_cursor();

UWordBounds::size_hint at src/word.rs:219 does write self.string.len(),
but there it is correct, because next reslices with
self.string = &self.string[idx..] (src/word.rs:447) and next_back with
self.string = &self.string[..idx] (src/word.rs:705). SentenceBreaks is the
only one of the three that walks with an index and then measures the original.

(Line numbers from 048d51f.)

What a caller sees

let mut it = "ab".split_sentence_bounds();
it.next();                 // Some("ab")
it.next();                 // None - exhausted
assert_eq!(it.size_hint(), (1, Some(2)));   // promises >= 1 more item

The lower bound stays at 1 with nothing left. On a longer input it is frozen for
the whole walk:

"Hi. There."  fresh:          hint = (1, Some(10))   left = 2
              after 1 next(): hint = (1, Some(10))   left = 1
              after 2 next(): hint = (1, Some(10))   left = 0

USentenceBounds::size_hint (src/sentence.rs:377) and
USentenceBoundIndices::size_hint (:414) both inherit this. unicode_sentences()
does not: it is a Filter, and std's Filter::size_hint zeroes the lower bound,
so it is unaffected.

The change

Measure from pos. The pos == 0 branch keeps today's bound for a fresh
iterator, where the start-of-text break is still to come as well.

Happy to simplify this to a plain cmp::min(remaining, 1) if you prefer. In
#146 you wrote "If 2 is a valid lower bound, a number that is 2 or lower is
definitely also valid", so the one-liner is equally sound - I kept the branch
only so that a fresh iterator's hint is byte-identical to what it is today
rather than weakening it.

Why the upper bound is left alone

Tightening it to Some(remaining + 1) is arithmetically right for
SentenceBreaks on its own, but USentenceBounds subtracts 1 unconditionally
even after it has been primed, and that combination under-reports on "\r\r":

split_sentence_bounds: with 1 of 2 items taken, the 1 left exceed size_hint upper 0 on "\r\r"

That looks like a separate latent issue in the same function. The narrower fix
would be for USentenceBounds::size_hint to subtract 1 only while
sentence_start.is_none(), after which the upper bound here could be tightened
too. I left that out of this PR rather than rewrite a recently merged change;
happy to follow up.

Relation to #146

#146 was closed because the reporter had misread cmp::min, and that part was
correct. But the direction it half-identified is real, and it is the direction
you named yourself: "I think a lower bound of 2 is incorrect for small
strings." You also left the thread with "It's possible that there's a way to
produce sentence breaks of length 1 that this code does not account for."

The added test answers that question. Driven at every position over the full
TEST_SENTENCE/TEST_WORD/TEST_SAME corpus, the fresh bound is sound - the
bug is not in the fresh value but in the fact that it never updates. It also
clears the other two types #146 accused: UWordBounds and Graphemes pass the
stronger invariant unmodified.

I have not reproduced the Vec::with_capacity capacity overflow that reporter
mentioned, and I am not claiming this is that. size_hint is documented as
untrustworthy, so the practical effect here is over-reservation, not UB.

Tests

test_size_hint_is_a_valid_bound only ever queries iterators that have not been
advanced, so a bound measured from the whole input passes it at every input.
The new test_size_hint_is_a_valid_bound_while_iterating uses the same corpus
and all ten public iterators but checks the hint at every position. On master
only split_sentence_bounds fails it; the other nine already satisfy it.

There is also a unit test on SentenceBreaks directly, and it is load-bearing:
USentenceBounds does saturating_sub(1), so a SentenceBreaks lower bound of
0 and one of 1 are indistinguishable through every public API. Without that test,
replacing remaining with slen again passes the whole public suite.

Verification

  • cargo test green: 18 lib + 11 integration + 23 doctests.
  • cargo clippy --all-targets --all and cargo fmt --all --check clean with
    RUSTFLAGS=-D warnings.
  • Green on 1.86.0 (nearest toolchain available to me above the 1.85.0 MSRV).
    The only construct that could matter, for s in [ ... ] by value in edition
    2018, is already shipped in src/grapheme.rs:931.
  • Reverting just the size_hint body fails at src/sentence.rs:436.
  • Four mutations of the fix, each caught: tightening the upper bound as well
    ("\r\r"), dropping the pos == 0 guard (fresh "ab" regresses to 0),
    measuring slen again, and off-by-one in the other direction.
  • src/tables.rs and tests/testdata/mod.rs are untouched, so the generator
    diff steps are unaffected. No public item is added, removed, or changed, so
    cargo-semver-checks is unaffected - SentenceBreaks lives in the private
    mod fwd.

Note on #179

cometkim's open #179 also touches src/sentence.rs, a few lines from this hunk.
It changes sentence_category call sites rather than size_hint, so there is no
semantic overlap, but whichever lands second may want a trivial rebase.

Disclosure

I used an AI assistant to help find and prepare this change. I reviewed and
tested it myself, and the outputs quoted above are from runs I performed.

SentenceBreaks walks its input with pos and never shortens string, so a
bound measured from string.len() keeps describing the whole input. The
hint stayed at its initial value for the life of the iterator, and once
the iterator was exhausted it still promised at least one more item.

Graphemes::size_hint already measures the span that is left, and
UWordBounds::size_hint can measure string.len() because next and
next_back reslice string. SentenceBreaks is the only one of the three
that advances an index and then measures the original.

Measure from pos instead. The upper bound is deliberately unchanged:
USentenceBounds subtracts one unconditionally, so tightening the upper
bound here under-reports on "\r\r".

Extend the size_hint test to query the hint at every position rather
than only on a fresh iterator, and add a unit test on SentenceBreaks
itself, whose lower bound of 0 and 1 are indistinguishable through the
public API.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant