Track what is left in SentenceBreaks::size_hint - #182
Open
youdie006 wants to merge 1 commit into
Open
Conversation
SentenceBreaks walks its input with pos and never shortens string, so a bound measured from string.len() keeps describing the whole input. The hint stayed at its initial value for the life of the iterator, and once the iterator was exhausted it still promised at least one more item. Graphemes::size_hint already measures the span that is left, and UWordBounds::size_hint can measure string.len() because next and next_back reslice string. SentenceBreaks is the only one of the three that advances an index and then measures the original. Measure from pos instead. The upper bound is deliberately unchanged: USentenceBounds subtracts one unconditionally, so tightening the upper bound here under-reports on "\r\r". Extend the size_hint test to query the hint at every position rather than only on a fresh iterator, and add a unit test on SentenceBreaks itself, whose lower bound of 0 and 1 are indistinguishable through the public API.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The inconsistency
SentenceBreaks::size_hintmeasuresself.string.len(), the whole originalinput, but
nextnever shortensstring- it only advancespos(
src/sentence.rs:193). So the hint never moves.Your own
Graphemes::size_hintatsrc/grapheme.rs:109already measures thespan that is actually left:
UWordBounds::size_hintatsrc/word.rs:219does writeself.string.len(),but there it is correct, because
nextreslices withself.string = &self.string[idx..](src/word.rs:447) andnext_backwithself.string = &self.string[..idx](src/word.rs:705).SentenceBreaksis theonly one of the three that walks with an index and then measures the original.
(Line numbers from
048d51f.)What a caller sees
The lower bound stays at 1 with nothing left. On a longer input it is frozen for
the whole walk:
USentenceBounds::size_hint(src/sentence.rs:377) andUSentenceBoundIndices::size_hint(:414) both inherit this.unicode_sentences()does not: it is a
Filter, and std'sFilter::size_hintzeroes the lower bound,so it is unaffected.
The change
Measure from
pos. Thepos == 0branch keeps today's bound for a freshiterator, where the start-of-text break is still to come as well.
Happy to simplify this to a plain
cmp::min(remaining, 1)if you prefer. In#146 you wrote "If
2is a valid lower bound, a number that is 2 or lower isdefinitely also valid", so the one-liner is equally sound - I kept the branch
only so that a fresh iterator's hint is byte-identical to what it is today
rather than weakening it.
Why the upper bound is left alone
Tightening it to
Some(remaining + 1)is arithmetically right forSentenceBreakson its own, butUSentenceBoundssubtracts 1 unconditionallyeven after it has been primed, and that combination under-reports on
"\r\r":That looks like a separate latent issue in the same function. The narrower fix
would be for
USentenceBounds::size_hintto subtract 1 only whilesentence_start.is_none(), after which the upper bound here could be tightenedtoo. I left that out of this PR rather than rewrite a recently merged change;
happy to follow up.
Relation to #146
#146 was closed because the reporter had misread
cmp::min, and that part wascorrect. But the direction it half-identified is real, and it is the direction
you named yourself: "I think a lower bound of
2is incorrect for smallstrings." You also left the thread with "It's possible that there's a way to
produce sentence breaks of length 1 that this code does not account for."
The added test answers that question. Driven at every position over the full
TEST_SENTENCE/TEST_WORD/TEST_SAMEcorpus, the fresh bound is sound - thebug is not in the fresh value but in the fact that it never updates. It also
clears the other two types #146 accused:
UWordBoundsandGraphemespass thestronger invariant unmodified.
I have not reproduced the
Vec::with_capacitycapacity overflow that reportermentioned, and I am not claiming this is that.
size_hintis documented asuntrustworthy, so the practical effect here is over-reservation, not UB.
Tests
test_size_hint_is_a_valid_boundonly ever queries iterators that have not beenadvanced, so a bound measured from the whole input passes it at every input.
The new
test_size_hint_is_a_valid_bound_while_iteratinguses the same corpusand all ten public iterators but checks the hint at every position. On master
only
split_sentence_boundsfails it; the other nine already satisfy it.There is also a unit test on
SentenceBreaksdirectly, and it is load-bearing:USentenceBoundsdoessaturating_sub(1), so aSentenceBreakslower bound of0 and one of 1 are indistinguishable through every public API. Without that test,
replacing
remainingwithslenagain passes the whole public suite.Verification
cargo testgreen: 18 lib + 11 integration + 23 doctests.cargo clippy --all-targets --allandcargo fmt --all --checkclean withRUSTFLAGS=-D warnings.The only construct that could matter,
for s in [ ... ]by value in edition2018, is already shipped in
src/grapheme.rs:931.size_hintbody fails atsrc/sentence.rs:436.(
"\r\r"), dropping thepos == 0guard (fresh"ab"regresses to 0),measuring
slenagain, and off-by-one in the other direction.src/tables.rsandtests/testdata/mod.rsare untouched, so the generatordiff steps are unaffected. No public item is added, removed, or changed, so
cargo-semver-checksis unaffected -SentenceBreakslives in the privatemod fwd.Note on #179
cometkim's open #179 also touches
src/sentence.rs, a few lines from this hunk.It changes
sentence_categorycall sites rather thansize_hint, so there is nosemantic overlap, but whichever lands second may want a trivial rebase.
Disclosure
I used an AI assistant to help find and prepare this change. I reviewed and
tested it myself, and the outputs quoted above are from runs I performed.