Skip positions that cannot start a regexp match - #1653
Open
andreasrosdal wants to merge 3 commits into
Open
Conversation
An unanchored pattern is compiled with a `.*?` prologue, so a search over a
non-matching subject re-enters lre_exec_backtrack() at every position in the
string and walks the whole pattern before failing. On a 880 KB subject that
is millions of interpreter dispatches to conclude that a literal is absent.
Read the first mandatory element of the pattern once, before the search, and
summarise what a match must start with: nothing usable, an anchor, a single
character, or a 256-bit bitmap of the Latin-1 characters that can start one
plus a flag for "anything >= U+0100 might". The search loop then advances
through the subject in C -- memchr() for the single-character case -- and only
enters the matcher at positions that survive the filter. An anchored pattern
tries position 0 and stops.
The filter is derived from the bytecode, so it follows whatever the compiler
emitted: REOP_char/char32 give the character case, the case-insensitive
variants and REOP_range* give the bitmap (canonicalising when the range is
case-insensitive), REOP_line_start gives the anchor, and the save/register
housekeeping opcodes are stepped over. Anything else leaves the filter unset
and the old path runs unchanged. Sticky patterns are excluded, since they do
not search.
Positions inside a surrogate pair are rejected when the subject is scanned by
code point, so /\udf06/u still fails to match "𝌆".
Best of 3, 880 KB subject, x86-64 -O2:
/needle/ (no match) 2401 ms -> 91 ms
/zebra jumps/ (rare first char) 2361 ms -> 91 ms
/[0-9]{4}-[0-9]{2}/ (class start) 3208 ms -> 223 ms
/xyzzy/i (case-insensitive, no match) 1341 ms -> 146 ms
/^the quick brown fux/ (anchored) 2021 ms -> 0 ms
replace(/fox/g) 540 ms -> 283 ms
/^delta/m (filter declines) 564 ms -> 637 ms
The last line is a path the filter does not handle, so the slowdown is code
layout rather than added work.
built-ins/RegExp (1879), built-ins/String (1223) and annexB/built-ins/RegExp
(36) are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014mv33YvfHz7t9mmkituBnn
The prefilter may only skip input positions that truly cannot start a match, so the test targets the places where the set of possible first characters is easy to get wrong: case folding that crosses the Latin-1 boundary (the Kelvin sign and long s fold into ASCII, but only under the u flag), ranges that stop just below or reach just past U+00FF, `^` with and without the m flag, the sticky flag, astral characters and lone surrogates. It also covers the first elements that cannot be summarised at all -- assertions, lookaround, alternation, a quantified first atom -- and that a failed attempt leaves no captures behind for a later one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K6eRbuuuCujKgQkrHgvMrc
The search skips positions that cannot begin a match, so the tests are
about the set of possible first characters staying a superset of the
truth. Added the shapes that set is easiest to get wrong and that the
first round did not reach:
- a NUL first character, where the 8-bit search is a memchr() and NUL is
the one byte a C string scan would stop at rather than find, over both
an 8-bit and a 16-bit buffer
- dotAll, which changes what the first element accepts
- inline modifier groups, where the first element's foldedness is not
the pattern's
- the v flag's set operations, string literals and negation
- lastIndex landing in the middle of a surrogate pair, at every index of
an astral string, with and without the u flag
- long 16-bit and astral buffers whose only candidate is the last
position
- full width digits and letters, which are neither \d nor \w
All of them pass unchanged on master, which is the point: the search is
an optimisation and must not be observable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An unanchored pattern is compiled with a
.*?prologue, so searching a non-matching subject re-enterslre_exec_backtrack()at every position and walks the whole pattern before failing. On a 880 KB subject that is millions of interpreter dispatches just to conclude a literal is absent.Read the first mandatory element of the pattern once, before the search, and summarise what a match must start with: nothing usable, an anchor, a single character, or a 256-bit bitmap of the Latin-1 characters that can start one plus a flag for anything ≥ U+0100 might. The search loop then advances through the subject in C —
memchr()for the single-character case — and only enters the matcher at positions that survive the filter. An anchored pattern tries position 0 and stops.The filter is derived from the bytecode, so it follows whatever the compiler emitted:
REOP_char/char32give the character case, the case-insensitive variants andREOP_range*give the bitmap (canonicalising when the range is case-insensitive),REOP_line_startgives the anchor, and the save/register housekeeping opcodes are stepped over. Anything else leaves the filter unset and the old path runs unchanged. Sticky patterns are excluded, since they do not search.Positions inside a surrogate pair are rejected when the subject is scanned by code point, so
/\udf06/ustill fails to match"𝌆".Benchmarks
Best of 3, 880 KB subject, x86-64
-O2:/needle/(no match)/zebra jumps/(rare first char)/[0-9]{4}-[0-9]{2}/(class start)/xyzzy/i(case-insensitive, no match)/^the quick brown fux/(anchored)replace(/fox/g)/^delta/m(filter declines)The last row is a path the filter does not handle — it leaves the old code untouched — so the slowdown is code layout rather than added work. It reproduces across runs; happy to look at it further if that matters.
Testing
built-ins/RegExpbuilt-ins/StringannexB/built-ins/RegExp🤖 Generated with Claude Code
https://claude.ai/code/session_014mv33YvfHz7t9mmkituBnn
Generated by Claude Code