Base compiles a regex and runs it on a Pike VM - #937
Open
phenomenon0 wants to merge 3 commits into
Open
phenomenon0 wants to merge 3 commits into
phenomenon0 wants to merge 3 commits into
Conversation
Regex.compile parses a pattern of CPython 3.11's re under re.ASCII, in
part, into Pike VM code; Regex.exec, Regex.match_at and Regex.fullmatch
are Pattern.search(s, at), match(s, at) and fullmatch(s), a Regex.Match
giving the span and each group's (None if it took no part), and
Regex.group a span's text. The part: literals, `.`, sets with ranges and
negation, \d \w \s \D \W \S \b \B and the control escapes, ^ and $,
groups and (?:, |, and * + ? greedy or lazy. Offsets count code points.
The VM keeps its threads in priority order and visits a pc once per
step, so a run is linear in the subject: ^(a+)+$ on twenty a's and a b
is one pass. A pattern out of the part fails to compile: counted
repeats (any `{` outside a set, where CPython reads a lone one as a
literal), flags and group extensions, \x \u \A \Z and backrefs,
possessive repeats, and a * or + on a body that can match empty (CPython
takes it; the VM would need an empty-iteration check).
It is Base alone: comp.ts and bend.ts are untouched. The C and JS
emitted for the 623 tests that build are byte-identical but for one
rename, List.contains's specialization _0 -> _1 in tests/base/list_fold,
since Regex.close takes the first. The types
are Regex.Node, Regex.Inst and Regex.Match, since a top-level Match
collides with the def of that name in
evals/hell.glob_backtrack_match.bend. base.bend 25423 -> 31449 ttok.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
parse pins the code and the errors, the cuts among them; exec and captures pin search, match_at, fullmatch and group text on rows each checked against CPython first; laws ties the three together (exec is the first match_at from at on, match_at(i) starts at i, fullmatch spans the subject), and each law fails under a mutant of the VM; oracle is 160 random rows (seed 7) in the slice's grammar and a tenth of patterns CPython rejects, with CPython's answers pinned; smoke runs subjects across one !. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
abcd|a on "abab" finds 0-1 while the abcd thread is still alive; a VM that kept adding fresh starts after that would end on 2-3. The laws passed such a VM (only the oracle caught it); with this row the first law fails under it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This follows the regex design in #873, taken in smaller steps. It is the first
step: a regex in Base alone, with no runtime or compiler change.
Regex.compile(p)gives aResult<String, Regex>. On a compiled regex:Regex.exec(re, s, at)issearch.Regex.match_at(re, s, at)ismatch.Regex.fullmatch(re, s)isfullmatch.Regex.group(s, m, k)gives a group's text.The semantics are CPython 3.11
reunderre.ASCII, for literals,.,sets,
\d \w \s \D \W \S \b \B,^ $, groups,(?:,|, and* + ?,greedy or lazy. Offsets count code points. The VM is a Pike VM with threads
in priority order, so a run is linear in the subject.
A pattern outside the slice fails to compile; it never runs with a different
meaning. The cut constructs each fail this way, and each is a later step:
{fails too, where CPython reads a literal)\x \u \0, backrefs,\A \Z*/+on a body that can match emptyAlso not in this step:
findall/split/sub, Unicode classes, and anative VM.
The types are
Regex.*because a top-levelMatchwould collide withdef Matchinevals/hell.glob_backtrack_match.bend.bun gates/repo.tstests/regex, 6 testsgates/test.ts's steps run locally)tests/regex/oracle.bend, 160 generated rows: 0 diffs. Offline, 5 × 1,000 rows: 0 diffs on C and JS.match_atfromaton;match_at(i)starts at i;fullmatchspans all. Each law fails under a VM mutant.!smoke!; on CUDA,--gpu onandoffgive the pin--checkupover all 1,412 tests is byte-identical. The C and JS emitted for the 623 tests that build are byte-identical but for one rename (List.contains's specialization_0→_1intests/base/list_fold).comp.ts,bend.ts,main.tsuntouchedNot run here: the cluster gate and Metal. The
!smoke has run on CUDAonly.
Size: +1,061 lines over 7 files. Base is +485 of those, and 369 test
lines are oracle data.
Base has 551 ttok left after this, so the next Base step will need room.
🤖 Generated with Claude Code