Skip to content

Base compiles a regex and runs it on a Pike VM - #937

Open
phenomenon0 wants to merge 3 commits into
bendlang:mainfrom
phenomenon0:regex-upstream
Open

phenomenon0 wants to merge 3 commits into
bendlang:mainfrom
phenomenon0:regex-upstream

Conversation

@phenomenon0

Copy link
Copy Markdown

This follows the regex design in #873, taken in smaller steps. It is the first
step: a regex in Base alone, with no runtime or compiler change.

Regex.compile(p) gives a Result<String, Regex>. On a compiled regex:

  • Regex.exec(re, s, at) is search.
  • Regex.match_at(re, s, at) is match.
  • Regex.fullmatch(re, s) is fullmatch.
  • Regex.group(s, m, k) gives a group's text.

The semantics are CPython 3.11 re under re.ASCII, for literals, .,
sets, \d \w \s \D \W \S \b \B, ^ $, groups, (?:, |, and * + ?,
greedy or lazy. Offsets count code points. The VM is a Pike VM with threads
in priority order, so a run is linear in the subject.

A pattern outside the slice fails to compile; it never runs with a different
meaning. The cut constructs each fail this way, and each is a later step:

  • flags
  • counted repeats (so a lone { fails too, where CPython reads a literal)
  • \x \u \0, backrefs, \A \Z
  • named groups and lookarounds
  • possessive repeats
  • */+ on a body that can match empty

Also not in this step: findall/split/sub, Unicode classes, and a
native VM.

The types are Regex.* because a top-level Match would collide with
def Match in evals/hell.glob_backtrack_match.bend.

check result
bun gates/repo.ts PASS 46 / 46 at each commit. No cap changed. base.bend 25,423 → 31,449 of 32,000 ttok.
tests/regex, 6 tests pass check, interp, C and JS (gates/test.ts's steps run locally)
differential vs CPython 3.11 tests/regex/oracle.bend, 160 generated rows: 0 diffs. Offline, 5 × 1,000 rows: 0 diffs on C and JS.
laws exec = first match_at from at on; match_at(i) starts at i; fullmatch spans all. Each law fails under a VM mutant.
! smoke one compiled regex over a list with !; on CUDA, --gpu on and off give the pin
everything else --checkup over all 1,412 tests is byte-identical. The C and JS emitted for the 623 tests that build are byte-identical but for one rename (List.contains's specialization _0_1 in tests/base/list_fold).
runtime comp.ts, bend.ts, main.ts untouched

Not run here: the cluster gate and Metal. The ! smoke has run on CUDA
only.

Size: +1,061 lines over 7 files. Base is +485 of those, and 369 test
lines are oracle data.

Base has 551 ttok left after this, so the next Base step will need room.

🤖 Generated with Claude Code

phenomenon0 and others added 3 commits September 20, 2026 23:18
Regex.compile parses a pattern of CPython 3.11's re under re.ASCII, in
part, into Pike VM code; Regex.exec, Regex.match_at and Regex.fullmatch
are Pattern.search(s, at), match(s, at) and fullmatch(s), a Regex.Match
giving the span and each group's (None if it took no part), and
Regex.group a span's text. The part: literals, `.`, sets with ranges and
negation, \d \w \s \D \W \S \b \B and the control escapes, ^ and $,
groups and (?:, |, and * + ? greedy or lazy. Offsets count code points.

The VM keeps its threads in priority order and visits a pc once per
step, so a run is linear in the subject: ^(a+)+$ on twenty a's and a b
is one pass. A pattern out of the part fails to compile: counted
repeats (any `{` outside a set, where CPython reads a lone one as a
literal), flags and group extensions, \x \u \A \Z and backrefs,
possessive repeats, and a * or + on a body that can match empty (CPython
takes it; the VM would need an empty-iteration check).

It is Base alone: comp.ts and bend.ts are untouched. The C and JS
emitted for the 623 tests that build are byte-identical but for one
rename, List.contains's specialization _0 -> _1 in tests/base/list_fold,
since Regex.close takes the first. The types
are Regex.Node, Regex.Inst and Regex.Match, since a top-level Match
collides with the def of that name in
evals/hell.glob_backtrack_match.bend. base.bend 25423 -> 31449 ttok.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
parse pins the code and the errors, the cuts among them; exec and
captures pin search, match_at, fullmatch and group text on rows each
checked against CPython first; laws ties the three together (exec is the
first match_at from at on, match_at(i) starts at i, fullmatch spans the
subject), and each law fails under a mutant of the VM; oracle is 160
random rows (seed 7) in the slice's grammar and a tenth of patterns
CPython rejects, with CPython's answers pinned; smoke runs subjects
across one !.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
abcd|a on "abab" finds 0-1 while the abcd thread is still alive; a VM
that kept adding fresh starts after that would end on 2-3. The laws
passed such a VM (only the oracle caught it); with this row the first
law fails under it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant