Add C++ ECMAScript regular-expression parser and tree view by Copilot · Pull Request #22220 · github/codeql

Copilot · 2026-07-21T09:52:52Z

Introduces a C++ regex parse library and parse-tree view targeting the ECMAScript grammar (default for std::regex), ported from the Ruby regex library. Implements RegexTreeViewSig from codeql/regex. No queries, dataflow, or flow-configs — parser and tree view only.

Structure

9 commits, each a single concern:

Verbatim Ruby copy — ParseRegExp.qll + RegExpTreeView.qll copied unchanged (non-compiling baseline)
Compile against C++ — swap Ruby AST types → StringLiteral; rewire getValue()/getLocation()/imports
Hermetic test corpus — test.cpp with corpus wrapped in std::regex stubs (no #includes); real .expected from codeql test run --learn
Curate corpus — remove Ruby-only quantifier {,8}, mode variants, single-quote named groups (?'foo'...)
ECMAScript dialect — remove \A/\Z/\z/\G anchors, \h/\H escapes; fix nonCapturingGroupStart to exclude < (was mis-tokenizing (?<=/(?<!); remove Ruby single-quote group parser branch
POSIX extension tests (pre-fix) — [[:alpha:]], [[.a.]], [[=a=]], mixed/adjacent cases; expected captures wrong pre-fix tokenization
POSIX extension impl — add posixCollatingSymbol/posixEquivalenceClass predicates; inAnyPosixBracket helper prevents inner [/] from acting as charset delimiters
Location tests (pre-fix) — locations.ql + cases for plain "...", raw R"(...)", custom-delimiter R"x(...)x", L"...", LR"(...)", escape-containing; expected captures wrong pre-fix columns
Location fix — regexpContentOffset computes true content offset from getValueText():

"a+b"        → offset 1   (unchanged)
L"a+b"       → offset 2   (was 1)
R"(a+b)"     → offset 3   (was 1)
R"x(a+b)x"  → offset 4   (was 1, custom delim)
LR"(a+b)"    → offset 4   (was 1)

Files

Path	Purpose
`cpp/ql/lib/semmle/code/cpp/regex/internal/ParseRegExp.qll`	ECMAScript regex parser over `StringLiteral`
`cpp/ql/lib/semmle/code/cpp/regex/RegexTreeView.qll`	Tree view implementing `RegexTreeViewSig`
`cpp/ql/test/library-tests/regex/test.cpp`	Hermetic corpus (no external headers; `std::` stubs inline)
`cpp/ql/test/library-tests/regex/{parse,regexp,locations}.{ql,expected}`	Test queries + real output
`cpp/ql/lib/change-notes/2026-07-21-cpp-regex-tree-view.md`	Unreleased change note (`minorAnalysis`)

Test reproducibility

Bundle: CodeQL 2.26.1
Command: codeql test run --learn --search-path=<repo-root> cpp/ql/test/library-tests/regex/

All .expected files are real extractor output — none are hand-authored or placeholder.

…point) This is a verbatim copy of the Ruby ParseRegExp.qll and RegExpTreeView.qll, with only the file-header comments updated to reflect the new path. The test files (parse.ql, regexp.ql, regexp.rb, *.expected) are also copied verbatim from the Ruby test suite as a baseline reference. This commit does NOT compile against C++: it still imports codeql.ruby.AST and references Ruby AST types (RegExpLiteral, StringlikeLiteral, etc.). The adaptation to compile against C++ is in the next commit. Also adds the codeql/regex dependency to cpp/ql/lib/qlpack.yml and the unreleased change-note at cpp/ql/lib/change-notes/2026-07-21-cpp-regex-tree-view.md.

Changes from verbatim Ruby baseline (Commit 1) to compilable C++: ParseRegExp.qll: - Remove: private import codeql.ruby.AST as Ast - Remove: private import codeql.Locations - Add: private import semmle.code.cpp.exprs.Literal - Change RegExp base: Ast::StringlikeLiteral -> StringLiteral - Replace getText(): removes getConstantValue() Ruby accessor, uses getValue() (C++ StringLiteral.getValue() returns the string content, analogous to Ruby's getConstantValue().getString()) RegexTreeView.qll: - Remove Ruby-specific imports (codeql.ruby.AST, codeql.Locations, codeql.regex.nfa.NfaUtils) - Add: private import semmle.code.cpp.exprs.Literal - Keep: codeql.util.Numbers (for parseHexInt in RegExpEscape.getUnicode()) - Keep: codeql.regex.RegexTreeView (satisfies RegexTreeViewSig) - Change getParsedRegExp: Ast::RegExpLiteral -> StringLiteral - Simplify hasLocationInfo to use re.getLocation() directly with +1 offset for opening quote (approximate; fixed in commits 8-9) - Replace isExcluded: remove hasFreeSpacingFlag(), use none() - Ruby dialect features preserved unchanged (\\A, \\z, \\G, \\h, etc.); dialect shift is commit 5 Test queries: - parse.ql, regexp.ql: replace codeql.ruby.Regexp with semmle.code.cpp.regex.RegexTreeView - .expected cleared (no test.cpp until commit 3) Verified: both queries compile successfully with codeql query compile.

…expected Introduces cpp/ql/test/library-tests/regex/test.cpp with the Ruby regex corpus mechanically translated to C++ string literals and wrapped in std::regex construction calls using fully self-contained in-file stubs (no #include of any external/standard header). Stub surface: std::basic_regex<CharT> constructor(const char*), constructor(const char*, int), assign(const char*) std::regex typedef for basic_regex<char> std::regex_match / regex_search / regex_replace free functions 1:1 corpus mapping from regexp.rb: - All Ruby /regex/ literals translated to C++ "string" literals (each backslash doubled: /\d/ -> "\\d") - Dropped: /#{A}bc/ (Ruby string interpolation, no string-literal form) - Kept: a{,8}, .*m, \A, \z, \G, \h\H, (?'foo'...) — these are removed in commits 4 and 5 (keeping them here preserves 1:1 mapping) Also removes `abstract` from RegExp class in ParseRegExp.qll so that all StringLiterals are regex candidates (trivial syntactic gate; no dataflow). .expected generated by: codeql test run --learn --search-path=. cpp/ql/test/library-tests/regex/ (CodeQL CLI 2.26.1, bundled C/C++ extractor) All 2 tests passed.

Content-only edits to test.cpp to remove Ruby-only syntax that has no C++ ECMAScript equivalent: 1. Removed: "a{,8}" — Ruby-only "{,n}" no-lower-bound quantifier (ECMAScript requires an explicit lower bound in {n,m}) 2. Removed: second ".*" for /.*/m — mode flags in C++ are construction-site arguments (e.g. std::regex::multiline), not part of the pattern string. This mode variant was the exact same pattern string as r_meta1 so it added no coverage. 3. Removed: "(?'foo'fo+)" — Ruby single-quote named-group form. ECMAScript only supports the angle-bracket form (?<name>...). Left in place for now (removed with the parser in commit 5): \\A, \\z, \\G, \\h, \\H — Ruby-only anchor/escape classes. The POSIX-bracket cases are kept through commit 7. .expected regenerated by: codeql test run --learn --search-path=. cpp/ql/test/library-tests/regex/ (CodeQL CLI 2.26.1) All 2 tests passed.

- ParseRegExp.qll: - specialCharacter: remove \A, \Z, \z, \G (Ruby-only anchors); keep only \b, \B - firstPart: remove \A arm; keep only ^ for start-of-string - lastPart: remove \Z, \z arms; keep only $ for end-of-string - nonCapturingGroupStart: remove < from char set (handled by namedGroupStart and lookbehindAssertionStart; was causing mis-tokenization of lookbehind) - namedGroupStart: remove Ruby-only single-quote (?'name'...) branch; keep only ECMAScript (?<name>...) syntax - RegexTreeView.qll: - RegExpCharacterClassEscape: remove h, H (Ruby hex-digit classes); keep d, D, s, S, w, W (ECMAScript set) - RegExpAnchor: change to [^, $] only (remove \A, \Z, \z) - RegExpDollar: change to $ only (remove \Z, \z) - RegExpCaret: change to ^ only (remove \A) - test.cpp: - r_cc3: replace \A[+-]?\d+ with ^[+-]?\d+ (ECMAScript caret) - r_meta5 \h\H: removed (Ruby-only hex classes) - r_anc1 \Gabc: removed (\G not in ECMAScript) - Regenerated parse.expected and regexp.expected via codeql test run --learn (CodeQL 2.26.1); all 2 tests pass.

…rect tokenization) Add new test cases to test.cpp exercising POSIX bracket expressions inside character classes as supported by std::regex ECMAScript mode: - Single POSIX classes: [[:alpha:]], [[:digit:]], [[:space:]], [[:upper:]], [[:lower:]], [[:alnum:]], [[:print:]], [[:punct:]] - Negated outer bracket: [^[:space:]] - Mixed: regular char + POSIX class [a[:space:]] - POSIX collating symbol [[.a.]] - POSIX equivalence class [[=a=]] - Mixed POSIX + range [[:alpha:]0-9] The existing r_posix1-r_posix4 cases remain untouched. New cases use distinct names (r_posix_alpha, r_posix_digit, etc.) to avoid redeclaration. Generated .expected via codeql test run --learn (CodeQL 2.26.1); all 2 tests pass. The expected output captures the CURRENT (pre-fix) tokenization — commit 7 will change these rows when correct POSIX nesting is implemented.

Add posixCollatingSymbol([.x.]) and posixEquivalenceClass([=x=]) predicates to ParseRegExp.qll, enabling correct tokenization of all three POSIX bracket atom types inside character classes: - [:name:] — already handled, unchanged - [.x.] — new: posixCollatingSymbol predicate - [=x=] — new: posixEquivalenceClass predicate Refactored: - charSetDelimiter: use inAnyPosixBracket helper (covers all three types) - charSet closing: use inAnyPosixBracket helper - inPosixBracket: updated to cover all three types - simpleCharacter: updated to exclude all three types - namedCharacterProperty: includes posixCollatingSymbol and posixEquivalenceClass - inAnyPosixBracket: new private helper; uses pos in [x..y-1] for QL binding Before this commit (commit 6): [[.a.]] → RegExpCharacterClass([[.a.]) + stray ] (WRONG) [[=a=]] → RegExpCharacterClass([[=a=]) + stray ] (WRONG) After this commit: [[.a.]] → RegExpCharacterClass containing RegExpNamedCharacterProperty [.a.] [[=a=]] → RegExpCharacterClass containing RegExpNamedCharacterProperty [=a=] Regenerated parse.expected and regexp.expected via codeql test run --learn (CodeQL 2.26.1); all 2 tests pass.

Add location test cases to test.cpp covering various C++ string literal forms: - r_plain: "a+b" — plain, offset 1 (already correct) - r_raw: R"(a+b)" — raw, offset 3 (R"( = 3); currently uses 1 (WRONG) - r_raw2: R"(\s+$)" — raw with metacharacters; currently wrong - r_raw3: R"($([,\w]+)+$$)" — complex raw; currently wrong - r_raw4: R"x(a+b)x" — custom-delimiter raw, offset 4; currently uses 1 (WRONG) - r_wide: L"a+b" — L" prefix, offset 2; currently uses 1 (WRONG) - r_wide_raw: LR"(a+b)" — combined LR"( prefix, offset 4; currently uses 1 (WRONG) - r_esc1: "\\s+" — escape-containing plain; offset 1 (correct) - r_esc2: "a\\.b" — escaped dot; offset 1 (correct) Added std::wregex typedef to stubs (for L"..." and LR"(...)" wide-char literals). New locations.ql query reports per-term: litStartCol, valueStart, valueEnd, termStartCol, termEndCol — restricted to test_locations() function. Generated locations.expected via codeql test run --learn (CodeQL 2.26.1). All 3 tests pass. The expected output captures the CURRENT (pre-fix) columns: raw/prefixed rows show termStartCol = litStartCol + 1 (wrong); plain rows show litStartCol + 1 (correct). Commit 9 changes the raw/prefixed rows to their correct values.

…de escape RegexTreeView.qll - hasLocationInfo fix (RULE 4): - Add regexpContentOffset(RegExp re) private helper that computes the correct number of source chars before the first content char, from getValueText(): - Plain "...": offset 1 - L"...": offset 2 (L" prefix) - u8"...": offset 3 (u8" prefix) - R"(...)": offset 3 (R"( opener) - R"x(...)x": offset 4 (R"x( with custom delim) - LR"(...)": offset 4 (LR"() - Uses vt.matches("%R\"%(%") to detect raw strings; finds '(' position - For non-raw, finds '"' position - Replace hardcoded `+ 1` with `+ regexpContentOffset(re)` in hasLocationInfo - Update comment: documents plain/encoding/raw/combined forms, notes that escaped non-raw strings are approximate (mirroring Java/Python) test.cpp: - Fix r_uni: change "\\u{9879}" to "\\u9879" — braced \u{...} is not standard C++ regex syntax; the 4-digit \uHHHH form is valid ECMAScript \u escape Regenerated all 3 .expected files via codeql test run --learn (CodeQL 2.26.1). All 3 tests pass. Location test confirms: - Plain "a+b" (line 144): litStartCol 22 → termStartCol 23 (22+1) UNCHANGED ✓ - Raw R"(a+b)" (line 147): litStartCol 20 → termStartCol 23 (20+3) FIXED ✓ - Raw R"x(a+b)x" (line 156): litStartCol 21 → termStartCol 25 (21+4) FIXED ✓ - L"a+b" (line 159): litStartCol 22 → termStartCol 24 (22+2) FIXED ✓ - LR"(a+b)" (line 162): litStartCol 26 → termStartCol 30 (26+4) FIXED ✓

std::regex ECMAScript mode does not support \p{Name}; \p tokenizes as an identity escape. Remove pStyleNamedCharacterProperty and its call sites; namedCharacterProperty now covers only POSIX brackets ([:name:], [.x.], [=x=]) and namedCharacterPropertyIsInverted keeps only the [[:^name:]] case (fixing the offset from start+3 to start+2 for POSIX). Update RegExpNamedCharacterProperty getName/isInverted docs to describe POSIX brackets. Remove the four \p{...}/\P{...} corpus lines from test.cpp; keep a plain [a-f\d]+ case for the class-with-escape shape. Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).

In ECMAScript std::regex, \0 matches NUL. The previous escapedCharacter arms all rejected it: the final arm's `not exists(getChar(start+1).toInt())` guard fails for "0", and the numbered-backref arm excludes 0. Add an explicit \0 case (end=start+2) guarded so the following character is not a digit, mirroring EcmaRegExp.escapedCharacter in PR #22200. Add corpus case "a\\0b" to test.cpp; \0 now parses as RegExpEscape. Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).

\U is a Ruby/PCRE-ism, not ECMAScript. Restrict RegExpEscape.isUnicode()/getUnicode() to lowercase \uHHHH only. No \U path exists in ParseRegExp.qll's escapedCharacter, and no corpus case references \U. .expected files unchanged. Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).

In default ECMAScript std::regex (no multiline flag; this parser cannot detect construction-site flags), ^ and $ anchor to string start/end, not line boundaries. Update RegExpCaret, RegExpDollar, and RegExpAnchor doc comments to describe string-start/string-end semantics and note that multiline is not modelled, mirroring PR #22200. Tokenization unchanged. .expected files unchanged. Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).

Add corpus cases: bare [:alpha:], mid-pattern a[:b:]c, POSIX class as range endpoint [[:alpha:]-z], malformed [[:alpha], leading literal ] combined with a POSIX class []a[:alpha:]], three POSIX classes in one class, additional names [[:xdigit:]] / [[:blank:]] / [[:cntrl:]] / [[:graph:]], and the integration case combining all of the above. The regenerated parse.expected reveals genuine mis-parses that this commit deliberately records (fixes in Commit 6): - unnested [:alpha:] and [:b:] are treated as RegExpNamedCharacterProperty even though POSIX brackets are only valid inside a character class, - [[:alpha:]-z] emits a spurious [RegExpCharacterRange] ":]-z" instead of a POSIX class + literal '-' + 'z'. Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).

POSIX bracket expressions ([:name:], [.x.], [=x=]) are only valid nested inside another character class in std::regex. The three predicates now additionally require that at position `start` we are inside an outer character class, approximated by requiring more non-escaped `[` than non-escaped `]` before `start`. A well-formed POSIX bracket contributes one `[` and one `]` at/after `start`, so the check is unaffected by earlier POSIX brackets. Also stop emitting individual charSetTokens for characters inside a POSIX bracket (`inAnyPosixBracket` guard), which removes the spurious `[RegExpCharacterRange] :]-z` on [[:alpha:]-z]. Post-fix, unnested `[:digit:]`, `[:alpha:]`, and `a[:b:]c` are correctly parsed as ordinary character classes / literal sequences rather than `RegExpNamedCharacterProperty`; and the [[:alpha:]-z] range no longer crosses the POSIX bracket boundary. Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).

New cpp/ql/test/library-tests/regex/Consistency.ql selects, for each corpus RegExp, characters that failedToParse. The empty Consistency.expected asserts that no corpus regex has any unparsed character, mirroring PR #22200's Consistency test as the primary regression guard for the parser. Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).

Add u8"..." and u8R"(...)" cases to test_locations in test.cpp, guarded by #if __cplusplus >= 201703L. Add an options file forcing -std=c++17 so the u8 char8_t-era rows are actually extracted. Plain "..." rows are unchanged. New u8/u8R rows in locations.expected show correct columns computed by regexpContentOffset: offset 3 for `u8"` and offset 5 for `u8R"(`. No change to regexpContentOffset was needed. Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).

cpp/ql/test/library-tests/regex/regexp.rb was the commit-1 verbatim Ruby reference and has been fully replaced by test.cpp. Nothing in the C++ regex tests references it. Deletion has no effect on .expected. Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).

Remove "commit N does X" and "currently wrong / pre-fix" narrative from test.cpp and locations.ql; where appropriate, replace with concise process-free descriptions of the code (e.g. "Raw string literal; content offset 3"). Line count and column positions of every regex literal are preserved (empty lines take the place of removed comment lines), so no .expected file changes. Remaining "Note:" comments in the .qll files describe code semantics (excluded forms / handling), not the build process, and are kept. Bundle: github/codeql-action codeql-bundle-v2.26.1 (CodeQL CLI 2.26.1).

Initial plan

f4dbbcd

Copilot AI assigned Copilot and jketema Jul 21, 2026

Copilot started work on behalf of jketema July 21, 2026 09:53 View session

Copilot AI added 9 commits July 21, 2026 10:01

Copilot AI changed the title ~~[WIP] Add C++ ECMAScript regular-expression parser and tree view~~ Add C++ ECMAScript regular-expression parser and tree view Jul 21, 2026

Copilot finished work on behalf of jketema July 21, 2026 10:38

Copilot AI requested a review from jketema July 21, 2026 10:38

Copilot started work on behalf of jketema July 21, 2026 12:03 View session

Copilot finished work on behalf of jketema July 21, 2026 12:05

Copilot started work on behalf of jketema July 21, 2026 12:11 View session

Copilot AI added 10 commits July 21, 2026 12:16

Copilot finished work on behalf of jketema July 21, 2026 12:32

Copilot started work on behalf of jketema July 21, 2026 14:49 View session

Remove named capture group support from C++ ECMAScript regex parser

f28d7f5

Copilot finished work on behalf of jketema July 21, 2026 14:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add C++ ECMAScript regular-expression parser and tree view#22220

Add C++ ECMAScript regular-expression parser and tree view#22220
jketema with Copilot wants to merge 21 commits into
mainfrom
copilot/add-cpp-ecmascript-parser

Copilot AI commented Jul 21, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Copilot AI commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Structure

Files

Test reproducibility

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Jul 21, 2026 •

edited

Loading