Python: fix files being silently dropped over Rust/Python escape mismatch

[redsun82]

A Python file could be silently dropped from the analysis, reported only as a generic "A parse error occurred" diagnostic pointing at syntax that is in fact perfectly valid. The reproducer in #22435 needs three ingredients at once, which is why it looked so arbitrary: PEP 695 syntax, a U+FE0F variation selector, and a % directive.

Those three ingredients turn out to be one trigger and two separate bugs.

The trigger

modules.py tries the old blib2to3 parser first and only falls back to tsg-python when it fails. PEP 695 (like t-strings or lazy import) is simply what pushes a file onto the tsg-python path. It is not implicated otherwise.

Bug 1: Rust's Debug format is not a subset of Python's

tsg-python prints its result with tree-sitter-graph's Graph::pretty_print(), which renders string values through Rust's Debug. The Python reader parses those values with ast.literal_eval. The two formats disagree in two ways:

• Characters Rust considers non-printable, including grapheme-extending ones, are rendered as \u{...}, which Python does not accept at all. This is much broader than the reported U+FE0F: U+200D zero width joiners, combining accents from NFD text, and soft hyphens all qualify, in string literals, comments and identifiers alike.
• NUL is rendered as \0, which Python reads as the start of an octal escape. So "\0" alone happens to work, but NUL followed by an octal digit silently decodes to the wrong character, with no exception raised.

Rather than reason about which escapes diverge, I checked. rust_to_python_escapes translates both cases, and I verified the result by generating Rust's Debug rendering of every Unicode scalar value and running literal_eval over each, plus a cross product of every distinct escape shape with every printable ASCII neighbour on both sides:

	SyntaxError	silently wrong value
before	960,486	8
after	0	0

out of 1,114,344 cases. The NUL bug was found by that audit, not by inspection. test_every_escape_shape_round_trips walks the same shape and neighbour table without needing a Rust toolchain, so the property stays guarded.

Worth noting: the non-fatal path was a silent correctness bug, not just a warning. The except handler left the value undecoded, so a Str node's s ended up holding the Rust-escaped source text including its quotes.

Bug 2: the warning about the failure was itself fatal

Logger.log applied %-formatting unconditionally. The warning message embeds repr(value), i.e. the offending source snippet, so a literal containing %s raised TypeError: not enough arguments for format string. modules.py converted that into a SyntaxError and the whole file was dropped. format_message now matches stdlib logging semantics and only formats when there are arguments.

Why this was invisible

StdoutLogger in the parser test harness counted warn and error towards its error count, but tsg_parser calls warning, which the harness silently ignored. So the parser tests could not have caught this class of bug. warning now counts.

While fixing that I standardised on warning as the method name, matching semmle.logging.Logger and the standard library, where warn was deprecated in 3.3 and removed in 3.13. Logger.warn never existed here, so the two remaining logger.warn calls in worker.py and imports.py would have raised AttributeError. Both sit inside except handlers, so they would have replaced a warning with a crash, the same shape as bug 2.

Notes for review

• The translation is deliberately applied once, to the raw tsg-python output line, and not to the reconstructed source that evaluate_string evaluates in its second literal_eval pass.
• The regex matches every escape sequence rather than only the offending ones. This keeps the scan in step with the backslashes so an escaped backslash, which is how a source-level r"\u{fe0f}" is serialised, is never mistaken for an escape introducer.
• Fixing the producer side instead was considered and rejected: same transformation, but tree-sitter-graph is third-party and the literal_eval contract lives on the Python side. There is now a comment at the pretty_print() call recording the coupling.
• is_printable follows Rust's Unicode tables, so a toolchain upgrade could escape more characters. That is fine, since anything new takes the already-handled \u{...} path. The audit ran on rustc 1.91.0.

Testing

tests/parser/unicode_escapes_new.py covers a variation selector next to a %s directive, a ZWJ sequence, NFD combining marks in both a literal and an identifier, a soft hyphen, a raw string containing a literal \u{fe0f}, an f-string, implicit concatenation and a comment. tests/test_tsg_parser.py adds fast unit coverage that needs no cargo build. Both original defects were confirmed red before the fix by reverting each in turn.

Fixes: #22435

[Copilot]

Pull request overview

Fixes Rust/Python escape incompatibilities that could corrupt parser output or drop valid Python files.

Changes:

• Translates Rust Unicode and NUL escapes before Python evaluation.
• Corrects logger formatting and deprecated warning calls.
• Adds parser and regression coverage for affected Unicode constructs.

Show a summary per file

File	Description
python/ql/lib/change-notes/2026-08-27-tsg-parser-unicode-escapes.md	Documents the parser fix.
python/extractor/tsg-python/src/main.rs	Documents serialization coupling.
python/extractor/tests/test_tsg_parser.py	Tests escape conversion and logging.
python/extractor/tests/parser/unicode_escapes_new.py	Adds Unicode parser cases.
python/extractor/tests/parser/unicode_escapes_new.expected	Records expected AST output.
python/extractor/semmle/worker.py	Uses the supported warning API.
python/extractor/semmle/python/passes/flow.py	Reuses safe message formatting.
python/extractor/semmle/python/parser/tsg_parser.py	Translates Rust escape syntax.
python/extractor/semmle/python/parser/dump_ast.py	Correctly counts parser warnings.
python/extractor/semmle/python/imports.py	Uses the supported warning API.
python/extractor/semmle/logging.py	Avoids formatting argument-free messages.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 11/11 changed files
• Comments generated: 2
• Review effort level: Balanced

[tausbn]

Overall, this looks sensible to me, but I have one comment about the logging fix that should maybe be addressed.