fix(linter): exclude string-literal contents from paren/quote balance checks - #50
Conversation
… checks
The linter counted parentheses and quotes inside string literals,
producing false positive "unbalanced" warnings for code like छपाउ("(क").
Use _tokenize_preserving_strings from the transpiler to extract only
the code portions of each line before counting. Also add single-quote
balance checking (previously only double quotes were checked).
Regression tests cover: parens/quotes inside strings (no error),
genuinely unbalanced lines (still error), single-quote balance,
apostrophe in double-quoted strings.
Closes alphacrack#30
alphacrack
left a comment
There was a problem hiding this comment.
Really solid contribution — the fix reuses _tokenize_preserving_strings exactly as #30 suggested, adds the missing single-quote balance check, registers against the existing security marker correctly (the project runs --strict-markers, and security is declared in pyproject, so you're safe), and includes genuine negative controls. Thank you!
One test needs a fix before merge — it silently doesn't test what its name says:
def test_quote_inside_string_not_flagged():
code = 'छपाउ("it''s ok")\n'Inside a single-quoted Python literal, '' closes and reopens the string — adjacent-literal concatenation. The value is actually छपाउ("its ok") (verified: repr(code) shows no apostrophe), so the apostrophe-inside-double-quotes case is never exercised. The underlying behavior is correct (I checked: छपाउ("it's ok") lints clean with your change), the test just needs to construct the string it claims to:
code = 'छपाउ("it\'s ok")\n' # escaped apostropheStyle suggestion while you're in there (non-blocking): the rest of test_linter.py uses literal Devanagari rather than \uXXXX escapes — literals would make these tests much easier to read for the Maithili-speaking contributors this project hopes to attract.
Happy to approve as soon as the one-line test fix lands. CI run is approved on my end.
|
Thanks for catching that. You were right—the adjacent string literals removed the apostrophe, so the test wasn't exercising the intended case. I've corrected the test, added an assertion that verifies the apostrophe is present in the constructed input, reran the focused linter/security tests, and pushed the update. |
alphacrack
left a comment
There was a problem hiding this comment.
Both review points are fully addressed — thank you for the careful follow-up. ✅
- The apostrophe test now uses an escaped
\'and guards withassert "it's ok" in code, so it genuinely exercises an apostrophe inside a double-quoted string (the earlier version collapsed via string concatenation). - Assertions switched to readable literal Devanagari (
"गोल ब्रैकेट","उद्धरण"). - Merge conflict reconciled against current
main; the paren/quote balance now tokenizes via_tokenize_preserving_stringsand correctly counts only code regions, and single-quote balance is checked too.
Verified locally: full suite is 159 passed, and ci-ok is green. Nice work — merging.
Problem
The linter in
maithili_dsl/transpiler/linter.pyflagged false warnings forunbalanced parentheses and quotes when those characters appeared inside string
literals. For example, valid code such as
छपाउ("(क")could produce anunbalanced-parenthesis warning.
Additionally, only double-quote balance was checked; single quotes were not
validated.
Root cause
lint_maithili_codecounted parentheses and quote characters across the fullline, including string-literal contents. Characters inside string literals
should not contribute to code-level balance checks.
Changes
_tokenize_preserving_strings, the existing string-aware tokenizer;_make_keyword_pattern-based function call-sitedetection from
main;Files changed:
maithili_dsl/transpiler/linter.pytests/test_linter.pytests/test_security.pyBehavior
The change ensures that:
strings;
sites;
Validation
Validated after reconciling the branch with the current
mainbehavior:tests/test_linter.py: 44 passedtests/test_security.py: 43 passedpython -m compileall maithili_dsl tests: passedmain: passed without conflictsThe full Windows suite still has five
cp1252subprocess-output failures. Thesame five failures were reproduced on the unchanged upstream baseline and are
not introduced by this PR.
GitHub Actions for the fork PR require maintainer approval before running.
Security considerations
This change reuses the existing tokenizer only for lint balance and call-site
checks. It does not alter transpilation, import validation, sandbox execution,
or dependency handling.
Security regression tests verify that code-like content inside strings remains
data and that disallowed code outside strings continues to be rejected.
Risk
Low. The change uses existing parsing infrastructure, introduces no dependency
or public API change, and is limited to the linter and its regression tests.
Closes #30
Disclosure: This contribution was prepared with AI assistance and independently
reviewed and tested locally before publication.