Skip to content

Fix RecursionError in reindent filters for chained BETWEEN...AND - #873

Open
lilu5458 wants to merge 2 commits into
andialbrecht:masterfrom
lilu5458:fix-recursion-between-and-v2
Open

Fix RecursionError in reindent filters for chained BETWEEN...AND#873
lilu5458 wants to merge 2 commits into
andialbrecht:masterfrom
lilu5458:fix-recursion-between-and-v2

Conversation

@lilu5458

@lilu5458 lilu5458 commented Aug 4, 2026

Copy link
Copy Markdown

Problem

_next_token in ReindentFilter and AlignedIndentFilter used recursion to skip BETWEEN ... AND pairs. With ~1000 chained pairs (8000 tokens, well under MAX_GROUPING_TOKENS=10000), this exceeded Python's default recursion limit of 1000, causing SQLParseError: Maximum recursion depth exceeded via FilterStack.run()'s RecursionError handler.

This means a valid SQL statement with many chained BETWEEN ... AND clauses cannot be formatted with sqlparse.format(sql, reindent=True) or sqlparse.format(sql, reindent_aligned=True).

PoC

import sqlparse

parts = ["x"]
for i in range(1000):
    parts.append(f"BETWEEN {i} AND {i + 1}")
sql = "SELECT * FROM t WHERE " + " ".join(parts)

# Raises SQLParseError: Maximum recursion depth exceeded
sqlparse.format(sql, reindent=True)

Fix

Convert the recursive _next_token calls to an iterative while loop that skips all consecutive BETWEEN ... AND pairs without growing the call stack. The iterative version handles chained and nested BETWEEN ... AND pairs identically to the original recursive version.

Testing

  • All existing tests pass (452 passed, 3 xfailed)
  • Added test_format_chained_between_and to verify 1000 chained BETWEEN ... AND pairs can be formatted with both reindent=True and reindent_aligned=True

Checklist

  • Tests added
  • Code follows project style (ruff)
  • No new dependencies

_next_token in ReindentFilter and AlignedIndentFilter used recursion
to skip BETWEEN...AND pairs. With ~1000 chained pairs (8000 tokens,
under MAX_GROUPING_TOKENS) this exceeded Python's recursion limit,
causing SQLParseError via FilterStack's RecursionError handler.

Convert the recursive calls to an iterative while loop that skips
all consecutive BETWEEN...AND pairs without growing the call stack.
Merge upstream/master into fix-recursion-between-and-v2 to resolve conflicts. The BETWEEN...AND recursion fix is preserved alongside upstream changes (issue 772, 782, 783, etc.). Conflicts in tests/test_format.py resolved by keeping both upstream tests and test_format_chained_between_and.

Note: .github/workflows/python-app.yml is kept at the old version due to token scope limitations. It will be updated when the PR is merged into upstream/master.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant