Fix RecursionError in reindent filters for chained BETWEEN...AND - #873
Open
lilu5458 wants to merge 2 commits into
Open
Fix RecursionError in reindent filters for chained BETWEEN...AND#873lilu5458 wants to merge 2 commits into
lilu5458 wants to merge 2 commits into
Conversation
_next_token in ReindentFilter and AlignedIndentFilter used recursion to skip BETWEEN...AND pairs. With ~1000 chained pairs (8000 tokens, under MAX_GROUPING_TOKENS) this exceeded Python's recursion limit, causing SQLParseError via FilterStack's RecursionError handler. Convert the recursive calls to an iterative while loop that skips all consecutive BETWEEN...AND pairs without growing the call stack.
Merge upstream/master into fix-recursion-between-and-v2 to resolve conflicts. The BETWEEN...AND recursion fix is preserved alongside upstream changes (issue 772, 782, 783, etc.). Conflicts in tests/test_format.py resolved by keeping both upstream tests and test_format_chained_between_and. Note: .github/workflows/python-app.yml is kept at the old version due to token scope limitations. It will be updated when the PR is merged into upstream/master.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_next_tokeninReindentFilterandAlignedIndentFilterused recursion to skipBETWEEN ... ANDpairs. With ~1000 chained pairs (8000 tokens, well underMAX_GROUPING_TOKENS=10000), this exceeded Python's default recursion limit of 1000, causingSQLParseError: Maximum recursion depth exceededviaFilterStack.run()'sRecursionErrorhandler.This means a valid SQL statement with many chained
BETWEEN ... ANDclauses cannot be formatted withsqlparse.format(sql, reindent=True)orsqlparse.format(sql, reindent_aligned=True).PoC
Fix
Convert the recursive
_next_tokencalls to an iterativewhileloop that skips all consecutiveBETWEEN ... ANDpairs without growing the call stack. The iterative version handles chained and nestedBETWEEN ... ANDpairs identically to the original recursive version.Testing
test_format_chained_between_andto verify 1000 chainedBETWEEN ... ANDpairs can be formatted with bothreindent=Trueandreindent_aligned=TrueChecklist