ANTLR grammar fails to parse tabular subquery on the right of in

[mmaitre314]

The KQL docs mention that this is a valid expression:

StormEvents 
| where State in (
    StormEvents
    | summarize count() by State
    | top 5 by count_
    ) 
| count

https://learn.microsoft.com/en-us/kusto/query/in-cs-operator?view=azure-data-explorer#tabular-expression

The KQL ANTLR grammar in the repo fails to parse it:

line 5:4 mismatched input '|' expecting {')', ','}
line 7:4 mismatched input ')' expecting {<EOF>, ';'}

This seems to be due to this line in Kql.g4:

listEqualityExpression:
    Left=relationalExpression OperatorToken=(IN | NOT_IN | IN_CI | NOT_IN_CI | HAS_ANY | HAS_ALL) '(' Expressions+=invocationExpression (',' Expressions+=invocationExpression)* ')';

which may need to be replaced with something like this (extra pipeExpression):

listEqualityExpression:
      Left=relationalExpression OperatorToken=(IN | NOT_IN | IN_CI | NOT_IN_CI | HAS_ANY | HAS_ALL) '(' Subquery=pipeExpression ')'
    | Left=relationalExpression OperatorToken=(IN | NOT_IN | IN_CI | NOT_IN_CI | HAS_ANY | HAS_ALL) '(' Expressions+=invocationExpression (',' Expressions+=invocationExpression)* ')'
    ;

Full repro:

• Install Java 11: https://learn.microsoft.com/en-us/java/openjdk/download
• Install Python 3.14 https://www.python.org/downloads/
• Run pip install antlr4-python3-runtime==4.13.2
• Download https://repo1.maven.org/maven2/org/antlr/antlr4/4.13.2/antlr4-4.13.2-complete.jar
• Run java -jar antlr4-4.13.2-complete.jar -Dlanguage=Python3 -visitor -o out Kql.g4
• Use the Python KQL parser:

  from antlr4 import InputStream, CommonTokenStream
  from KqlLexer import KqlLexer
  from KqlParser import KqlParser
  
  def parse(query: str):
      parser = KqlParser(CommonTokenStream(KqlLexer(InputStream(query))))
      tree = parser.top()
      print(tree.toStringTree(recog=parser))
  
  parse('''
  StormEvents 
  | where State in (
      StormEvents
      | summarize count() by State
      | top 5 by count_
      ) 
  | count
  ''')

[mattwar]

the KQL grammar file was recovered from an older antlr3 grammer file that was not as current as the actual parser. The current parser solves this with a bit of magic look-ahead.. Solving it by adding the extra alternate with the pipe expression might work, but it might also cause an ambiguity with anltr due to having two branches that could possibly parse the same input in some cases.

[mmaitre314]

Would there be some better solution? I could try a different patch in my project so see what that gives.

[mattwar]

The the approach you already tried successfully built with antlr4 then that’s the correct solution.