Exploring Neuro-Symbolic AST Parsing for Dynamic Metaprogramming

[aditya-8108]

I have been looking into the limitations of static indexers handling dynamic DSLs and define_method generation. Relying on manual add-ons or strict RBS annotations works for now, but I wanted to explore if we could solve Ruby's dynamic method problem directly at the AST level.

I just built a lightweight Python proof-of-concept called SymbIONet (https://github.com/aditya-8108/symbionet).

It uses a hybrid neuro-symbolic approach:

1. It uses a deterministic static parser for standard syntax.
2. When it detects a dynamic boundary (like a metaprogramming block), it hands the context to a simulated inference engine.
3. The engine deduces the runtime footprint and injects the resulting methods back into the AST as virtual nodes.

Has the LSP team explored secondary inference passes for dynamic boundaries like this, or is the architecture strictly limited to static parsing for performance reasons? I would love to hear your thoughts on this approach as I continue refining the prototype.

[vinistock]

We have not explored anything of the sort yet. Performance is indeed critical as any delays will cause lack of responsiveness in the editor, but I can't judge the proposal as I don't quite understand it yet.

Can you provide an example case and how it would be handled by the engine? Let's say you have this:

class Foo
  LIST_1 = [:bar, :baz]
  LIST_2 = [:qux, *LIST_1]

  LIST_2.each do |name|
    define_method(name) do
      puts "hello"
    end
  end
end

How would you be able to statically understand what is defined here?

[aditya-8108]

Thanks for taking a look and raising a great example! You're completely right that latency is the make-or-break metric for an LSP.

Here is a step-by-step breakdown of how SymbIONet processes this exact snippet, and how performance remains unblocked on the hot path:

---

1. Fast-Path Static Parsing (Synchronous)

• Immediate Response: The deterministic static parser (like Prism or Tree-sitter) parses class Foo on every keystroke. Editor features for standard syntax operate at full native speed.
• Boundary Flagging: When the parser hits LIST_2.each do |name| define_method(name) ... end, the Boundary Detector flags this sub-tree as a Dynamic Boundary because define_method receives a variable (name) rather than a literal symbol.

2. Scope Context Extraction

Instead of re-analyzing the entire project, SymbIONet isolates the localized scope boundary:

• Constant assignments: LIST_1 = [:bar, :baz] and LIST_2 = [:qux, *LIST_1]
• The iteration block: LIST_2.each { |name| define_method(name) ... }

3. Secondary Inference Pass (Async Background Worker)

This step runs asynchronously on a background thread, so it never blocks the editor's main thread or keystroke responsiveness:

• Symbolic / Neural Evaluation: The isolated sub-tree is passed to the inference pass. By tracking local constant evaluation (LIST_1 -> [:bar, :baz] -> LIST_2 -> [:qux, :bar, :baz]), the engine deduces that define_method will generate three runtime signatures: :qux, :bar, and :baz.

4. Virtual Node Injection & Caching

• AST Patching: The engine constructs three virtual definition nodes (DefNode(:qux), DefNode(:bar), DefNode(:baz)) and injects them into Foo's AST scope table.
• LSP Autocomplete: Typing Foo.new. instantly offers qux, bar, and baz in autocomplete without requiring manual RBI/RBS shims.
• Incremental Caching: The result is cached against the hash of that specific AST boundary. Unless LIST_1, LIST_2, or the define_method block is edited, the background pass never re-runs.

---

Summary on Performance

The primary LSP thread remains 100% static and synchronous for instant typing feedback. The inference pass operates strictly as an asynchronous background pass on isolated dynamic boundaries, caching virtual AST nodes incrementally.

Does this boundary-isolation and virtual-node model align with how you think about extending AST scope visibility?