Allow SCIP to contribute symbols for unsupported languages (COBOL on IBM i)

[mi-alto]

Ripwire currently uses SCIP as a precision overlay over symbols already extracted by its built-in grammars.
Would you consider an opt-in mode allowing a SCIP index to contribute definitions and references for files whose language is not natively supported?
This would support proprietary and legacy-language frontends without runtime grammar loading or additional dependencies. The default path would remain byte-identical, deterministic and unchanged when the option is absent.
If this fits Ripwire’s architecture, I would be interested in preparing a small, gate-first implementation.

[joyful-ii-V-I]

Thanks @mi-alto. This is a thoughtful proposal, and the offer to build it gate-first is appreciated. Here is where we stand.

Our plan for public languages is first-class support. SCIP's official indexers cover a finite set:

Indexer	Languages
scip-java	Java, Scala, Kotlin
scip-typescript	TypeScript, JavaScript
rust-analyzer	Rust
scip-clang	C, C++
scip-ruby	Ruby
scip-python	Python
scip-dotnet	C#, Visual Basic
scip-dart	Dart
scip-php	PHP
debian-lsp	Debian packaging files

ripwire already indexes nearly all of these first-class: Java, TypeScript, JavaScript, Rust, C, C++, Ruby, Python, C#, Dart and PHP. Kotlin is in review (#126). Scala and Visual Basic are the remaining gaps, and a Scala starter kit is on its way.

The plan is to support every language on that list first-class, as well as we can: tree-sitter extraction, the full verb set, and red-first gates. For those languages SCIP stays what it is today, a precision overlay (--scip=index.scip) that replaces name-based guesses with exact edges. It won't become a second source of symbols.

We're interested for languages that can't have a public grammar, but not as a replacement for first-class support. That is the case first-class support can never reach: proprietary or legacy languages whose grammars can't ship in ripwire, but whose owners have a frontend that can emit SCIP. Before anyone writes code, two questions:

1. What language or frontend do you have in mind? Real demand should decide this.
2. Would this bar work for you?
   • Opt-in only, and only for files with no built-in grammar. Output is byte-identical when the option is absent.
   • Every SCIP-derived row and header discloses its provenance. Features that need a parse tree (complexity, clones, lint, --match, parse health) are named as unavailable for those files, never silently empty.
   • Deterministic ordering regardless of index order, and a disclosed signal when the index may be stale against the checkout.
   • Gates: a small synthetic .scip fixture for a made-up language, byte-identity without the flag, determinism, and red-first mutation arms.

A question for anyone reading this: which languages, niche or common, would you most like ripwire to support first-class? Reply here with the language and a public repository you'd point it at. It helps decide what comes after Kotlin, Scala and Visual Basic, and a real repository to measure on is the first step of every language port (prompts/add-a-language.md).

None of this is set in stone. It's open for discussion, and a good use case or a good argument could change our minds, so please push back if you see it differently.

Timing-wise, this would come after 0.6.0. We tagged this help wanted earlier; we're removing that for now, because the design needs agreement before anyone picks it up.

[mi-alto]

Thanks @joyful-ii-V-I , that scope sounds reasonable to me.

To give you some context on the use case: we're building proprietary parsers and resolution logic for financial-sector clients working with legacy COBOL codebases. That means dealing with dialect variations, compiler generations, and third-party libraries. Right now we're focused on IBM i COBOL OPM/ILE and DDS.

At the moment we plug our Tree-sitter grammars and language-specific extraction/resolution logic into a fork of Ripwire, then drive Ripwire through MCP to help an agent make sense of undocumented programs and trace their dependencies. What we'd really like is to keep that language-specific tooling separate, without having to maintain invasive changes to Ripwire itself.

Across 17 test sessions, our agent made 317 MCP calls. The native Ripwire tools used were fetch_body, grep, find_symbol, find_referencing_symbols, uses, explore, for, and analyze. The total also includes calls to our own coverage and source-provenance tools.

None of the sessions explicitly used the capabilities you listed as unavailable. Do you think those native MCP tools could be supported for unsupported languages through direct SCIP ingestion? Preserving them would cover the workflows we've tested so far.

We get that this would need a proper exporter on our side and explicit support in the importer, not just importing symbol names. We don't have a SCIP exporter yet; our current integration is direct.

Happy to lock down that scope before writing any code, on the post-0.6.0 timeline you mentioned.

[joyful-ii-V-I]

Thank you for the detail — and especially for the 317-call breakdown. That is the most useful thing anyone has put on this issue, because it turns "which features would be unavailable" from a design argument into a list we can check one at a time.

Short answer: yes. Every verb on your list is a symbol / range / edge question, not a parse-tree question, so all eight can be served from a SCIP-sourced population. Here is the verb-by-verb reading, with what each one actually needs from the index:

MCP verb	What it needs	From SCIP
find_symbol	name → definition site	Occurrence with Definition role + SymbolInformation.display_name
find_referencing_symbols	reference sites, attributed to their enclosing definition	reference occurrences + definition ranges for containment
uses	use-sites with a role (call/read/write/import/extends/type)	symbol_roles bits: Import=0x2, WriteAccess=0x4, ReadAccess=0x8, and Relationship.is_implementation for extends
fetch_body	an exact body range	see the catch below — this is the one that needs care
grep	literal scan + enclosing symbol	scan is bytes, enclosure is definition ranges
for	ranked retrieval: names, bodies, doc text, graph	names + ranges + the reference graph; SymbolInformation.documentation if you emit it
explore / analyze	the map: files, symbols, edges, PageRank	the same three inputs

The one real catch is fetch_body. A definition occurrence's range is the identifier, not the body. The body comes from typed_enclosing_range — single_line_enclosing_range = 10 / multi_line_enclosing_range = 11 in the current proto; the older enclosing_range = 7 is deprecated and superseded by those. If an exporter does not emit one of them, fetch_body has nothing exact to cut, and we are not going to ship a heuristic that guesses where a symbol ends and calls the result a body. So that field is not a nice-to-have on your side; it is the thing that decides whether fetch_body exists at all for those files.

So the scope you would be locking down is roughly this, on the exporter side:

1. Definition occurrences carrying typed_enclosing_range — this is what makes fetch_body real.
2. Reference occurrences with symbol_roles set honestly. This is the part that is better than our native path rather than worse: our own roles are inferred from the tree, and yours would be stated by a resolver that actually knows.
3. SymbolInformation.display_name (6) and kind (5) → the name and kind columns.
4. documentation (3), optional — it is what for matches against besides names and bodies. Without it, ranking still works, on names and bodies alone.
5. relationships (4) with is_implementation, optional — that is what would make lego and the extends role work.

One thing you get for free: the reader already tracks refOccurrences and defsUnmatched and surfaces the ratio, precisely for "this index was built from an older commit than the checkout". That gauge was built for the overlay, and it is the same question in your case.

And the part that genuinely does fall away, so nobody is surprised later: complexity, nesting, clone detection, lint and --match need a tree, and there is no way around that. Worth splitting the quality lens finely, though — churn (git history) and tested (graph reachability) survive a SCIP-sourced population; cx and clone do not. You are right that none of your sessions touched those.

---

The other half of the answer, and I want to put it plainly because it may be the better deal for you.

You are maintaining a fork with grammars, extraction, and resolution logic in it. That is a real recurring cost, and the SCIP path removes only part of it: you would still own the grammars, the extraction, and now a SCIP exporter too — and you permanently give up the parse-derived features for those files.

If a grammar is going to be used repeatedly, compiling it in first-class is a much better deal, and the cost of compiling one in is low. We have done it 24 times. A tree-sitter grammar is a vendored source file and a pinned commit; there is no runtime loading, no plugin ABI, no dependency. And the expensive part of a language port — the extraction rules, the resolution shape, the gates — is dramatically cheaper than it used to be, because AI has moved exactly that kind of work. This is why we would rather add your language than build a side door around it.

Forking an existing grammar to cover a dialect looks tractable for the same reason, though I will be honest that we have added grammars rather than forked one, so that part is an expectation and not experience.

What exists publicly for COBOL today, with the caveats that matter for you:

• yutaro-sakamoto/tree-sitter-cobol — MIT, COBOL85, syntax rules based on opensource-cobol, tested against the NIST COBOL85 suite, and still being pushed to this week. The most credible base to fork.
• Spantree/tree-sitter-cobol-enterprise — parses EXEC CICS and EXEC SQL as typed nodes with named fields rather than opaque text, which is the right shape for tracing dependencies out of mainframe code. Two caveats: it targets IBM Enterprise COBOL on z/OS, which is not your IBM i OPM/ILE dialect, and GitHub reports no recognised licence on it, so it could not be vendored as-is.
• Kathalyst/tree-sitter-cobol-copybook — copybooks, i.e. the COPY edges that are most of what dependency tracing in COBOL actually is.
• Neppord/tree-sitter-cobol — MIT, but archived since 2020.

We found no public tree-sitter grammar for DDS, or for IBM i RPG. If you have built those, they do not exist anywhere else that we can see.

Which means the split might be cleaner than "fork or SCIP". The dialect grammar could be public and vendored here — carried by us, gated by us, no fork for you to rebase. Your resolution logic, coverage and source-provenance tooling, and anything client-specific stays entirely yours, out of this repo. You would stop maintaining a fork of ripwire without giving up fetch_body or the quality lens, and without writing a SCIP exporter at all.

If the grammars themselves cannot be public, then the SCIP path is the right one and the scope above is what we would build to.

So the question back to you is which of those two you actually want — and both can be true, since a COBOL grammar here and a SCIP door for the parts that can never be public are not in conflict. Happy to lock down whichever scope you pick, on the post-0.6.0 timeline.

[mi-alto]

Thanks @joyful-ii-V-I, the verb-by-verb breakdown is exactly what we needed. Knowing that all eight MCP tools used in our tests could work with SCIP-sourced data makes this a promising fit.

We would go with the SCIP path. We're developing our IBM i parsers and analysis tooling as a proprietary component for reuse across client projects, so we're not offering the grammars for upstream inclusion at this stage.

We would, however, be happy to contribute to the generic integration: agreeing on the exporter/importer contract, providing synthetic fixtures and regression tests, and helping validate it against our existing MCP workflows without exposing client material.

Your proposed requirements sound reasonable. In particular, we understand that precise typed_enclosing_range values are essential for fetch_body, and that symbol kinds and reference roles need an agreed mapping. We're also comfortable with explicitly marking the parse-derived capabilities as unavailable.

We don't yet have a SCIP exporter; our current integration is direct. Our acceptance target would be to replay the workflows from those 17 sessions against SCIP-only ingestion and verify that the tools still provide the evidence the agent needs.

What would you suggest as the next step? Should we draft the exporter/importer contract here first, then agree on implementation responsibilities before starting a PR? The post-0.6.0 timeline works for us.

[joyful-ii-V-I]

An update on the COBOL grammar landscape, including three corrections to what I wrote on 11 September. It comes from a measurement we ran this week over 716 public COBOL files: NIST COBOL85, AWS CardDemo, the Open Mainframe Project course samples, Microsoft's sample repositories, and a small IBM i set.

Corrections

• A public IBM i grammar set does exist. barrettotte/treesitter-ibmi (MIT, v0.1.0, released 1 August) ships tree-sitter grammars for IBM i COBOL, DDS, RPG and CL. I wrote that none existed for DDS or RPG. I missed it.
• Spantree's grammar is MIT. GitHub shows no recognised licence only because its LICENSE file carries a second, appended MIT notice. It can be vendored.
• yutaro-sakamoto's grammar is not under active development. The recent push I cited only changed the LICENSE file. The grammar itself last changed in July 2024, and several pull requests are waiting.

What the measurement found

• Scanner bugs. The yutaro-sakamoto grammar and its forks share two external-scanner bugs that affect any tool embedding them:
  • an out-of-bounds read, reported as start_with_word reads past the end of keyword string literals (ASan global-buffer-overflow) yutaro-sakamoto/tree-sitter-cobol#43;
  • an infinite loop when a file's last line is shorter than 6 columns, fixed by the open fix: guard the LINE_PREFIX_COMMENT scan against end of file yutaro-sakamoto/tree-sitter-cobol#42.
• No single grammar gives both clean parses and call edges on programs it was not tuned on.
  • The best statement-level grammar parsed 74% of those files without errors.
  • barrettotte's COBOL grammar parsed 92%, but it has no statement nodes, and its picture token currently splits words such as PERFORM (cobol: picture token splits words starting with a picture character (PERFORM → P + ERFORM), with no ERROR node barrettotte/treesitter-ibmi#1).
• Two grammars, merged by role, close the gap. Definitions come from whichever parse covers a region cleanly, call edges come from the statement-level grammar (or from the other grammar's tokens where that tree failed), and a reference-format pass runs before both. This covered 96.7% of those untuned files fully. Against a lexical reference, PERFORM, CALL and COPY edges came out at 100% recall and precision.

Where this leaves ripwire

• Now: we're working on first-class COBOL for COBOL85 and IBM Enterprise (z/OS), using that two-pass approach.
• Later: IBM i OPM/ILE and DDS.
• Unchanged: for grammars that stay proprietary, the SCIP path discussed above.

[mi-alto]

Thanks @joyful-ii-V-I for the update. I wasn’t aware of barrettotte/treesitter-ibmi I’ll give it a try.

If you could use a hand testing or improving COBOL support against public datasets, I’d be happy to help.

We remain interested in the SCIP ingestion path for our custom language tooling, and I think we’re broadly aligned on the scope and requirements discussed above. I’m happy to fork Ripwire and start working on the contribution.

I see that v0.6.1 is already released and "main" has moved on considerably. Should I base the work on current "main", or would you prefer a specific branch or commit? Is now a good time to start?

[joyful-ii-V-I]

Answering your two questions first, since you are waiting on them.

Base on main. It moved again today — v0.6.2 is tagged, commit 15a20855. That is the base to fork from; there is no special branch for this. And yes, now is a good time to start.

On the offer to help with public datasets — accepted, gratefully, and here is the shape that would help most. Every COBOL number in this thread came from 716 public files (NIST, CardDemo, an OMP course set, IBM/MS/exercism samples). That is not a census, and the conclusions drawn from it are load-bearing for what gets built. So I have written the round up as a prompt anyone can run rather than leaving it as something you would have to reconstruct from comments: #319 adds prompts/cobol-measure-on-your-corpus.md. Paste it at the root of a checkout and it writes a plan and stops before doing anything.

The part where an outside measurement is worth more than ours is the two-pass, two-grammar merge, because the trap in it is not obvious. Merging by best tree per file looks better — about 95% clean files — and collapses PERFORM recall to 54.5%, because barrettotte/treesitter-ibmi trees carry no statement nodes. The configuration that works merges by role: structure from the patched yutaro-sakamoto PR #41 branch, the artifacts it cannot parse from barrettotte, plus token-tier edges read only for paragraphs whose tree failed. Someone re-deriving that independently, on corpora we have never touched, is worth considerably more than us asserting it — and if it does not reproduce, that is the more valuable outcome.

Two warnings before you point anything at real code: every scanner in the yutaro family has an ASan out-of-bounds read in start_with_word (143–171 of 716 files abort), and an infinite loop at EOF when the last line is under six columns — a 1-byte file hangs. Upstream PR #42 fixes the hang and has been unanswered since 07-24; the OOB read is issue #43. Patch both before running a corpus sweep.

The question we cannot answer ourselves is in the prompt twice, deliberately: how often EXEC CICS LINK/XCTL, SQL INCLUDE, COPY REPLACING, nested programs and free format actually appear. Our gap list is guesswork about what real shops use, and it reorders the roadmap directly. Public sample code under-represents CICS and SQL badly, so a public-corpus frequency is a floor rather than an estimate — which is why the prompt asks the same question again in an optional section for anyone with production code they cannot share. That section is aggregate counts only: no file names, no symbol names, no snippets, no paths, and any step that cannot be answered without naming something is skipped and the skip reported. We cannot see that code and do not want to.

On the SCIP ingestion path, one question decides it, and it is worth settling before you write ingestion code. A SCIP definition occurrence's range is the identifier, not the body, so serving fetch_body honestly needs typed_enclosing_range — single_line_enclosing_range = 10 / multi_line_enclosing_range = 11; the old enclosing_range = 7 is deprecated and superseded.

• If your extraction already knows where a symbol's body ends, the SCIP-source path is viable, all eight verbs in your telemetry are servable from it, and the fork can go away.
• If it does not, a first-class vendored grammar is the only honest route, and we should stop designing for SCIP-source on your behalf.

We will not ship a heuristic that guesses where a symbol ends, so "we could approximate it" is a no rather than a maybe. Either answer is useful and the second is not a disappointment — it just changes which work is worth starting.

Last thing, since it affects what you measure against: if you re-run the verb telemetry on v0.6.2, we would like to see whether the mix moved. Your earlier 17 sessions / 317 calls used exactly eight native verbs and none of the parse-tree features, and several of those verbs changed in 0.6.2.

[mi-alto]

Thanks for the detailed protocol. We’ve completed the measurements and attached the report with supporting evidence.

It covers:

• The v0.6.2 setup and upstream checks: 660 passed across the full run and four isolated retests, with two optional pre-change comparisons skipped.
• The role-based combination of the two public grammars on 41 public IBM i files, with independently checked definitions and references, recall separated by tier, construct frequencies, and explicit losses and exclusions.
• The optional production-corpus measurements, reported only as aggregates, without client code or identifiers.
• Symbol-body boundaries: our parser and adapter compute exact boundaries within the tested scope, with 82/82 byte-exact checks passing. This supports producing typed_enclosing_range, but SCIP export and ingestion remain untested.
• The MCP comparison: 17 historical sessions versus 17 sessions on v0.6.2, with counts per operation and the limitations of the comparison.

The package also includes the scanner fix and its regression tests. The README explains the attachments and how to verify them.
We’ve kept the SCIP integration separate from this measurement round.

Does this give you enough evidence to agree on the next step?

cobol-measurement-review-v5.zip