support markdown as a language

ripwire constructs a graph of source files, classes, methods/functions, variables, ...etc
and their relations

markdown can be treated as source files,
with head section as functions
and links as calls (dependencies) for example section Configurations in README.md have [config.ini file format](docs/07_config_ini.md)

although basic section navigation can be done with a skills.md that might say something like

grep -n '^#' *.md docs/*.md
sed -n '<line_from>,<line_to>p; <line_end>q' <filename>

or

find . -name '*.md' -exec grep -Hn '^#' '{}' ';'
sed -n '<line_from>,<line_to>p; <line_end>q' <filename>

but link navigation and interconnections can't be done using that simple grep.

I wonder if someone see a potential in ripwire to facilitate document navigation just like symbol navigation.

@muayyad-alsadi

Thanks @muayyad-alsadi. Document navigation is a good fit for ripwire, and part of it is already there. Here's what exists today and where the gap you describe is.

What ripwire already does with documents

Markdown is indexed alongside the code. Each heading starts a section, and that section's own prose (up to the next heading of any depth) is the unit served.
Other formats are converted to text and indexed the same way (src/docparse.h):
- Jupyter notebooks, HTML and CSV with built-in parsers;
- PDF, DOCX, PPTX and XLSX through the optional markitdown bridge, used only when markitdown is on PATH.
--recall=TASK searches the docs, not the code. It returns the most relevant sections in full, in relevance order, within a token budget, and says what it cut.
--mentions=SYM finds the markdown docs that name a code symbol in backticks: the doc → code link.
--for=TASK surfaces the docs that name the task's top symbols (--no-doc-mention turns that off).
--doc-drift checks which documented claims about the code are now false: file:line references whose file is gone or whose line moved, backticked symbols that no longer exist, = N constants and [N] array sizes.

The gap you describe
Links between documents are not edges yet. [config.ini file format](docs/07_config_ini.md) is just text to ripwire, so there's no "what links to this section", no "what does this section link to", and no path between two docs. Your grep '^#' + sed recipe also points at a missing outline view.

Improvements we could make

Doc links as graph edges. Treat [text](path#anchor) and reference-style links as references from the enclosing section to the target file or heading. The verbs agents already use would then work on docs: --callers as backlinks, --callees as outgoing links, --path between two sections, --impact for "what refers to this page". One detail: the vendored tree-sitter markdown grammar is the block grammar. Inline links need either its companion inline grammar (markdown_inline, same upstream repo, MIT) or a small link scanner over each section's text. Either way, the edges should go through the shared reference path, not a markdown-only mechanism.
Broken-link and dead-anchor checks in --doc-drift: a link whose target file or heading no longer exists is a fifth kind of "documented claim that is now false", next to the four it already checks.
An outline view: each heading of a doc with its line range, so an agent can open just the section it needs (the grep '^#' + sed -n pattern, in one call).
Sections for converted formats. Notebooks, HTML, CSV and markitdown output currently become one whole-file section. HTML headings, and the markdown that markitdown produces from PDF or DOCX, could be split into sections like native markdown, so --recall can serve the relevant part of a long PDF instead of all of it.
Doc → file links: relative links to source files ([the parser](src/docparse.h)) could link a section to that file, so --impact on a file also names the docs that describe it.

Item 1 is the core of your request, and items 2 and 3 follow from it cheaply. If you, or anyone reading, would like to take one, comment here to claim it, and we'll point you at the capture files and gates. Each item is its own small PR.

Doc links as graph edges. Treat text and reference-style links as references from the enclosing section to the target file or heading. The verbs agents already use would then work on docs: --callers as backlinks, --callees as outgoing links, --path between two sections, --impact for "what refers to this page". One detail: the vendored tree-sitter markdown grammar is the block grammar. Inline links need either its companion inline grammar (markdown_inline, same upstream repo, MIT) or a small link scanner over each section's text. Either way, the edges should go through the shared reference path, not a markdown-only mechanism.

yes, please, tell me and I'll try.

the mapping of the example might be like this right?

the caller
file=readm.md, class/section=installation, method/link=config.ini

the callee
file=docs/07_config_ini.md, class/section=document root method/anchor=first head

in ripwire language

p=readme.md, t=sec, n='installation', c=['docs/07_config_ini.md']

I made a WIP that worked like this

alsadi@fedora:ripwire_test$ cat README.md 
# Example

## Installation

see [how to edit config.ini](docs/02_config.md)

alsadi@fedora:ripwire_test$ cat ./docs/02_config.md 
# Config file format

## config.ini sections

### Main section

alsadi@fedora:ripwire_test$ ../ripwire/build/ripwire . --json  | jq -c
{"files":3,"symbols":8,"edges":1,"shown":8,"est_tokens":287,"ambiguous":0,"unresolved":0,"unindexed":{"tag":1},"order":"important-first","root":".","pr_iters":6,"r":[
  {"p":"docs/02_config.md","s":[
    {"t":"sec","n":"02_config","k":0.2189,"c":[]},
    {"t":"sec","n":"Config file format","k":0.1459,"c":[]},
    {"t":"sec","n":"config.ini sections","sc":"Config file format","k":0.1459,"c":[]},
   {"t":"sec","n":"Main section","sc":"config.ini sections","k":0.1459,"c":[]}
   ]
  },
  {"p":"README.md","s":[
    {"t":"sec","n":"Example","k":0.0858,"c":[]},
    {"t":"sec","n":"README","k":0.0858,"c":[]},
    {"t":"sec","n":"Installation","sc":"Example","k":0.0858,"c":[
      {"n":"02_config"}
    ]
  }]},
  {"p":"map.html","s":[
    {"t":"sec","n":"map","k":0.0858,"c":[]}]}]}

it worked like this

$ ../ripwire/build/ripwire . --json --callers='02_config' | jq

{
  "of": "02_config",
  "defs": 1,
  "count": 1,
  "root": ".",
  "hop_tested": 0,
  "hop_untested": 1,
  "graph_ambiguous": 0,
  "graph_unresolved": 0,
  "graph_unindexed": 1,
  "counts_floor": true,
  "callers": [
    {
      "t": "sec",
      "n": "Installation",
      "p": "README.md:3"
    }
  ]
}

$ ../ripwire/build/ripwire . --json --callees='Installation' | jq

{
  "of": "Installation",
  "defs": 1,
  "count": 1,
  "root": ".",
  "hop_tested": 0,
  "hop_untested": 1,
  "graph_ambiguous": 0,
  "graph_unresolved": 0,
  "graph_unindexed": 1,
  "counts_floor": true,
  "callees": [
    {
      "t": "sec",
      "n": "02_config",
      "p": "docs/02_config.md:1"
    }
  ]
}

is this what do we expect?

I got awesome results when I tested searching for documentation related to file formats roadmap in a large project

../ripwire/build/ripwire . --for='file format roadmap' --legend=compact | xmllint --format - | grep '\.md'
../ripwire/build/ripwire . --recall='file format roadmap'

@muayyad-alsadi

@muayyad-alsadi, this made our day, thank you! Hearing that --for and --recall found the right documentation for file format roadmap in a large real project is exactly the kind of feedback we can't get from our own test repos. Doc search on real codebases is what this work is for, and you're the one making it better.

And thank you for today on #414. You fixed the macOS sed test, marked doc links with via="link" on --safe-delete and --impact, and extended the marking to in-file anchors, wikilinks and root-relative links. On top of that, you found and fixed the ripwire $PWD vs ripwire . difference yourself. That's a lot of careful work in one day, and the quality of how you approach this (probing edge cases, explaining what you saw, coming back with fixes) is a pleasure to work with. We're taking a look at the latest head now and will follow up on #414 with whatever, if anything, is still open.

If you're up for it, we'd love to hear more about that large project run: roughly how big it is, which answers were useful, and anything that came back noisy or missing. Real-world reports like this are what tell us where to improve next.

We're genuinely excited about where this is going. Markdown links becoming real, path-exact graph edges opens a lot up, and we'd love to keep the momentum. HTML is the natural next step, and many of the same ideas carry straight over: <a href="page.html#id"> bound to its exact anchor, in-file #id links, dead links and anchors, and via="link" honesty. HTML adds its own edges back into code too: <script src>, event-handler attributes pointing at JS functions, and id/class attributes used by querySelector. We're writing that up as a help wanted issue modeled on your #414. If you'd like to take a slice of it once #414 lands, you'd be more than welcome. Your approach here is exactly the pattern we'd point other contributors to.

If you're up for it, we'd love to hear more about that large project run: roughly how big it is, which answers were useful, and anything that came back noisy or missing. Real-world reports like this are what tell us where to improve next.

large book-like repos

to test on real-world examples, I asked google for projects with many interconnected markdowns
it suggested the following repos

and we might add

https://github.com/jacobian/djangobook.com (.rst not .md)

I tested the first one, here is an example session from zed
it used query and grep

least recently used

I’ll use ripwire to locate the least-recently-used (LRU) cache material in this repository.


Found the **least recently used (LRU) cache** material:

- **Implementation exercise:** [LRU cache notebook](system-design-primer/solutions/object_oriented_design/lru_cache/lru_cache.ipynb), linked from `README.md:364`.
- **System-design example:** `system-design-primer/solutions/system_design/query_cache/README.md:71` — uses LRU to evict entries when the cache reaches capacity.
- **Overview:** `system-design-primer/README.md:1162` — discusses LRU under application caching.

**LRU** evicts the entry that has gone longest without being accessed. A typical implementation combines a hash map with a doubly linked list for **O(1)** lookup, recency updates, and eviction.

key-value store

I’ll use ripwire to find the key-value store overview and related design examples.


Found the **key-value store** material:

- **Overview:** `system-design-primer/README.md:1003` — hash-table abstraction, performance, use cases, and limitations.
- **Design exercise:** [Key-value cache for search queries](system-design-primer/solutions/system_design/query_cache/README.md), linked from `README.md:335`.
- **Caching applications:** `system-design-primer/README.md:1162` — Redis and Memcached, including the connection to LRU eviction.
- **Further design resources:** `system-design-primer/README.md:1658` — implementing a Redis-like store and the Amazon Dynamo paper.

A **key-value store** maps unique keys to values, typically through `get`, `put`, and `delete` operations. It suits caching and simple lookup-heavy workloads; richer queries often require additional application logic.

large code base

Ansible

I also tested the source code of ansible, it has a lot of modules

$ ../ripwire/build/ripwire . --recall='file'
# ...

━━ context/documentation-standards.md  (relevance 0.524) ━━  [sections: 2 of 7, section-granular; whole doc 2574 B; lines="36-46,47-55"]
### Linking

- `M(ansible.builtin.copy)` -- link to a module (FQCN required).
- `P(ansible.builtin.file#lookup)` -- link to a plugin (FQCN and type required).
- `L(title,https://example.com)` -- link with custom title.
- `U(https://example.com)` -- bare URL.
- `R(title,rst_anchor)` -- cross-reference to an RST anchor.

`O()` and `RV()` can also link to options/return values in other plugins using the syntax
`O(ansible.builtin.copy#module:src)`.

but I did not see those when I searched for "copying files"

I was expending those

https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/lineinfile_module.html
https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/copy_module.html

PhotoCraft

I asked about "file format roadmap"

Notes:

navigate seems to be different than --recall=, it uses query or regex ..etc.

@muayyad-alsadi

@muayyad-alsadi, markdown section links are out: ripwire 0.6.8 ships your #414, and the release notes, CHANGELOG and README credit you by name.

A link like [text](other.md#slug) now edges to that heading instead of the whole file, and --callers, --callees, --path, --connect and --impact mark those rows via="link". So "which sections point at this one" and "how do these two docs connect" are one call each, the link navigation you described in this issue that grep can't do.

Two things we tightened while landing it, in case you see them in output:

anchor="missing" now appears only when the heading is provably absent. Non-ASCII headings, percent-encoded slugs, GitHub's -1 duplicate suffix, headings with inline markup and one-line files fall back to the file node with no mark. Thank you for the test that started this.
--json map rows now say "via":"link" for these edges.

Thank you also for the system-design-primer session; that report shaped what we checked. If you get a chance to rerun it on 0.6.8, we'd love to hear whether section-level links help on a book-like repo like that, and anything that comes back noisy or missing.

This issue stays open for the rest of the markdown work. #417 (HTML as a graph) is the sibling you inspired, with slices anyone can claim.

I used route skill for "least recently used" and it highlighted the exact section section but the link itself showed the file without slug

under the hood

$ ripwire --version
ripwire 0.6.8 (Release, GNU 14.2.1, emit=std::print, built_from=unknown)
$ ripwire . --query='least recently used' | xmllint --format - | grep -A3 -B3 -i 'Application caching'
    <s t="sec" n="&#x6B20;&#x70B9;: &#x30AD;&#x30E3;&#x30C3;&#x30B7;&#x30E5;&#x30A2;&#x30B5;&#x30A4;&#x30C9;" sc="&#x30AD;&#x30E3;&#x30C3;&#x30B7;&#x30E5;&#x30A2;&#x30B5;&#x30A4;&#x30C9;" k="0.0000"/>
  </f>
  <f p="README.md">
    <s t="sec" n="Application caching" sc="Cache" k="4.8868"/>
    <s t="sec" n="Object-oriented design interview questions with solutions" sc="The System Design Primer" k="3.5500">
      <c n="Contributing" via="link"/>
    </s>

previous were from cursor and here is from zed

it seems that the numbers are returned but some agents show it other's don't

This issue stays open for the rest of the markdown work. #417 (HTML as a graph) is the sibling you inspired, with slices anyone can claim.

I don't have many html on hand. I'll not pick it

Introducing Light Stemming

but I have another very important idea
ripwire used to work mostly on symbol names, exact matches were enough
now since we are dealing with documents, I was very disappointed from the results from ansilbe repo example,
copying vs. copy

I propose using something as basic as the following two regex matching (can be implemented without regex if needed)

light stemming is like a hash function (I don't care if the remaining is not a proper word) as long as you apply same hash to both sides it will match, stem(query_term)=stem(document_term)
here is a simple function that does the job

const regex_stem1 = /(?:ly)\b/g; 
const regex_stem2 = /(?:ing|ed|es|s)\b/g; // done in two phases as in `interestingly`

function light_stem_en(txt: string) {
    return txt.replace(regex_stem1, "").replace(regex_stem2, "");
}

remove the suffixes ly if any
remove any of the suffixes ing|ed|es|s if any in the remaining

copying → copy
copy → copy

interestingly → interesting → interest

Problems with the regex as written, which the above avoid:

is → i, this → thi, bed → b, sing/ring → s/r, yes → y
-ly removal mangles family → fami, only → on, fly → f
bias, class, bus lose their s

as I said there is no problem as long as you treat it as a hash and apply it to both query and document.
I choose this because it's just enough to experiment

a better version with safe guards on length ..etc.

function light_stem_en(w: string): string {
  if (w.length < 4) return w;
  // -ly (only on longer words)
  if (w.length > 5 && /ly$/.test(w)) w = w.slice(0, -2);
  // -ing / -ed: require a vowel left in the stem and a stem of 3+ chars
  const m = w.match(/^(.{3,}?)(ing|ed)$/);
  if (m && /[aeiouy]/.test(m[1])) w = m[1];
  // Harman S-stemmer
  if (/ies$/.test(w) && !/[ae]ies$/.test(w)) return w.slice(0, -3) + "y";
  if (/es$/.test(w) && !/[aeo]es$/.test(w)) return w.slice(0, -1);
  if (/s$/.test(w) && !/(us|ss)$/.test(w)) return w.slice(0, -1);
  return w;
}

we might pick one of the many existing ones like this simple one from lucene.

https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/java/org/apache/lucene/analysis/en/PorterStemmer.java

and here is a simpler one

https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/java/org/apache/lucene/analysis/en/EnglishMinimalStemmer.java

class EnglishMinimalStemmer {
  @SuppressWarnings("fallthrough")
  int stem(char[] s, int len) {
    if (len < 3 || s[len - 1] != 's') return len;

    switch (s[len - 2]) {
      case 'u':
      case 's':
        return len;
      case 'e':
        if (len > 3 && s[len - 3] == 'i' && s[len - 4] != 'a' && s[len - 4] != 'e') {
          s[len - 3] = 'y';
          return len - 2;
        }
        if (s[len - 3] == 'i' || s[len - 3] == 'a' || s[len - 3] == 'o' || s[len - 3] == 'e')
          return len; /* intentional fallthrough */
      default:
        return len - 1;
    }
  }
}

Condition	Action	Examples
Fewer than 3 characters, or doesn’t end in `s`	Leave unchanged	`is`, `cat`
Ends in `us` or `ss`	Leave unchanged	`bonus`, `class`
Ends in `ies`, has a character before that ending, and that character is neither `a` nor `e`	Replace `ies` with `y`	`ponies` → `pony`, `flies` → `fly`
Ends in `ies`, `aes`, `oes`, or `ees` but didn’t match the previous rule	Leave unchanged	`ties`, `shoes`, `trees`
Any other word ending in `s`	Remove the final `s`	`cats` → `cat`, `houses` → `house`

this one does not cover copying/copy

How?

we start by the most simple one (the two suffixes implemented without regex, if it worked, we might pick the proper next one)
we store the text and the stemmed version
we use bm25 on exact text and on stemmed text
score = p*bm25(exact query in exact document) + (1-p)*bm25(stemmed query in stemmed document)

where p=90% for exact + 10% for stemmed

@muayyad-alsadi

@muayyad-alsadi, thank you for running 0.6.8 on system-design-primer straight away, and for including the exact --query output. That made it easy to reproduce on a small fixture.

What you saw is how the rows are built today, so here is the plain version of why:

A row from --query (and the default map) names a section with two separate attributes: the file in p= (README.md) and the heading text in n= (Application caching, under sc="Cache"). There is no line number and no #slug on those rows. The section link the edge knows about is shown as <c n="..." via="link"/>, by heading name only.
The rows from --callers and --callees are more exact: they carry p=README.md:1162, the file and the line of the heading. That is the location an editor jumps to, and it is what your agent turned into the README.md:1162 links in your Zed session.
Nothing prints the GitHub-style slug (#application-caching). An agent that wants a README.md#application-caching link has to build it from the heading text itself, and that goes wrong easily with duplicate headings (-1), non-ASCII headings like the Japanese ones in that repo, and headings with inline markup.

So the file-only link in your screenshot is the agent using p= as written, not the link edge losing the heading. If it would help, we can look at giving section rows an anchor or line, so agents don't have to guess one. Before we do, two questions so we fix the right thing:

In the Cursor and Zed runs, was the file link built from the --query rows, or from a --callers/--callees/--recall answer? (Zed showing numbers and Cursor not may just be which verb each agent chose.)
When you read the answer, what link would you have wanted to click: README.md#application-caching on GitHub, or README.md:1162 in the editor?

On your light-stemming suggestion (copying vs copy, scoring exact and stemmed terms together): we hadn't answered that yet, sorry. A handful of missed answers in our own measurements do come down to a word form that didn't match, so today we tried it: a light suffix stripper (-ing, -ed, -s …) as an extra ranking signal, on a fixed set of 44 code questions. It didn't pay off as-is. Most stem-only matches were plural forms of the question's ordinary words (files, modes, sessions), and they pulled busy hub files into the top results ahead of right ones. Only one of the five "word form" misses got fixed before correct answers started dropping out. We'll keep the idea and try a narrower version (stemming identifier parts only, not the question's plain words) when we next work on ranking.

Thanks again for testing on a real book-like repo. This is the kind of report that shows where the docs work still needs polish.

Navigation Menu

Repository navigation

support markdown as a language #409

large book-like repos

large code base

Ansible

PhotoCraft

Notes:

Introducing Light Stemming

How?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

support markdown as a language #409

Description

Activity

joyful-ii-V-I commented on Oct 15, 2026

muayyad-alsadi commented on Oct 15, 2026

muayyad-alsadi commented on Oct 16, 2026

muayyad-alsadi commented on Oct 16, 2026

muayyad-alsadi commented on Oct 18, 2026

joyful-ii-V-I commented on Oct 18, 2026

muayyad-alsadi commented on Oct 18, 2026

large book-like repos

large code base

Ansible

PhotoCraft

Notes:

joyful-ii-V-I commented on Oct 20, 2026

muayyad-alsadi commented on Oct 20, 2026

Introducing Light Stemming

How?

joyful-ii-V-I commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions