tabgram

Tabulate n-gram matches.

A fast command-line tool for checking occurrence frequency of patterns across a collection of files — paths, names, or any line-based dataset. Give it a list of file paths and it tells you which multi-word patterns appear most often, so you can spot recurring themes, near-duplicates, and naming conventions across tens or hundreds of thousands of files.

Built for the question "I have 100,000 paths — what phrases keep repeating?" — common in file library analysis, course-archive deduplication, content classification, and naming-convention audits.

Written in Go, statically linked, single binary. Small dependency set (golang.org/x/text for Unicode normalization; charmbracelet bubbletea/bubbles for the optional interactive -tui mode). Runs anywhere.

Quick start

# Build for current host
make build

# Find the most common patterns in a list of paths
find /library -type f | ./tabgram

# Restrict to specific file types
find /library -type f | ./tabgram -type ebook,document

# Show longer patterns (4, 5, 6-grams)
./tabgram -n 4,5,6 <your-corpus>.txt

# Browse interactively, drill into which paths contain a pattern
./tabgram -tui <your-corpus>.txt

Documentation

Detailed docs are split by topic:

• Build & install — cross-compile matrix, static linking guarantees, versioned releases.
• Usage & flags — every flag with examples, input modes (stdin/file/-input), output format.
• Examples — recipes for books, movies, courses, naming audits, dedup prep.
• Performance — numbers, memory profile, tuning tips.

When to use tabgram

• Checking occurrence frequency of name patterns in a file collection — "which course titles repeat?", "which release groups dominate this archive?", "which authors show up most?", "which codec/source tags appear together?"
• Naming-convention audits across a file library — find that 80% of paths use a particular date format, that release tags are inconsistent, or that a single course name appears under multiple variant spellings.
• Deduplication prep — spot near-duplicates that differ only by separator (foo-bar vs foo.bar vs foo_bar are merged automatically).
• Content classification of a mixed corpus — once you know which patterns appear most in each find output, you can tag and route.

When NOT to use tabgram

• TB-scale file collections where the working set doesn't fit in RAM — use Count-Min Sketch or Misra-Gries summaries instead.
• Noisy OCR text with thousands of unique character-level patterns — add language-specific tokenization first.
• Need exact phrase boundaries — tabgram collapses variant separators, so "the quick brown fox" and "the.quick.brown.fox" collapse to one pattern by design.
• Binary file contents — tabgram reads paths/filenames, not file bytes.

Project layout

.
├── README.md                 ← this file
├── main.go                   ← CLI entry point
├── stoplist.txt              ← default stoplist (override with -stoplist)
├── Makefile                  ← build, test, cross-compile matrix
├── go.mod / go.sum
├── docs/                     ← detailed documentation
│   ├── build.md
│   ├── usage.md
│   ├── examples.md
│   └── performance.md
├── detector/
│   ├── normalize.go          ← tokenization pipeline
│   ├── stoplist.go           ← stoplist loader
│   ├── detector.go           ← CountMap, n-gram extractors, PMI filter
│   ├── types.go              ← file-type registry
│   └── *_test.go             ← unit tests
└── testdata/
    ├── README.md
    ├── seed.txt              ← 61 hand-crafted paths (correctness)
    ├── big.txt               ← 100k synthetic (gitignored)
    ├── gen/gen.go            ← deterministic generator
    └── private/              ← gitignored — drop your own real data here

License

MIT.