gitcorpus

Live-only GitHub repository discovery for evidence-backed research corpora.

This is a research prototype, not production software. CI runs on pushes to main and on pull requests.

New here? Read The one-paragraph version, then Why this exists and Why not just script the GitHub API?. Those three answer "what is this and do I need it." Everything after that is reference.

Table of contents

• The one-paragraph version
• Why this exists
• Cast of characters (the five things to know)
• The journey, stage by stage
• Why not just script the GitHub API?
• How this relates to existing tools
• What's in the box
• How it's built
• Data model and run artifacts
• Getting it running
• Using gitcorpus
• Testing, linting, and type checking
• Being a good citizen
• Where the project is today
• Glossary
• Documentation map
• License

---

The one-paragraph version

You describe what you want in a filter — an ordinary GitHub search query plus the filters GitHub never built, like "has a Dockerfile" or "owner is in Germany." gitcorpus checks that filter for mistakes before spending a single API call, asks GitHub's live search for matching repositories, follows that search past the 1,000-result wall by slicing the query until every slice fits, saves each candidate's full current details, drops the ones that fail your own filters, resolves owner countries, merges everything by immutable repo id, shows it to you with its provenance, and freezes the whole run into a bundle you can replay, compare, export, and cite. Every stage exists because of a specific GitHub limitation, and this README explains each one where it comes up.

The design principle behind all of it: live-only. A run measures GitHub as it is at that moment (ran_at). There are no background pollers, no stale indexes, no watermarks, and nothing to keep alive between measurements. When you want a corpus, you take a fresh measurement and you keep the evidence.

Why this exists

GitHub's repository search looks like a friendly little API call. It is actually a front door with a very small lobby, and most crawlers find that out the hard way:

• The lobby holds 1,000 people. No single query can return more than 1,000 results, no matter how many pages you ask for. Past result 1,000, GitHub's REST search returns an error (422), not more pages, and gitcorpus's GraphQL discovery is kept under the same ceiling rather than pushed past it. Ranking also considers at most ~4,000 matching repos, so a very broad query can't even see the whole crowd. The only way through is to slice the query into smaller pieces (by creation date, usually) until each piece fits — a technique called sharding.
• There are two prepaid meters, and they are small. Visiting a repo for its full details spends the main meter (5,000 requests/hour) — or the GraphQL points meter (5,000 points/hour) when details are fetched in batches, which is the default. Discovery spends that same points meter now: shard-planning count probes are batched (up to 20 windows per query, 1 point per query) and result pages are one GraphQL search connection per request, so the REST search meter (30 requests/minute) is not touched at all — the since cursor scan and org/user enumeration run on the core REST meter (5,000/hr) instead. On top of those sit GitHub's anti-abuse ("secondary") limits, which punish bursting but never punish steady pacing. Details and the arithmetic live in design/corpus-building-efficient-engineering.md; the measured 67.9-second, 38,969-repo discovery ledger is §9.4.
• All filtering lives inside one text string. GitHub supports a closed set of qualifier:value filters inside q, and nothing else. There is no server-side "has a Dockerfile," no "owner located in Germany," no "at least 100 commits." Those questions have to be answered locally, one repo at a time — after you've paid to fetch the repo.
• Typos fail silently. Type updated:>2024 (the real qualifier is pushed:) and GitHub answers 200 OK with your filter treated as plain search text. You get wrong data and no error. This is why validation is the very first stage of a run.
• The search index lags pushes, and total_count is an estimate. GitHub says so in its docs, and search pagination carries no stability guarantee at all — identical paginated requests can shift or skip items between runs. That's why gitcorpus labels incompleteness instead of hiding it, and why the same filter can return a slightly different set hours later.
• Repos change underneath you. They rename, transfer, get deleted, or go private. Search never tells you which. gitcorpus keys everything on the immutable id, follows renames, and records deletions as tombstones.

A one-off script can answer "show me some Rust repos." It cannot answer "give me the complete set of Rust repos matching this frame as of today, with the Dockerfile, owner-country, and commit-count filters applied, plus the evidence to defend every row." That second question is the one this project exists for.

Cast of characters (the five things to know)

These five concepts recur everywhere, so here they are once, in plain words. No prior GitHub API knowledge is assumed.

• GitHub search (GET /search/repositories, or the GraphQL search connection gitcorpus uses) — the front door. You send one text string (q) with filters inside it (e.g. language:rust stars:>100) and get back thin records: name, description, stars, language, dates, plus a score. It returns at most 100 per page and 1,000 per query, allows ~30 requests/minute per token on REST, silently treats typos as words, and excludes forks unless you ask (fork:true to include them, fork:only for forks alone). gitcorpus reaches it through GraphQL (points meter); the since scan and org/user enumeration use the core REST meter (5,000/hr) — no 1,000-result cap on those endpoints — so the 30/minute search endpoint is no longer used. Reference: findings/01-search-repos-parameters.md.
• Hydration (GET /repos/{owner}/{repo}) — asking for one repo's full current record: topics, license, exact counts, flags, dates, language bytes. Costs one call per repo from the separate 5,000/hour main meter — or, by default, one batched GraphQL request per 20 repos on the points meter. Unchanged repos can be re-fetched almost free via caching headers (ETag). This is not cloning — no files are downloaded.
• Virtual filters — the filters GitHub never built (has_dockerfile, min_commits, owner_country, …). gitcorpus computes them itself, per repo, after hydration, cheapest first.
• Enrichment — extra detail fetched per repo beyond the full record: file trees, language byte breakdowns, releases, funding links, discussions. Each answer costs calls, so enrichment is spent only on repos that survived your filters.
• Run bundle — the saved artifact of one search: your filters, when it ran, what GitHub said, what survived, and the raw upstream responses behind it. It is how a second device replays your search and how a study proves what was measured.

The journey, stage by stage

Here is the whole pipeline at a glance, then each stage with its reason for existing.

filters.json → Validate → Live search → Thin matches → Hydrate → Virtual filters
→ Enrich → Geo resolve → Merge + cache → Display → Bundle/export

Stage 0 — Your filters (a file, not just a form). Everything starts as a filter-spec v1 JSON document: the q string GitHub understands, sort/order/page options, and virtual filters only gitcorpus understands. Because it's a file, it travels between devices; because it holds no tokens, it's safe to share. The console's "Download as JSON" produces exactly this document, and "Upload and run" accepts it.

Stage 1 — Validation (mistakes die here, free). Before any network call, every qualifier is checked against the documented allowlist. updated:>2024-01-01 (doesn't exist — the real qualifier is pushed:) or is:archive (the real one is archived:true) is rejected with a hint, because GitHub would have returned 200 OK with unfiltered results and no error at all. A passing filter set gets a filter_hash fingerprint used for caching and replay. Why this stage exists: GitHub's silent-200 behavior makes typos the most expensive kind of bug — paid in wrong data, not errors.

Stage 2 — Live search (thin matches arrive). The validated q goes to GitHub's GraphQL search. A planner first probes repositoryCount in batches — up to 20 date windows per query, 1 point per query — and bisects every window over the cap until each shard is fetchable. Then a pool of discovery_concurrency workers (32 by default) fetches the shards, one GraphQL search connection per page of 100. Each match is thin; its score is discarded immediately, because that score is relevance relative to that one query and meaningless to store or sort by later. Two honest limits surface here: the 1,000-result cap is unchanged, so broad queries are narrowed by recursive created: bisection rather than forced; and total_count is approximate, so it is displayed as an estimate with an incomplete flag when a timeout cuts a page short.

Stage 3 — Hydration (thin → full, candidates only). Each surviving match gets its full current record — batched by default (up to 20 aliased repos per GraphQL request, with per-repo REST fallback when one node fails). This is the most expensive stage per repo, so two economies apply: unchanged repos revalidate almost free (ETag → 304, which GitHub documents as not counting against the primary limit), and nothing downstream starts for repos that will fail a later filter. A repo renamed since discovery is followed (301); a deleted or privatized one becomes a recorded tombstone, not a crash.

Stage 4 — Virtual filters (gitcorpus's own screening room). Now the filters GitHub can't express run, cheapest first: counts already in the hydrated record (min_stars, team_topic, min_language_bytes/max_language_bytes), then commit counts (min_commits/max_commits), then file checks (has_dockerfile via one recursive file-tree call covering every path question — never one call per file), then owner_country via the geo pipeline. Failures drop with a recorded reason and are counted in the run metadata, so "47 repos matched search, 31 survived filters" is always explainable.

Stage 5 — Enrichment (detail only for survivors). Only repos that pass every filter get enriched: language byte breakdowns, releases, funding links, discussions. The rules that keep this cheap: one file-tree call answers all path questions; GraphQL batches are used only where one query replaces three or more REST calls; identical files across repos are fetched once by content hash; package data comes from zero-token mirrors (ecosyste.ms, deps.dev) before any GitHub call is spent. Nothing is fetched for a display you won't show.

Stage 6 — Geo resolve (free text → country, honestly). Owner location is unstructured human text: "🇩🇪 Berlin", "Lagos", "🌍 remote", often empty. The pipeline decodes flag emojis directly (deterministic, free), normalizes and splits multi-place strings ("Berlin / NYC" → two candidates), resolves cities through an offline gazetteer at zero call cost, catches country-name variants ("USA", "Nederland") with an alias table, and only lets the residue touch a geocoder — cached forever by normalized string. Weak hints (domain endings, timezones) break ties, never lead. Every result stores {country_iso, confidence, raw_location} with an explicit unmatched bucket: an honest unknown beats a confident wrong answer.

Stage 7 — Merge + cache (one result set, many devices). Survivors merge by immutable repo id — never by name, because names change on rename or transfer — so the same filter file run on two laptops produces unionable, dedupe-safe sets. The merged set caches under its filter_hash briefly: an identical search minutes later costs zero GitHub calls.

Stage 8 — Display (cards with provenance). Each row shows its metadata, virtual-filter outcomes, owner country with confidence, and when it was measured (ran_at). Truncation is always declared: capped-by-1,000, timed-out pages, budget stops, plan/page caps, incomplete or retry-exhausted shards, unique under-coverage (collected < ~total_count), tombstones, unresolved hydration, and unenforceable filters appear as labeled warnings, never as silent gaps.

Stage 9 — Bundle and export (the replayable artifact). The run persists its filter hash, ran_at, API version, uniform counts, and the raw upstream JSON per repo. Export downloads it as JSON or CSV. Byte-identical reproduction comes from this bundle — re-running the same file later may drift (stars move, repos vanish), and the bundle is what explains the difference. For research use, this bundle is the replication package's raw layer.

And a closing note on scale: none of the above changes when a run gets big. An exhaustive run is the same live flow with more shards; it is not a second, background pipeline.

Why not just script the GitHub API?

Fair question, and the honest answer starts on your side of the table: for a one-off sample, you should just script it. This is the right tool for "show me some repos":

gh search repos 'language:rust stars:>100' --limit 100 --json fullName,stargazersCount

Here is what quietly changes when the output has to be a complete, defensible corpus:

The script	What happens at scale	What gitcorpus adds
One query, first page(s)	Result 1,001 returns 422; total_count is approximate	Recursive created: bisection inside your own date window (the date token is replaced, never appended — GitHub unions duplicate same-type qualifiers), batched GraphQL count probes, parallel page fetching with inline retry, coverage counters
A straight request loop	30 search req/min; 403/429/422 mean three different things	Per-bucket Redis pacing from response headers; a classifier for retry-after / reset-wait / backoff / fix / fail-loud; multi-token support
No file/geo/commit filters server-side	You fetch thousands of repos to answer one file question	Cost-ordered enrichment: GraphQL file-presence batches, trees API, owner geo resolver, commit counts — cheap screens first, survivors only
owner/name as the key	Renames and transfers silently fork your data	Immutable id primary key, full_name_history, 301 follow, 404 tombstones
"It worked on my machine Tuesday"	You cannot reproduce last week's sample	Filter-spec hash, frozen bundles with raw upstream evidence, replay, run-to-run diff
Crash = start over	A long run fails at hour three	Orphan recovery, Stop/resume (failed, partial, and stopped runs; re-fetches from the start, safely), live phase progress with ETA, a quality report per run
No telemetry	You cannot show what you spent or when	Per-request audit log (rate-limit headers, status, latency, token fingerprint) and SLO dashboards

The honest summary: a script answers a query; gitcorpus produces a corpus you can defend. If you only need the query, stay with the script — genuinely.

How this relates to existing tools

The same question has another form: "why not use the tool that already exists?" The landscape splits into four families, and each is good at its slice:

Family	Examples	What they provide	What they cannot do
Pre-crawled research catalogs	SEART GitHub Search + Data Hub	A queryable database of ~1.9M continuously crawled repos	Fixed columns, a 10-star floor, crawl-time freshness (not yours), no file/geo filters, no per-run evidence
Offline archives at ecosystem scale	World of Code, GH Archive, BigQuery GitHub data	Raw historical objects and events at enormous scale	Monthly snapshots or raw events, not live filtered selection with today's metadata
Analytics dashboards	OSS Insight, DevStats	Aggregated trends and rankings	Aggregate answers, not frozen per-repo corpora
Thin API wrappers	gh search repos, MCP/assistant connectors	Ergonomic pass-through with JSON output	The same 1,000-result cap, no virtual filters, no persistence, no evidence bundle

gitcorpus targets the intersection none of them covers: live measurement at a chosen ran_at, arbitrary filters including file presence and owner location, coverage below the 10-star floor, frozen per-run evidence with raw upstream responses, audited pacing, and coding-agent detection. For a standard off-the-shelf sample, SEART is genuinely the better choice; for ecosystem-scale history, use World of Code. The full honest comparison, including a capability matrix and the case for building rather than adopting, lives in design/landscape-comparison.md and design/gitcorpus-vs-seart.md.

What's in the box

Discovery

• Sharded live search. gitcorpus slices your query by created: date ranges until every shard is under GitHub's 1,000-result cap, then fetches each shard with one GraphQL search connection per page (first: 100, up to max_pages pages) from a pool of discovery_concurrency workers (32 by default). Planning counts are batched GraphQL probes — up to 20 windows per query, 1 point per query — so the whole engine rides the GraphQL points meter (5,000 points/hr) and the REST search meter is not touched. The plan lives inside your own created: window (only a dateless query falls back to 2008→today), and every child replaces the date token rather than appending a second one — GitHub unions duplicate same-type qualifiers, so appending would never let a plan converge. A window that can't be narrowed below the cap even at one day is still fetched and flagged incomplete rather than dropped; a failed page retries inline with backoff, and a shard that still can't finish is marked incomplete instead of aborting the run; a run deadline stops discovery gracefully so the work stays resumable. Every shard records its total_count, fetched count, and state (pending/active/done/incomplete). One accepted cost: discovery upserts rewrite the full row from the lean fields, so a run with max_hydrate < max_candidates can replace previously hydrated detail for repos outside the hydrate window — the bounds of that overwrite are recorded in the discovery spec §3.
• A validation gate with teeth. Every qualifier is checked against the documented allowlist with typo hints; a "delta test" in the test suite proves filtering actually narrows results, so a silent 200 can never masquerade as filtering.
• Org and user enumeration (GET /orgs/{org}/repos, GET /users/{u}/repos) is implemented and tested for single-scope jobs; GET /repositories?since= cursor scanning exists too. Both are currently quarantined from the console (see Where the project is today).

Storage and lifecycle

• Repos are keyed on the immutable id BIGINT, with full_name unique, a full_name_history log for renames, and deleted_at tombstones for deletions and privatizations.
• Revalidation follows 301s and tombstones 404s instead of letting rows vanish or fork.
• Upserts are batched INSERT ... ON CONFLICT DO UPDATE; a COPY-based bulk path exists in the code but is quarantined as unused.

Hydration and enrichment

• Two hydration paths: REST GET /repos/{full_name} with ETag/If-None-Match, and aliased GraphQL batches of up to 20 repos with automatic per-repo REST fallback — so one bad repo never poisons its neighbours. Batches run with bounded concurrency (capped at 20, from limiter_max_concurrent), and the same query carries language bytes and watcher counts, so no extra /languages call is spent.
• Commit counts come from GraphQL defaultBranchRef.target.history.totalCount or REST Link: rel=last.
• File presence is answered by a GraphQL batch first, with the recursive Git Trees API as fallback — never N× contents loops. When a recursive tree is truncated and the path isn't found, the non-recursive root listing is merged in; if presence still can't be proven, the repo is skipped with a warning rather than silently filtered out.
• Owner locations resolve through the offline gazetteer described in Stage 6 and cache per location in geo_cache.
• The cost planner orders all of this record → mirror → single → batched → deep, and the run records per-field source and calls spent.

Virtual filters

All are validated locally with actionable hints; the table says where each is enforced.

Filter	Type	How it is enforced
min_stars	integer ≥ 0	Translated to stars:>=N in the GitHub query
team_topic	topic slug (^[a-z0-9][a-z0-9-]{0,49}$)	Translated to topic:x
has_dockerfile	boolean	File-presence batch (GraphQL → trees fallback)
owner_country	ISO 3166-1 alpha-2 (e.g. DE)	Owner location → geo resolver → confidence tier
min_geo_confidence	exact-iso | name | gazetteer-city | geocoder | weak (default gazetteer-city)	Confidence floor for owner_country
min_commits / max_commits	integer ≥ 0	Hydrated default-branch commit count
min_language_bytes / max_language_bytes	integer ≥ 0	Primary-language byte size carried by the hydration query (no extra call)
min_loc / max_loc	integer ≥ 0	Recorded only — not enforceable yet; a run using them is flagged incomplete

The operator console

Everything is server-rendered Jinja2 with vendored htmx and hand-rolled CSS — no Node, no CDN, no build step at runtime. The pages: dashboard, Find (the filter form, with download-as-JSON, upload-and-run, and save), Searches (run history, live status with phase, done/total, percent bar, ETA, and a ticking elapsed clock, full-width paginated results, Stop, resume, save-filter-to-library, replay, export, clone, compare-with), Corpora (freeze a finished search into a named snapshot), Library (saved filters you can run again), System (status, performance, limits), Settings (run caps), Metrics (SLOs), and Health. Keyboard navigation (/, g h/f/r/l, j/k, Enter, ?, Esc), dark mode, and CSRF-protected forms come standard; page, sort, and page size always live in the URL, so any view is linkable.

Reproducibility

• Filter-spec v1 JSON in, run bundle out: bundle.json, corpus.csv, and persisted run metadata.
• Replay by filter hash, run-scoped export (json/csv), and run-to-run diff (new repos / gone / changed).
• A per-run quality report: count parity, duplicates, null ratios, incompleteness, bundle readability, field coverage.
• An audit log with one row per GitHub response (params, status, rate-limit headers, retry-after, Link, totals, token fingerprint, latency) and an SLO dashboard on top (limiter state, GraphQL points remaining, run counts, incomplete ratio, 422/403+429 rates, p95 latency, geo unmatched rate).

Optional cloning

Fetching never clones. If you want code on disk, the Clone control takes a count (top-N by current sort) and a mode — shallow (--depth 1), file_only (--depth 1 --no-checkout), or windowed (--filter=blob:none --no-checkout) — previews the disk estimate, warns below 2 GiB free, and clones into clones/<filter_hash>/<run_id>/<owner__repo>/. Retries skip finished repos, progress persists, and cancelling is supported. Zero clones is a complete, valid run.

Guardrails

Localhost binding by default, TrustedHost allowlist, Origin/CSRF middleware, and request body caps (1 MiB JSON, 10 MiB uploads). Tokens live only in the environment; logs record fingerprints, not secrets; filter-spec files explicitly reject token-like fields; and startup recovers orphaned runs instead of leaving them spinning.

How it's built

Everything runs from the repo source with PYTHONPATH=src (implicit namespace packages — there is no installed package or console script). The only runnable entry point is python -m serve, and the layout follows the pipeline:

Path	Responsibility
src/lib/	GitHub client (httpx), token loading/fingerprinting, retry orchestration, generic GraphQL batch engine, deadlines, audit records, query tokenizing/qualifier validation
src/discover/	GraphQL search client (batched count probes, parallel page fetching), since cursor scan (quarantined runner), org/user enumeration, discovery pipeline (shard planning/execution, upserts, audit buffering)
src/limiter/	Redis Lua token-window buckets and the throttle classifier
src/scheduler/	created: shard planner, shard state machine + shard store
src/store/	SQLAlchemy models, upserts/dedupe, rename history, lifecycle (tombstones), app settings
src/hydrate/	REST repo hydration (ETag, redirects, commit counts), GraphQL repo adapter, batched refresh driver
src/enrich/	Geo resolver + gazetteer, trees-first file presence, GraphQL file/owner adapters, cost planner, segment executor, repo cloner
src/serve/	FastAPI app and middleware, run runner + background executor, filter-spec parsing/hashing, virtual params, console pages/templates/static, exports, clone registry, library, corpora, diff, quality, metrics, settings
migrations/	Alembic revisions 0001–0013 (11 tables; 0010–0011 add the runs progress columns, 0012 adds app_settings.discovery_concurrency, 0013 raises the batch-size cap)
tests/	unit/, integration/, contract/, golden/, js/ suites
design/, findings/, docs/	Research, frozen specs, plans, environment and legal runbooks (not runtime code)

The stack, in one sentence: Python 3.12+ (PEP 695 syntax is used, so 3.12 is mandatory), FastAPI + Uvicorn, Jinja2 + vendored htmx + hand-rolled CSS, SQLAlchemy 2.x + Alembic on PostgreSQL, Redis 7 (or fakeredis), httpx, and Pydantic. Pinned versions are in requirements.txt and requirements-dev.txt.

Rate limiting, the short version

Pacing is a first-class subsystem here, not a retry loop bolted on:

• Four Redis token buckets — search (30/min; retained, but the shipped paths no longer use it), core (5,000/hr), code_search (10/min), graphql (5,000 pts/hr; one request costs round(connections ÷ 100) points, minimum 1, which is why batching 20 repos at once is far cheaper than 20 requests) — keyed by resource and token fingerprint.
• Response headers are authoritative. X-RateLimit-* values are reconciled into the buckets, and remaining == 0 pauses only the exhausted bucket until its reset.
• Every failure maps to exactly one action: retry_after, wait_reset, backoff (jittered and bounded by the run deadline), shard (the cap-422; the GraphQL planner keeps every shard under the cap, so discovery no longer provokes it), fix (a validation-422), or fail_loud (bad credentials). SSO partial results — a 200 that quietly withholds some orgs — degrade to per-repo fallback instead of aborting the run. No loop without a limit; no silent degradation. Credentialed requests refuse non-GitHub hosts, so a redirect can never leak a token.
• Without Redis the runner falls back to fakeredis with a warning (GITCORPUS_REDIS_STRICT=1 forbids that fallback). Unit tests are hermetic via fakeredis[lua].

Data model and run artifacts

Eleven PostgreSQL tables, across thirteen Alembic revisions:

Table	Purpose
owners	Owner identity (login, type) plus enrichment: location raw, country_iso, geo confidence, company/blog, ETag
repos	Repo metadata mirror keyed on immutable id: names, description, language, license, topics, flags, counts, created/pushed/updated, ETag, deleted_at
full_name_history	Rename/transfer log (repo_id, full_name, seen_at)
geo_cache	Normalized location → country, confidence, hit count
shards	Discovery shard state: kind, GraphQL query, created: ranges, state, totals, fetched, incomplete
audit_log	One row per GitHub response: params, status, rate-limit headers, retry-after, Link, totals, token fingerprint, latency
runs	One row per search: filter hash + spec, status (queued/running/done/partial/failed/cancelled, the last shown as "Stopped"), timestamps, counters, live progress columns (phase, done/total), error, bundle dir
run_items	Frozen per-run result snapshot (repo id, stars, pushed, flags, geo, virtuals)
saved_filters	Named saved filter specs
app_settings	Single-row run limits and toggles (editable at /settings, pinnable by env)
corpora	Named frozen corpora referencing their source run

One run leaves three things on disk, in a directory named by its filter hash and run id:

runs/<filter_hash>/<run_id>/
├── bundle.json          # filter, ran_at, api_version, totals, items[], field_stats
├── corpus.csv           # id,full_name,stargazers,pushed_at,archived,language,license_spdx,country_iso,geo_confidence
└── clone-progress.json  # written only while cloning
clones/<filter_hash>/<run_id>/<owner__repo>/   # one dir per clone, with a done marker

field_stats inside the bundle records hydration statistics, per-field survivors (the filter funnel), per_field_sources, calls_spent, requeues, warnings, and GraphQL batch statistics — so every enriched field can be traced back to how it was obtained. runs/ and clones/ are runtime artifacts and git-ignored.

Getting it running

What you need

Requirement	Notes
Python 3.12+	Mandatory (PEP 695 syntax). CI uses 3.12.
PostgreSQL 17+	The schema needs citext, jsonb, and arrays. docker-compose.yml provides 17.
Redis 7	Strongly recommended. Without it the app degrades to fakeredis (fine for tests, not for real pacing).
GitHub token	Required to crawl. A classic token or a fine-grained PAT with metadata:read over public repos is enough. Never commit it.
git CLI	Optional — only for the clone feature.
Node.js 18+	Optional — only for the JS unit tests (their pytest wrappers skip without it).
Docker	Optional — only if you use the provided Compose services.

Quickstart (Docker Compose)

git clone https://github.com/Thabettt/gitcorpus.git
cd gitcorpus
docker compose up -d          # PostgreSQL 17 on 5432, Redis 7 on 6379

Create and activate a virtual environment:

# Windows (PowerShell)
python -m venv .venv
.venv\Scripts\Activate.ps1

# macOS / Linux
python -m venv .venv
source .venv/bin/activate

Install the pinned dependencies:

pip install -r requirements.txt -r requirements-dev.txt

Set the environment (swap in your own token):

# macOS / Linux
export DATABASE_URL='postgresql+psycopg://gitcorpus:gitcorpus@localhost:5432/gitcorpus'
export REDIS_URL='redis://localhost:6379/0'
export GITHUB_TOKEN='ghp_your_token_here'

# Windows (PowerShell) — session only
$env:DATABASE_URL = 'postgresql+psycopg://gitcorpus:gitcorpus@localhost:5432/gitcorpus'
$env:REDIS_URL = 'redis://localhost:6379/0'
$env:GITHUB_TOKEN = 'ghp_your_token_here'

Apply the migrations:

alembic upgrade head

Start the console:

# Windows (PowerShell)
$env:PYTHONPATH = 'src'
python -m serve

# macOS / Linux
PYTHONPATH=src python -m serve
# equivalent explicit form
PYTHONPATH=src uvicorn serve.app:app --host 127.0.0.1 --port 8000

Open http://127.0.0.1:8000/. The dashboard greets you, and /health tells you plainly whether the database, Redis, and a GitHub token are all present. You're ready to crawl.

Manual / native services

Already run PostgreSQL and Redis, or prefer not to use Docker? Skip Compose and point the environment at your instances:

• DATABASE_URL — any PostgreSQL 17+ database; alembic upgrade head creates everything.
• REDIS_URL — any Redis 7 instance. If it's omitted or unreachable, the app logs a warning and falls back to fakeredis; set GITCORPUS_REDIS_STRICT=1 to make an unreachable Redis a hard failure instead.

Everything else is identical to the quickstart.

Windows local development (project-local services)

The development environment ran from a non-elevated Windows shell, so it deliberately avoids Windows services and Docker Desktop:

• A project-local PostgreSQL cluster (created with the same PG binaries, port 5433, data under %LOCALAPPDATA%\gitcorpus\pgdata) started by pg_ctl, with databases gitcorpus and gitcorpus_test.
• Redis 7 from WSL2 Ubuntu via wsl -u root -- service redis-server start.
• User-level environment variables (DATABASE_URL, TEST_DATABASE_URL, REDIS_URL, GITHUB_TOKEN). One gotcha worth internalising: a process that is already running does not see newly set variables, so restart your terminal or editor after changing them.
• Per-user logon scheduled tasks (gitcorpus-postgres, gitcorpus-redis) start both services automatically.

Every command — creation, start/stop/status, password rotation, troubleshooting — is documented in docs/environment.md. That document also records the deliberate deviation from Compose: docker-compose.yml stays in the repo for environments that do have Docker.

Environment variables

Everything is read from the environment only. Secrets are never stored in the repo or the database.

Variable	Default	Purpose
DATABASE_URL	— (required)	SQLAlchemy DSN, e.g. postgresql+psycopg://user:pw@host:5432/gitcorpus. The app refuses to start without it.
GITHUB_TOKEN	— (required to crawl)	Single GitHub token. Audit logs store only its SHA-256 fingerprint.
GITHUB_TOKENS	—	Comma-separated token list; takes precedence over GITHUB_TOKEN when it parses to ≥ 1 token.
REDIS_URL	fakeredis fallback	Redis DSN for limiter buckets and metrics.
GITCORPUS_REDIS_STRICT	unset	1 disables the fakeredis fallback; an unreachable Redis fails loudly.
GITCORPUS_HOST / GITCORPUS_PORT	127.0.0.1 / 8000	Bind address for python -m serve.
GITCORPUS_ALLOWED_HOSTS	loopback only	Extra comma-separated hosts for TrustedHost (behind a proxy, set the public hostname and run uvicorn with --proxy-headers).
GITCORPUS_REQUEST_DEADLINE_SECONDS	3600	Deadline for a synchronous API/run request; pins the /settings field.
GITCORPUS_CLONE_TIMEOUT_SECONDS	1800	Per-clone timeout.
GITCORPUS_MAX_SHARDS	10	Shard cap per run (pins /settings).
GITCORPUS_MAX_CANDIDATES	500	Candidate cap per run.
GITCORPUS_MAX_HYDRATE	200	Hydration cap per run (must be ≤ candidates).
GITCORPUS_MAX_ENRICH	100	Enrichment-check cap per run.
GITCORPUS_GRAPHQL_BATCH	true	Toggle batched GraphQL (falls back to per-repo REST when off).
GITCORPUS_GRAPHQL_BATCH_SIZE	20	Aliases per GraphQL request (1–50); env overrides are clamped into range at load.
GITCORPUS_MAX_CONCURRENT	10	Max in-flight requests per endpoint/token.
GITCORPUS_DISCOVERY_CONCURRENCY	32	Discovery worker pool size (1–64); one GraphQL search connection per page.
GITCORPUS_DETECT_CONCURRENCY	32	Detection scan workers (1–64); git-source detection runs clone this many repos in parallel. Detection only — searches are unaffected.
GITCORPUS_CLONES_ROOT	clones	Clone root for detection scans. Point this at a non-synced disk (e.g. D:\gitcrawl-clones) so OneDrive/backup watchers do not churn the git objects.
GITCORPUS_WALK_VERIFY_PCT	0	Percentage of order-dependent PR windows additionally verified against git rev-list; off by default, capped at 2 spawns per walk_windows call, and exhausted-before-depth walks are never sampled. Mismatches self-heal and disable the in-process path after 3. Gates/tests set 100.
TEST_DATABASE_URL	—	Test database; its name must end in _test (destructive-safe). Suites skip without it.
GITCORPUS_REQUIRE_TEST_DB	unset	1 turns a missing test database into a failure instead of a skip (used in CI).
UPDATE_GOLDEN	unset	1 regenerates the golden snapshots.

The default run caps are intentionally small — laptop-friendly, interactive. Raise them at System → Limits (/settings) for corpus builds — the Set to corpus-build limits button applies 1,000 shards; 100,000 candidates/hydrate/enrich; a 24 h deadline; batch 20; concurrency 10; discovery workers 32; detection workers 64 — or pin them from the environment; the settings page renders env-pinned fields as read-only.

Database migrations

alembic upgrade head        # run from the repo root, with DATABASE_URL set

One warning worth reading before you point this at a large database: migrations 0005–0007 rewrite constraints and indexes on live tables and take write-blocking locks. On a big database, run them in a maintenance window or convert them to CONCURRENTLY builds as described in docs/environment.md. A downgrade of 0004 can also be lossy for rows with a NULL repo_id; see the migration's docstring.

Using gitcorpus

A first search

1. Start the app and open http://127.0.0.1:8000/.
2. Go to Find (/find): enter a GitHub query (e.g. language:rust stars:>100) and optionally virtual filters such as has_dockerfile or owner_country.
3. Run it. The run page polls live status — phase, done/total, percent bar, phase-relative ETA, and a ticking elapsed clock — with a Stop search button that leaves the run resumable as "Stopped". When it finishes it shows counts (found / saved / passed / unavailable / with Dockerfile), warnings, and partial-result flags.
4. Inspect the full results page, export JSON/CSV, compare against the previous same-filter run, or freeze the result as a corpus.
5. Optional: clone the top-N with the Clone control.

A first run on default limits will intentionally stop after a few hundred candidates and mark itself partial — that's the laptop-friendly default working as designed. Raise the caps on /settings for full-corpus runs.

Filter-spec v1, the document behind every search

{
  "gitcorpus_filter": 1,
  "q": "language:rust stars:>100",
  "sort": "stars",
  "order": "desc",
  "virtual": {
    "has_dockerfile": true,
    "owner_country": "DE",
    "min_commits": 100
  },
  "page": {"per_page": 100, "max_pages": 10}
}

• q — the GitHub query string; validated against the qualifier allowlist (max 256 keyword characters and 5 boolean operators; props.* requires exactly one org: scope).
• sort — one of stars, forks, help-wanted-issues, updated; order — desc or asc (ignored without sort).
• virtual — see the virtual filter table; owner_country implicitly raises min_geo_confidence to its default tier.
• page.per_page — 1–100; page.max_pages — 1–10.
• frame — opaque corpus-frame config recorded verbatim (for study designs); as_of — an ISO-8601 date/datetime recorded with the run.
• The same document is what "Download as JSON" produces on /find and what "Upload and run" accepts. Files are validated before execution and may not contain tokens or local state.
• Canonical serialization of the normalized spec is hashed (SHA-256) into the filter_hash that identifies runs.

JSON API

Examples assume the app is on http://127.0.0.1:8000.

Synchronous search with virtual filters:

curl "http://127.0.0.1:8000/vsearch/repos?q=language:rust%20stars:%3E100&has_dockerfile=true&per_page=5"

Run a full filter spec and get the frozen result body:

curl -X POST http://127.0.0.1:8000/vsearch/run \
  -H 'Content-Type: application/json' \
  -d '{
    "gitcorpus_filter": 1,
    "q": "language:rust stars:>100",
    "sort": "stars",
    "order": "desc",
    "virtual": {"has_dockerfile": true, "min_commits": 100},
    "page": {"per_page": 100, "max_pages": 10}
  }'

The response includes a filter_hash. Replay the latest ready run for that filter, or export it:

curl "http://127.0.0.1:8000/vsearch/runs/<filter_hash>"
curl -OJ "http://127.0.0.1:8000/vsearch/runs/<filter_hash>/export?format=csv"
curl -OJ "http://127.0.0.1:8000/runs/<run_id>/export?format=json"

Unknown parameters are rejected locally with a 400 and a hint — a typo like updated:>2024 never silently becomes a text search. Invalid virtual values are rejected the same way, before any GitHub call.

Console routes, in one table

Path	What it does
/	Dashboard: health badges, quick find, recent runs
/find (/vsearch/ alias)	Filter form: run, download as JSON, upload and run, save
/runs	Run history with status/hash filters and paging
/runs/{id}	Run detail: live phase progress + ETA, Stop, counts, warnings, save-filter-to-library, export/replay/resume/clone, compare-with
/runs/{id}/results	Full-width result table; page, sort, and page size live in the URL
/runs/{id}/diff?against={baseline_id}	Compare with an earlier run (defaults to the previous same-filter run)
/runs/{id}/detect	Detection options for a finished search — trace pack, agents, channels, top-N — with a live estimate; submitting starts a detect run
/runs/{id}/detect-estimate	Calls/time estimate for the detection options form
/partials/runs/{id}/detect-status	Live detect-run progress partial (polled while running)
/corpora, /corpora/{id}	Frozen corpora and their contents
/detections	Detection runs: source search, status, per-agent summary, filterable per-repo evidence, schema-specific exports
/filters	Saved filter library (HTML for browsers, JSON for other clients)
/system	Status / Performance / Limits overview
/settings	Run limits for new runs (env-pinned fields read-only)
/health	DB/Redis/token-present booleans only
/metrics, /api/metrics	SLO dashboard and JSON payload

API routes, in another

Method	Path	Purpose
GET	/vsearch/repos	Synchronous search (q, sort, order, per_page, page, virtuals)
POST	/vsearch/run	Execute a filter spec, return the result body
GET	/vsearch/runs/{filter_hash}	Replay the latest ready run for a filter hash
GET	/vsearch/runs/{filter_hash}/export	Export by filter hash (format=json|csv)
GET	/runs/{run_id}/export	Run-scoped export (format=json|csv, plus detections and detections-json for detect runs; unfinished detect runs answer 409)
GET	/api/runs/{run_id}/diff	Compare a run with a baseline
POST	/runs/{run_id}/cancel	Stop a queued/running run (cooperative; it becomes resumable cancelled)
POST	/runs/{run_id}/resume	Resume a failed, partial, or stopped run (resets artifacts, re-fetches)
POST	/runs/{run_id}/save-filter	Save the run's filter into the library
GET	/runs/{run_id}/quality	Per-run quality report
GET	/runs/{run_id}/clone-estimate	Clone disk estimate preview
POST / DELETE	/runs/{run_id}/clone	Start / cancel cloning the top-N
GET	/filters	Saved filters (HTML or JSON)
POST	/filters/{id}/run	Run a saved filter

Cloning, the optional epilogue

From a finished run, GET /runs/{id}/clone-estimate?limit=N&mode=MODE previews disk usage, POST /runs/{id}/clone starts the clone, and DELETE /runs/{id}/clone cancels it. The three modes:

• shallow — git clone --depth 1 (working tree, last commit)
• file_only — --depth 1 --no-checkout (no working tree)
• windowed — --filter=blob:none --no-checkout (history without blobs)

Clones land in clones/<filter_hash>/<run_id>/<owner__repo>/, and completed directories are skipped on retry.

Testing, linting, and type checking

The suite collects 1,634 tests across five suites. CI runs it on Ubuntu and Windows against real PostgreSQL and Redis (service containers on Linux, native services on Windows).

pytest                                   # all suites; DB-backed suites skip without TEST_DATABASE_URL
pytest tests/unit                        # hermetic unit tests (no services needed)
pytest tests/integration tests/contract  # DB-backed (set TEST_DATABASE_URL)
pytest -q --cov=src --cov-report=term-missing   # coverage gate: fail_under = 93%
node --test tests/js/*.test.mjs          # JS helper tests (optional; pytest wrappers skip without Node)
UPDATE_GOLDEN=1 pytest tests/golden      # regenerate snapshots after intended console/API changes

With a test database (its name must end in _test):

TEST_DATABASE_URL='postgresql+psycopg://gitcorpus:gitcorpus@localhost:5432/gitcorpus_test' \
GITCORPUS_REQUIRE_TEST_DB=1 pytest

$env:TEST_DATABASE_URL = 'postgresql+psycopg://gitcorpus:gitcorpus@localhost:5432/gitcorpus_test'
$env:GITCORPUS_REQUIRE_TEST_DB = '1'
pytest

Lint, format, and types:

ruff check src tests        # rules E, F, I, UP, B; line length 100
black --check src tests     # formatting; black src tests to fix
mypy                        # gated packages: lib, limiter, store, scheduler (ratchet policy in pyproject.toml)

Two pieces of test furniture deserve a sentence each. The golden/ suite snapshots console pages and pins the OpenAPI schema hash, so any unintended API or markup change fails CI until regenerated deliberately. And the coverage gate (fail_under = 93) is a floor, not a target — coverage last measured around 95%.

Being a good citizen

• Localhost by default, no authentication. The console is a single-operator tool. It binds 127.0.0.1 and enforces a TrustedHost allowlist; do not expose it to a network without putting real auth in front and setting GITCORPUS_ALLOWED_HOSTS plus proxy headers.
• CSRF protection via an Origin/Referer check and a per-request token; state-changing bodies are size-capped.
• Tokens are read from the environment only. They are never written to the database, never rendered in the console, and never logged raw — audit records store a SHA-256 fingerprint. Filter-spec files explicitly reject token-like fields.
• Upstream responses stay upstream. API errors are mapped to defined local responses (400/502/503) instead of leaking raw GitHub bodies.
• API-first and ToS-aware. The default path is the official REST/GraphQL API. HTML scraping is off by default and gated on a robots.txt check and legal review. Do not share tokens across accounts to evade rate limits. Deletion/privatization handling and data-retention expectations are documented in docs/legal-gates.md and findings/05-research-gaps.md §A10.
• Attribution and trademark. This project is not affiliated with or endorsed by GitHub. Use a descriptive User-Agent (already set: gitcorpus/0.0.1) and avoid GitHub marks or logos in any published surface.

Where the project is today

Built and wired (verifiable in src/ and in the test suite): GraphQL two-phase live discovery (batched count probes, parallel page fetching from a discovery_concurrency worker pool) with the 1,000-cap planner staying inside the filter's own created: window; immutable-id storage and lifecycle; per-bucket rate limiting with the failure classifier; REST + GraphQL hydration with bounded batch concurrency; cost-ordered virtual enrichment (has_dockerfile, owner_country + confidence, min_commits/max_commits, min_language_bytes/max_language_bytes); the operator console (find/runs/results/diff/corpora/library/system/settings/health); run controls (Stop/resume, save filter, live phase progress with ETA); filter-spec v1 with replay and hashes; bundles and CSV; quality reports; audit log; SLO metrics; and top-N cloning.

Coding-agent detection shipped. A finished search or corpus offers Detect agent use (the nav's Detections page lists runs): choose a frozen trace pack, the agents to match, and any of the eight channels (file, author, branch, label, bot, commit trailer, PR metadata, issue). The run scans the frozen items API-only — no discovery calls, no cloning — storing per-repo evidence, a summary rollup, and exports. The per-repo evidence list is filtered server-side across the whole run (Has signal / Agent / Channel, state in the URL, pagination and counts over the filtered set). Exports unlock when the run finishes, as two actions each offered in CSV or JSON: Export evidence — per stored evidence item, either a flat, analysis-ready CSV row (agent, channel, commit_sha, path, pr_number, login, snippet, … plus the raw evidence_json; the CSV is Excel-friendly with a UTF-8 BOM, one record per physical line, embedded newlines escaped as \n) or a JSON object carrying the raw nested evidence. Export repos — one row per scanned repo in CSV or one object per repo in JSON with corpus fields plus detection rollup (agents, adoption_level, commit_ratio, per-channel counts). The detection detail page and the /detections history list show each run's time elapsed. Depth is windowed (last 100 commits, most recent 100 PRs and issues) and labelled as windowed, never a census; tier boundaries are in docs/detection-tiers.md.

Two selectable methods per run. Detection runs over the GitHub API — the default, unchanged, rate-limited method above — or over a local clone (source: "git"): a lean clone (--depth=N --filter=blob:none), one PR-refs fetch, and local matching, with evidence rows, rollups, and summaries landing in the same tables. The options page presents both with their pros and cons and a size-based estimate for each — API calls/hours; clone MB/minutes for the selected top-N. Git mode uses zero API calls (no token needed), deletes clones after scanning by default (clone.keep: true retains them), and records label/issue as skipped channels — about an hour and about 4 GB for the 17.7k-repo corpus.

Designed but not implemented — the console and docs label these clearly rather than pretending:

• min_loc/max_loc: accepted and recorded, but full-depth LOC enrichment is not wired; using them flags the run incomplete.
• GET /repositories?since= bulk enumeration: the cursor scanner and its tests exist, but the pipeline runner is quarantined (not exposed through the console or API).
• Org/user enumeration (run_org_enum): implemented and tested, not yet reachable from a route.
• Multi-segment enrichment parallelism: the executor supports segments, but the runner currently always uses one.
• Mid-run checkpointing: none — resume re-fetches from the start (upserts make that safe, only slower).

Intentional dead surfaces are tracked in tests/quarantine_manifest.txt and enforced by tests, so unwired code cannot silently rot. Migrations 0010–0011 carry the run progress columns, 0012 adds app_settings.discovery_concurrency, and 0013 raises the batch-size cap to 50.

Known operational constraints: migrations 0005–0007 take write-blocking locks; a run occupies the single executor for its whole duration; the default caps are small; the console has no auth (see above).

Glossary

• Search — one live discovery run. Shown as runs in the console; the noun for one measurement.
• Corpus — a named, immutable snapshot of a finished search.
• Detection — a run that scans a frozen search's items (a corpus) through GitHub's API or a local clone for coding-agent config and trace signals: eight channels, windowed depth, per-repo evidence with run-wide server-side filters and flat evidence/repo CSV exports. See docs/detection-tiers.md.
• ran_at — the UTC moment a search measured GitHub. Every bundle records it.
• Filter spec (v1) — the JSON document that fully describes a search: q, sort, order, virtual, page, optional frame, optional as_of.
• Filter hash — the SHA-256 of the canonical normalized spec; the identity of a search for replay and caching.
• Virtual filter — a condition GitHub search cannot express; some translate to legal qualifiers, others are enforced locally after hydration.
• Hydration — fetching one repo's full current details. Not cloning.
• Touch — one request spent on one repo (to save it or check it); the term comes from the efficiency doc's metering model.
• Sharding — splitting one broad filter into many smaller queries, usually by creation date, so each stays under the 1,000-result cap.
• ETag / 304 — a stored fingerprint for a repo; if nothing changed, GitHub says so for free.
• Run bundle — the frozen artifact of a search: filters, ran_at, counts, item rows, provenance, raw upstream evidence.
• Incomplete — the honest flag on a run that hit a budget, a cap, or an unenforceable filter. Never silently hidden.
• Stopped (cancelled) — a run you ended with the Stop button; it keeps its shards and stays resumable.
• Geo confidence tiers — exact-iso, name, gazetteer-city, geocoder, weak; the ladder behind every owner-country answer.

Documentation map

Document	What it contains
design/spec.md	Frozen feature specification (user stories, requirements, acceptance scenarios)
design/how-the-data-flows.md	The beginner-friendly narrative this README borrows its voice from
design/corpus-building-efficient-engineering.md	The meters, the math of what is findable vs. savable, the legal limits, and the batching engine
design/landscape-comparison.md	Honest tool-by-tool comparison and capability matrix
design/gitcorpus-vs-seart.md	The closest-prior-art comparison, in depth
design/plan.md, design/tasks.md	Stack decisions and task breakdown
design/data-model.md	Schema rationale and indexing notes
design/loc-dilemma.md	How to filter by lines of code without cloning — the LOC dilemma, measured
design/adoption-window.md	When the agent-adoption study starts, and why commits (not repos) are the unit of exclusion
design/run-limits.md	What the Limits page fields mean, the rate-budget math behind them, and the coherent presets
findings/00–06	Deep research on GitHub search limits, parameters, existing solutions, gaps, and the exhaustive capability matrix
docs/development-log.md	What was built, when, with which rulings and verification numbers
docs/environment.md	Windows/local environment: services, credentials, commands, troubleshooting, console manual pass
docs/legal-gates.md	Compliance checklist for crawling, storing, and serving
docs/superpowers/	Design specs, implementation plans, and execution reports for the hardening phases

License

No license has been chosen for this project yet. Until one is added, the code is "all rights reserved" by default: you may view it, but you may not redistribute or reuse it without permission. If you intend to reuse any of this work, open an issue to discuss licensing.