diff --git a/CHANGELOG.md b/CHANGELOG.md
index f02fc6d..c8e669e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -13,8 +13,24 @@ across the whole thing, honestly*.
### Added
-- **Generated-marker exclusion (opt-in)** (#114, plan 0002 Phase 2): with
- `CHECKUP_EXCLUDE_GENERATED=1`, files whose head carries a **banner-shaped**
+- **The honest banner — first-party coverage, made visible** (#117, plan 0002
+ Phase 3): `detection.json` now reports first-party coverage **by category**
+ (`coverage.tracked` / `firstParty` / `pctExcluded`, and `excluded.{generated,
+ authorDeclared,convention}`), surfaced on the console and the report headline, so
+ a shrunk scan can never read as "nothing here" — **"% excluded" is itself a
+ signal**. A **single-directory concentration** caveat catches the *markerless*
+ flat-vendored case no marker announces (the dotCMS class): a directory dominated
+ by a language **foreign** to the repo's primary is flagged as vendored-looking —
+ language-aware, so the primary source dir is never mistaken for vendored — and the
+ banner prints the exact one-line fix (`CHECKUP_EXCLUDE='
/*'`). Threshold via
+ `CHECKUP_CONCENTRATION_PCT` (default 25); advisory only, never auto-excluded.
+ `detection.json` schema → **1.5**.
+- **Generated-marker exclusion is now default-on** (#117): the banner-shaped marker
+ exclusion (#114, below) ships **on by default** — announced by a loud banner
+ naming the count plus the enumerated `raw/….generated` list — with the kill-switch
+ `CHECKUP_EXCLUDE_GENERATED=0` to restore the whole tree. The shrink is never silent.
+- **Generated-marker exclusion** (#114, plan 0002 Phase 2): files whose head
+ carries a **banner-shaped**
generated marker (Go `// Code generated … DO NOT EDIT.`, a comment-leader
`@generated`, or C# ``) are dropped from **both** the lizard
inventory and the scc keep-set — the lever that stops a codegen-heavy repo from
@@ -22,8 +38,8 @@ across the whole thing, honestly*.
#105 lizard blow-ups. Markers are case-sensitive and anchored (not the loose
substrings a draft used), so first-party code that merely *quotes* a marker — or
owns a bare `DO NOT EDIT` (IaC, hand-written migrations) — is kept. Every drop is
- **enumerated** to `raw/….generated` (loud, not a silent count). Default off until
- field-tested; surfacing "% generated" as a headline signal is Phase 3.
+ **enumerated** to `raw/….generated` (loud, not a silent count). Default-on since
+ Phase 3 (#117).
- **First-party source — the scc-based engines honour the inventory** (#109,
plan 0002 Phase 1; closes #18): `codebase-stats`, stack-identity, the scc
complexity arm and tech-viability now filter + re-aggregate a single
diff --git a/README.md b/README.md
index f63cedd..3fc3a9a 100644
--- a/README.md
+++ b/README.md
@@ -294,7 +294,9 @@ finding shape and the rest of the substrate carries it through unchanged.
| `CHECKUP_MODE` | closing verdict (`checkup.sh` + renderer) | `tailored` | `tailored` (a repo you own & tune): verdict framed for your own codebase ("where to focus next"); a low score exits non-zero as a quality signal you may act on — not a deploy gate. `audit` (a repo you don't own / due diligence): informational only, framed as "where to invest", **always exits 0**. checkup never gates ([ADR-0009](docs/decisions/0009-deterministic-health-localiser.md)). |
| `CHECKUP_SRC_ROOTS` | complexity + git-axis sections | whole tree (VCS-tracked source) | NARROWS the complexity + git-forensics scan to specific space-separated roots (e.g. `app cmd`). By default checkup assesses **all** VCS-tracked source ([honest coverage](docs/architecture.md)); set this only to focus the scan or speed up a very large monorepo. |
| `CHECKUP_FORENSIC_SINCE` | git-axis sections | `6.months.ago` | `git log --since` window for hotspots / change-coupling / bug-fix-density. Widen (e.g. `2.years.ago`) for repos with sparse recent history; an empty window degrades to `skip`, never a false `pass`. |
-| `CHECKUP_EXCLUDE` | lizard complexity + duplication scans | unset | Extra space-separated fnmatch globs excluded from the lizard scans, on top of the built-in generated/vendored defaults (node_modules, migrations, snapshots, `*.min.*`, …). |
+| `CHECKUP_EXCLUDE` | source inventory (all scanners + scc/identity/stats) | unset | Extra space-separated fnmatch globs excluded from the whole inventory — complexity, duplication **and** the scc-based stats/identity (#109) — on top of the built-in generated/vendored defaults (node_modules, migrations, snapshots, `*.min.*`, …). Also settable as a top-level `exclude:` list in `.checkup.yml`. |
+| `CHECKUP_EXCLUDE_GENERATED` | source inventory (generated-marker pass) | on (set `0` to disable) | Drop files carrying a banner-shaped generated marker (Go `// Code generated … DO NOT EDIT.`, comment-leader `@generated`, C# ``). **Default-on** since v0.2.0; the dropped set is enumerated to `raw/…generated` and announced by a loud banner. `CHECKUP_EXCLUDE_GENERATED=0` keeps the whole tree. |
+| `CHECKUP_CONCENTRATION_PCT` | coverage banner (single-dir concentration) | `25` | Threshold (% of all first-party code) at which one directory is flagged as a possible vendored tree — the banner names the directory and prints the exact `CHECKUP_EXCLUDE` snippet. Advisory only (never auto-excluded). |
| `CHECKUP_SHELL_DIRS` | `shellcheck` section | `scripts .husky .githooks .claude/hooks` | Space-separated dirs to search for shell scripts. Missing dirs are skipped silently. |
| `HADOLINT_DOCKERFILE` | `hadolint` section | auto-detect `Dockerfile*` at root | Override the Dockerfile filename when it is named non-conventionally. |
| `MUTATION_TEST` | `mutation` section | unset (skipped) | Set to `1` to enable Stryker; opt-in because mutation testing is slow (~2 min). |
diff --git a/bin/checkup-report.sh b/bin/checkup-report.sh
index 000c31e..04c8562 100755
--- a/bin/checkup-report.sh
+++ b/bin/checkup-report.sh
@@ -98,6 +98,8 @@ if [ -f "$OUT_DIR/detection.json" ]; then
COVERAGE_TXT=$(jq -r '
.coverage // empty
| "**Coverage:** \(.assessedFiles) source files assessed · scope: \(.scope) · excludes via \(.exclusionSource)"
+ + (if (.excluded.total // 0) > 0 then " · \(.firstParty)/\(.tracked) first-party (\(.pctExcluded)% excluded: \(.excluded.generated) generated, \(.excluded.authorDeclared) author-declared, \(.excluded.convention) convention)" else "" end)
+ + (if .concentration then " · ⚠️ \(.concentration.dir)/ is \(.concentration.pct)% of code (\(.concentration.files) files) \(.concentration.lang) tree vs \(.concentration.repoLang) codebase — exclude if vendored: CHECKUP_EXCLUDE=\u0027\(.concentration.dir)/*\u0027" else "" end)
+ (if .narrowed then " · ⚠️ scope NARROWED by CHECKUP_SRC_ROOTS" else "" end)
+ (if (.unmeasured // []) | length > 0 then " · ⚠️ not measured: " + ((.unmeasured) | join("; ")) else "" end)
' "$OUT_DIR/detection.json" 2>/dev/null)
diff --git a/bin/checkup.sh b/bin/checkup.sh
index 1521ad4..278984d 100755
--- a/bin/checkup.sh
+++ b/bin/checkup.sh
@@ -456,6 +456,29 @@ COVERAGE_NARROWED=false; [ -n "${CHECKUP_SRC_ROOTS:-}" ] && COVERAGE_NARROWED=tr
# can't run, #79) — surfaced so the gap is loud, never a silent false-pass.
COVERAGE_UNMEASURED=$(printf '%s\n' "${CPLX_UNMEASURED[@]}" | jq -R . | jq -s 'map(select(length>0))')
+# Coverage-by-category (#117 Phase 3): attribute every excluded file to a source
+# (convention / author-declared / generated markers) over the ALL-extension keep-set
+# — the same denominator scc/identity use — and surface "% excluded" as a first-class
+# signal. Built by build_scc_keepset; absent on a non-VCS / no-scc target → 0s.
+COVERAGE_TRACKED=${SCC_CANDIDATE_COUNT:-0}
+COVERAGE_FIRSTPARTY=${SCC_KEEP_COUNT:-0}
+COVERAGE_EXC_GENERATED=${GENERATED_EXCLUDED_COUNT:-0}
+COVERAGE_EXC_AUTHOR=${GITATTR_EXCLUDED_COUNT:-0}
+COVERAGE_EXC_CONVENTION=${CONVENTION_EXCLUDED_COUNT:-0}
+COVERAGE_EXC_TOTAL=$(( COVERAGE_EXC_GENERATED + COVERAGE_EXC_AUTHOR + COVERAGE_EXC_CONVENTION ))
+COVERAGE_PCT_EXCLUDED=0
+[ "$COVERAGE_TRACKED" -gt 0 ] && COVERAGE_PCT_EXCLUDED=$(( COVERAGE_EXC_TOTAL * 100 / COVERAGE_TRACKED ))
+
+# Single-directory concentration (#117 Phase 3, plan §6.5): the markerless flat-
+# vendored detector. Re-uses the one scc --by-file walk (ensured during detection,
+# above) filtered to the keep-set; null when scc is absent or no dir dominates.
+COVERAGE_CONCENTRATION=null
+if [ "${SCC_BYFILE_OK:-false}" = true ]; then
+ COVERAGE_CONCENTRATION=$(scc_concentration "$SCC_KEEP_JSON" "${CHECKUP_CONCENTRATION_PCT:-25}" < "$SCC_BYFILE" 2>/dev/null || echo null)
+ [ -z "$COVERAGE_CONCENTRATION" ] && COVERAGE_CONCENTRATION=null
+ echo "$COVERAGE_CONCENTRATION" > "$RAW_DIR/concentration.json"
+fi
+
# Topology (#78): the scan root is a hypothesis. Tell a single package from a
# declared workspace (healthy) from an UNDECLARED fan-out (a thin orchestrator
# root over real packages one level down — a mild structural smell, and the case
@@ -501,17 +524,24 @@ jq -n \
--arg scope "${SOURCE_SCOPE:-unknown}" --arg excl "$COVERAGE_EXCL" \
--argjson byArea "${COVERAGE_BY_AREA:-{\}}" --argjson narrowed "$COVERAGE_NARROWED" \
--argjson unmeasured "${COVERAGE_UNMEASURED:-[]}" \
+ --argjson tracked "$COVERAGE_TRACKED" --argjson firstParty "$COVERAGE_FIRSTPARTY" \
+ --argjson excGen "$COVERAGE_EXC_GENERATED" --argjson excAuthor "$COVERAGE_EXC_AUTHOR" \
+ --argjson excConv "$COVERAGE_EXC_CONVENTION" --argjson excTotal "$COVERAGE_EXC_TOTAL" \
+ --argjson pctExcluded "$COVERAGE_PCT_EXCLUDED" --argjson concentration "$COVERAGE_CONCENTRATION" \
--arg toposhape "$TOPO_SHAPE" --argjson topowstool "$TOPO_WSTOOL_JSON" \
--arg topolock "$TOPO_ROOT_LOCK" --arg toporeal "$TOPO_ROOT_REAL" \
--argjson toporoots "$TOPO_ROOTS_JSON" --argjson topochildren "$TOPO_CHILD_COUNT" \
--argjson topocapped "$TOPO_CAPPED" '
- {schemaVersion:"1.4",
+ {schemaVersion:"1.5",
primary: (if $primary=="" then null else $primary end),
primaryConfidence: $conf,
sccBreakdownAvailable: ($sccok=="true"),
stacks: $stacks, manifests: $manifests,
engines: {complexity:{engine:$ec, reason:$cr, slices:$slices}, duplication:{engine:$ed, reason:$dr}},
- coverage: {assessedFiles:$assessed, scope:$scope, exclusionSource:$excl, narrowed:$narrowed, byArea:$byArea, unmeasured:$unmeasured},
+ coverage: {assessedFiles:$assessed, scope:$scope, exclusionSource:$excl, narrowed:$narrowed, byArea:$byArea, unmeasured:$unmeasured,
+ tracked:$tracked, firstParty:$firstParty, pctExcluded:$pctExcluded,
+ excluded:{generated:$excGen, authorDeclared:$excAuthor, convention:$excConv, total:$excTotal},
+ concentration:$concentration},
topology: {shape:$toposhape, workspaceTool:$topowstool, rootHasLockfile:($topolock=="true"), rootHasRealScripts:($toporeal=="true"), assessmentRoots:$toporoots, childCount:$topochildren, capped:$topocapped},
overridden: ($overridden=="true")}' > "$OUT_DIR/detection.json"
@@ -526,6 +556,22 @@ echo -e " Cross-stack checks always run (secrets, SAST, forensics, stats, docs
COVERAGE_NOTE=" 📐 Coverage: ${SOURCE_FILE_COUNT:-0} source files assessed (scope: ${SOURCE_SCOPE:-unknown}, excludes via ${COVERAGE_EXCL})"
[ "$COVERAGE_NARROWED" = true ] && COVERAGE_NOTE="$COVERAGE_NOTE — NARROWED by CHECKUP_SRC_ROOTS"
echo -e "$COVERAGE_NOTE"
+# First-party split (#117 Phase 3): name what was excluded, by category, so the
+# shrink is never silent. "% excluded" is itself a signal (a codegen-heavy repo).
+if [ "$COVERAGE_EXC_TOTAL" -gt 0 ]; then
+ echo -e " 🧮 First-party: ${COVERAGE_FIRSTPARTY}/${COVERAGE_TRACKED} files (${COVERAGE_PCT_EXCLUDED}% excluded — ${COVERAGE_EXC_GENERATED} generated, ${COVERAGE_EXC_AUTHOR} author-declared, ${COVERAGE_EXC_CONVENTION} convention)"
+fi
+# Loud generated banner (#117 §10): default-on exclusion must announce itself + the
+# enumerated list + the kill-switch, so a reader never mistakes the shrink for the truth.
+if [ "${COVERAGE_EXC_GENERATED:-0}" -gt 0 ]; then
+ echo -e " ${YELLOW}🤖 Excluded ${COVERAGE_EXC_GENERATED} generated file(s)${NC} (banner-shaped markers) — enumerated in raw/scc-keep.lst.generated · keep them with CHECKUP_EXCLUDE_GENERATED=0"
+fi
+# Single-directory concentration caveat (#117 §6.5): a dir dominated by a language
+# foreign to the repo's primary — looks vendored. Name it + the one-line fix.
+if [ "$COVERAGE_CONCENTRATION" != "null" ]; then
+ read -r CONC_DIR CONC_PCT CONC_FILES CONC_LANG CONC_REPOLANG < <(echo "$COVERAGE_CONCENTRATION" | jq -r '"\(.dir) \(.pct) \(.files) \(.lang) \(.repoLang)"')
+ echo -e " ${YELLOW}📦 Concentration:${NC} ${CONC_DIR}/ is ${CONC_PCT}% of all code (${CONC_FILES} files, mostly ${CONC_LANG}; the codebase is mostly ${CONC_REPOLANG}) — looks vendored. To exclude: CHECKUP_EXCLUDE='${CONC_DIR}/*' (or add to .checkup.yml exclude:)"
+fi
[ "${#CPLX_UNMEASURED[@]}" -gt 0 ] && echo -e " ${YELLOW}⚠️ Not measured:${NC} ${CPLX_UNMEASURED[*]}"
case "$TOPO_SHAPE" in
undeclared-fan-out) echo -e " ${YELLOW}🧩 Topology:${NC} undeclared fan-out — ${TOPO_CHILD_COUNT} sub-package(s) below an orchestrator root (${TOPO_ASSESSMENT_ROOTS[*]}); the root scan can't see them" ;;
diff --git a/docs/architecture.md b/docs/architecture.md
index 82f8c6f..95c510b 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -131,7 +131,7 @@ The plan is printed for a human and persisted to `detection.json` (in `OUT_DIR`,
```jsonc
{
- "schemaVersion": "1.4",
+ "schemaVersion": "1.5",
"primary": "node", // largest stack, or null when ambiguous
"primaryConfidence": "high", // high (manifest + dominant) | medium | low
"sccBreakdownAvailable": true, // false → degraded to manifest/presence signal
@@ -152,10 +152,21 @@ The plan is printed for a human and persisted to `detection.json` (in `OUT_DIR`,
// dir; narrowed = true when CHECKUP_SRC_ROOTS restricted the scope; unmeasured
// names what a routed engine could NOT cover (e.g. JS/TS complexity when no
// resolvable ESLint config — #79), so the gap is explicit, not a false pass.
+ // tracked/firstParty/excluded/pctExcluded (#117, schema 1.5) attribute every
+ // excluded file to a category (generated markers | author-declared .gitattributes
+ // | convention globs) over the ALL-extension keep-set scc/identity use — "%
+ // excluded" is itself a signal. concentration (plan §6.5) names a single
+ // directory that dominates the first-party code (a possible markerless vendored
+ // tree) + the one-line fix; language-aware (a dir dominated by a language foreign
+ // to the repo's primary), so the source dir itself is never mistaken for vendored;
+ // null when scc is absent or no foreign dir dominates.
"coverage": {
"assessedFiles": 412, "scope": "git", "exclusionSource": ".gitignore",
"narrowed": false, "byArea": { "src": 280, "server": 110, "scripts": 22 },
- "unmeasured": [] // e.g. ["JS/TS complexity (no resolvable root ESLint config)"]
+ "unmeasured": [], // e.g. ["JS/TS complexity (no resolvable root ESLint config)"]
+ "tracked": 4820, "firstParty": 470, "pctExcluded": 90,
+ "excluded": { "generated": 4100, "authorDeclared": 180, "convention": 70, "total": 4350 },
+ "concentration": null // or { "dir": "webapp/html/js", "code", "files", "totalCode", "pct": 25, "lang": "JavaScript", "repoLang": "Java" }
},
// Package topology (#78): the scan root is a hypothesis. shape distinguishes a
// single package from a declared workspace (healthy) from an UNDECLARED fan-out
diff --git a/lib/scc-concentration.jq b/lib/scc-concentration.jq
new file mode 100644
index 0000000..6afe62b
--- /dev/null
+++ b/lib/scc-concentration.jq
@@ -0,0 +1,65 @@
+# scc-concentration.jq — single-directory concentration caveat (plan 0002 §6.5,
+# #117 Phase 3). The detector for MARKERLESS flat-vendored code that no generated
+# marker announces and no convention dir catches (the dotCMS class: a committed
+# JS library tree sitting in the first-party keep-set, skewing identity). Too fuzzy
+# to auto-exclude — surfaced as a BANNER that names the directory and prints the
+# one-line fix, never a silent drop.
+#
+# LANGUAGE-AWARE, not raw share: the spike (§9) showed the tell is a dir dominated
+# by a language FOREIGN to the repo's primary (dotCMS: webapp/html/js is JavaScript
+# while the codebase is Java). Pure share alone flags the primary SOURCE directory
+# on any normal repo (it is the largest dir) — a false positive that would fire
+# everywhere. So a directory is a candidate only when its dominant language differs
+# from the repo's dominant language: that is what "vendored-looking" means here.
+#
+# Input: scc --by-file --format json (array of language objects, each .Files[]).
+# Args: --slurpfile keep (JSON array of TARGET-relative keep paths,
+# ALL extensions — same set as scc-aggregate)
+# --argjson pct (threshold percent of total code, e.g. 25)
+# Output: the most-SPECIFIC (depth-maximal) directory whose share of the kept code
+# is ≥ pct AND whose dominant language ≠ the repo's, as
+# { dir, code, files, totalCode, pct, lang, repoLang }, or null.
+#
+# Depth-maximal, not share-maximal: a vendored tree's own subdirs each fall below
+# the threshold while the tree's own root is the deepest prefix still above it — so
+# "deepest ≥ threshold" names the vendored ROOT (e.g. .../webapp/html/js), the most
+# precise exclude path, rather than a shallow ancestor that also sweeps in siblings.
+# Deterministic (#96): sort by [depth, share, dir], take the last.
+
+( ($keep[0] // []) | map({ key: sub("^\\./"; ""), value: true }) | from_entries ) as $k
+| [ .[].Files[]?
+ | { loc: (.Location | sub("^\\./"; "")), code: (.Code // 0), lang: (.Language // "") }
+ | select( $k[.loc] // false ) ] as $files
+| ( $files | map(.code) | add // 0 ) as $total
+| if $total <= 0 then null
+ else
+ # The repo's dominant language (by kept code) — the baseline a candidate dir
+ # must differ from to read as "vendored-looking".
+ ( $files | group_by(.lang)
+ | map({ lang: .[0].lang, code: (map(.code) | add) })
+ | sort_by([ -.code, .lang ]) | .[0].lang ) as $repoLang
+ | ( $files
+ | map( . as $f
+ | ($f.loc | split("/")) as $parts
+ | ($parts[0:-1]) as $dirs # drop the filename
+ | [ range(1; ($dirs|length)+1) as $n
+ | { dir: ($dirs[0:$n] | join("/")), code: $f.code, lang: $f.lang } ] )
+ | add // []
+ | group_by(.dir)
+ | map( { dir: .[0].dir,
+ code: (map(.code) | add),
+ files: length,
+ share: ((map(.code) | add) / $total),
+ # this dir's own dominant language
+ lang: ( group_by(.lang)
+ | map({ lang: .[0].lang, code: (map(.code) | add) })
+ | sort_by([ -.code, .lang ]) | .[0].lang ) } )
+ | map( select( .share >= ($pct / 100) and .files >= 2 and .lang != $repoLang ) )
+ | sort_by( [ (.dir | split("/") | length), .share, .dir ] )
+ | last ) as $top
+ | if $top == null then null
+ else { dir: $top.dir, code: $top.code, files: $top.files,
+ totalCode: $total, pct: (($top.share * 100) | floor),
+ lang: $top.lang, repoLang: $repoLang }
+ end
+ end
diff --git a/lib/scc-inventory.sh b/lib/scc-inventory.sh
index 7ba9cc2..9396a79 100644
--- a/lib/scc-inventory.sh
+++ b/lib/scc-inventory.sh
@@ -77,6 +77,20 @@ scc_breakdown_toplangs() {
.[0:$n] | map("\(.Name) \(.Code)") | join(", ")'
}
+# scc_concentration [pct]
+# stdin: scc --by-file --format json
+# stdout: the single-directory concentration finding (plan 0002 §6.5, #117) as
+# { dir, code, files, totalCode, pct, lang, repoLang }, or "null" when no
+# kept directory holds ≥ pct (default 25) of the kept code IN A LANGUAGE
+# FOREIGN to the repo's primary. Language-aware so the primary source dir
+# (the largest dir on any normal repo) is not mistaken for a vendored tree
+# — the markerless-flat-vendored detector (the dotCMS class). Advisory
+# banner input. Pure transform in lib/scc-concentration.jq (unit-tested).
+scc_concentration() {
+ local keepfile="$1" pct="${2:-25}"
+ jq --slurpfile keep "$keepfile" --argjson pct "$pct" -f "$CHECKUP_HOME/lib/scc-concentration.jq"
+}
+
# scc_perfile_findings
# stdin: scc --by-file --format json
# stdout: complexity findings array (file/line/ccn/lines/code/severity/message),
diff --git a/lib/source-inventory.sh b/lib/source-inventory.sh
index 53871c3..f2b9713 100644
--- a/lib/source-inventory.sh
+++ b/lib/source-inventory.sh
@@ -168,6 +168,7 @@ _apply_gitattr_excludes() {
while IFS= read -r -d '' p && IFS= read -r -d '' a && IFS= read -r -d '' v; do
case "$v" in set|true) _flagged["$p"]=1 ;; esac
done < <(git check-attr -z --stdin linguist-generated linguist-vendored < "$lst" 2>/dev/null)
+ GITATTR_EXCLUDED_COUNT="${#_flagged[@]}" # author-declared (coverage signal, #117)
[ "${#_flagged[@]}" -eq 0 ] && return 0
local out="$lst.attr"
while IFS= read -r -d '' p; do
@@ -178,15 +179,18 @@ _apply_gitattr_excludes() {
}
# Drop files whose head carries a BANNER-SHAPED generated marker (#114, plan 0002
-# Phase 2) — the corvus lever. OPT-IN via CHECKUP_EXCLUDE_GENERATED (default off)
-# so the regex is field-tested with zero default-behaviour change before any flip
-# to default-on. Rewrites the NUL list at $1 in place (order-preserving) and
-# ENUMERATES the dropped files to ".generated" (loud, not a silent count —
-# the guard that earns the subtractive design; over-exclusion is the danger).
-# Sets GENERATED_EXCLUDED_COUNT. Runs with cwd == TARGET so the relative paths open.
+# Phase 2) — the corvus lever. DEFAULT-ON since Phase 3 (#117): the markers are
+# field-proven and the shrink is announced by a loud banner, so excluding generated
+# code is the honest default. KILL-SWITCH: `CHECKUP_EXCLUDE_GENERATED=0` (also
+# false/no/off) restores the whole tree. Rewrites the NUL list at $1 in place
+# (order-preserving) and ENUMERATES the dropped files to ".generated" (loud,
+# not a silent count — the guard that earns the subtractive design; over-exclusion
+# is the danger). Sets GENERATED_EXCLUDED_COUNT. Runs with cwd == TARGET so the
+# relative paths open.
_apply_generated_excludes() {
local lst="$1"
- [ -n "${CHECKUP_EXCLUDE_GENERATED:-}" ] && [ -s "$lst" ] || return 0
+ case "${CHECKUP_EXCLUDE_GENERATED:-1}" in 0|false|no|off) return 0 ;; esac
+ [ -s "$lst" ] || return 0
local flagged="$lst.generated"
# grep -l short-circuits on the first match per file (the marker is at the head),
# so even a mostly-generated tree is cheap; -I skips binaries; LC_ALL=C makes the
@@ -209,20 +213,38 @@ _apply_generated_excludes() {
# engines filter their --by-file output against this (lib/scc-inventory.sh), so
# stats/identity reflect first-party code, not the whole tree (#109). Same tiers
# as build_source_inventory; run AFTER resolve_scan_roots, cwd == TARGET.
+#
+# Subtraction is STAGED so the coverage signal (#117 Phase 3) can attribute every
+# excluded file to a category — convention (builtin globs + CHECKUP_EXCLUDE),
+# author-declared (.gitattributes), generated (markers) — computed from the count
+# delta at each stage (robust to the helpers' internals). Sets, over the ALL-
+# extension candidate set: SCC_CANDIDATE_COUNT (everything tracked), SCC_KEEP_COUNT
+# (kept), CONVENTION_EXCLUDED_COUNT, GITATTR_EXCLUDED_COUNT, GENERATED_EXCLUDED_COUNT.
build_scc_keepset() {
- local lst="$RAW_DIR/scc-keep.lst"
+ local lst="$RAW_DIR/scc-keep.lst" raw="$RAW_DIR/scc-keep.raw"
SCC_KEEP_JSON="$RAW_DIR/scc-keep.json"
mkdir -p "$RAW_DIR"
+ # Enumerate ALL tracked candidates first (every extension), so each subtraction
+ # stage below has an honest denominator.
if [ "${GIT_OK:-false}" = true ]; then
- git ls-files -z -- "${SCAN_ROOTS[@]}" 2>/dev/null | _filter_keep > "$lst"
+ git ls-files -z -- "${SCAN_ROOTS[@]}" 2>/dev/null > "$raw"
elif command -v fd > /dev/null 2>&1; then
- fd --type f --hidden --no-follow --print0 . "${SCAN_ROOTS[@]}" 2>/dev/null | _filter_keep > "$lst"
+ fd --type f --hidden --no-follow --print0 . "${SCAN_ROOTS[@]}" 2>/dev/null > "$raw"
else
find "${SCAN_ROOTS[@]}" \( -name node_modules -o -name .git \) -prune -o \
- -type f -print0 2>/dev/null | _filter_keep > "$lst"
+ -type f -print0 2>/dev/null > "$raw"
fi
+ SCC_CANDIDATE_COUNT=$(tr -cd '\0' < "$raw" | wc -c | tr -d ' ')
+ _filter_keep < "$raw" > "$lst" # convention globs + CHECKUP_EXCLUDE
+ rm -f "$raw"
+ local after_conv; after_conv=$(tr -cd '\0' < "$lst" | wc -c | tr -d ' ')
+ CONVENTION_EXCLUDED_COUNT=$(( SCC_CANDIDATE_COUNT - after_conv ))
_apply_gitattr_excludes "$lst" # author-declared generated/vendored (.gitattributes)
- _apply_generated_excludes "$lst" # generated-file markers (#114, opt-in CHECKUP_EXCLUDE_GENERATED)
+ local after_attr; after_attr=$(tr -cd '\0' < "$lst" | wc -c | tr -d ' ')
+ GITATTR_EXCLUDED_COUNT=$(( after_conv - after_attr ))
+ _apply_generated_excludes "$lst" # generated-file markers (#114, default-on; CHECKUP_EXCLUDE_GENERATED=0 disables)
+ SCC_KEEP_COUNT=$(tr -cd '\0' < "$lst" | wc -c | tr -d ' ')
+ GENERATED_EXCLUDED_COUNT=$(( after_attr - SCC_KEEP_COUNT ))
jq -Rs 'split("\u0000") | map(select(length > 0))' < "$lst" > "$SCC_KEEP_JSON" 2>/dev/null \
|| printf '[]' > "$SCC_KEEP_JSON"
return 0
diff --git a/test/scc-inventory.test.sh b/test/scc-inventory.test.sh
index bf5f561..c09e172 100644
--- a/test/scc-inventory.test.sh
+++ b/test/scc-inventory.test.sh
@@ -124,6 +124,56 @@ jq '[ .[] | {Name, Files: (.Files|reverse)} ] | reverse' "$BYFILE" > "$TMP/shuff
B2=$(scc_breakdown "$KEEP" < "$TMP/shuffled.json")
assert_eq "breakdown stable under input reorder" "$(printf '%s' "$BREAKDOWN" | jq -S -c '.')" "$(printf '%s' "$B2" | jq -S -c '.')"
+echo ""
+echo "single-dir concentration (§6.5, #117): foreign-language dir, depth-maximal, deterministic"
+# The repo is mostly C# (the real code under src/); a markerless flat-vendored JS
+# tree sits under webapp/html/js (still in the keep-set — no marker/convention
+# catches it). The detector must name the FOREIGN-language tree, not the larger
+# primary C# dir (the false positive that would fire on every normal repo).
+CONC_BYFILE="$TMP/conc.json"
+cat > "$CONC_BYFILE" <<'JSON'
+[
+ {"Name":"C#","Files":[
+ {"Location":"src/A.cs","Language":"C#","Code":500,"Complexity":50,"Lines":600},
+ {"Location":"src/B.cs","Language":"C#","Code":500,"Complexity":50,"Lines":600}
+ ]},
+ {"Name":"JavaScript","Files":[
+ {"Location":"./webapp/html/js/a.js","Language":"JavaScript","Code":200,"Complexity":5,"Lines":2000},
+ {"Location":"webapp/html/js/b.js","Language":"JavaScript","Code":200,"Complexity":5,"Lines":2000}
+ ]}
+]
+JSON
+CONC_KEEP="$TMP/conc-keep.json"
+echo '["src/A.cs","src/B.cs","webapp/html/js/a.js","webapp/html/js/b.js"]' > "$CONC_KEEP"
+
+CONC=$(scc_concentration "$CONC_KEEP" 25 < "$CONC_BYFILE")
+# 400/1400 = 28% under webapp/html/js; JS ≠ C# (repo primary) → flagged. src (71%,
+# C# = primary) is NOT flagged despite being larger.
+assert_eq "names the FOREIGN-language tree, not the bigger primary dir" "webapp/html/js" "$(printf '%s' "$CONC" | jq -r '.dir')"
+assert_eq "share floored to pct" "28" "$(printf '%s' "$CONC" | jq -r '.pct')"
+assert_eq "file count under the dir" "2" "$(printf '%s' "$CONC" | jq -r '.files')"
+assert_eq "totalCode = kept code" "1400" "$(printf '%s' "$CONC" | jq -r '.totalCode')"
+assert_eq "reports the foreign language" "JavaScript" "$(printf '%s' "$CONC" | jq -r '.lang')"
+assert_eq "reports the repo's language" "C#" "$(printf '%s' "$CONC" | jq -r '.repoLang')"
+
+echo ""
+echo "concentration: same-language large dir is NOT flagged (no primary-dir false positive)"
+# Keep ONLY the C# — now src is 100% C# = repoLang → nothing foreign → null.
+echo '["src/A.cs","src/B.cs"]' > "$TMP/conc-cs.json"
+assert_eq "all-primary-language repo → null" "null" "$(scc_concentration "$TMP/conc-cs.json" 25 < "$CONC_BYFILE")"
+
+echo ""
+echo "concentration: threshold respected + degrade cases"
+assert_eq "pct above share → null" "null" "$(scc_concentration "$CONC_KEEP" 40 < "$CONC_BYFILE")"
+assert_eq "empty keep → null" "null" "$(scc_concentration "$TMP/empty.json" 25 < "$CONC_BYFILE")"
+
+echo ""
+echo "concentration: deterministic under input reorder (#96)"
+jq '[ .[] | {Name, Files:(.Files|reverse)} ] | reverse' "$CONC_BYFILE" > "$TMP/conc-shuf.json"
+assert_eq "stable under reorder" \
+ "$(scc_concentration "$CONC_KEEP" 25 < "$CONC_BYFILE" | jq -S -c '.')" \
+ "$(scc_concentration "$CONC_KEEP" 25 < "$TMP/conc-shuf.json" | jq -S -c '.')"
+
echo ""
echo "Results: $PASS passed, $FAIL failed"
[ "$FAIL" -eq 0 ]
diff --git a/test/source-inventory.test.sh b/test/source-inventory.test.sh
index 274cf4e..82768eb 100755
--- a/test/source-inventory.test.sh
+++ b/test/source-inventory.test.sh
@@ -156,7 +156,7 @@ else
fi
echo ""
-echo "generated-marker exclusion: opt-in, banner-shaped, keeps marker-quoting first-party (#114)"
+echo "generated-marker exclusion: default-on (#117), banner-shaped, keeps marker-quoting first-party (#114)"
if command -v git > /dev/null 2>&1; then
G="$HOME/.checkup-gen.$$"; rm -rf "$G"; mkdir -p "$G/gen" "$G/src"
( cd "$G"
@@ -168,25 +168,54 @@ if command -v git > /dev/null 2>&1; then
printf '// This documents the @generated convention\npackage y\n' > src/notes.go # prose → keep
printf '# DO NOT EDIT - managed by our own tool\nx=1\n' > src/migrate.py # bare DO NOT EDIT → keep
git add -A && git commit -qm init >/dev/null 2>&1 )
- # Opt-in OFF (default): nothing dropped on markers.
- off=$( cd "$G"; GIT_OK=true; RAW_DIR="$G/raw"; SCAN_ROOTS=(.); build_source_inventory; tr '\0' '\n' < "$SOURCE_LST" | sort | paste -sd',' - )
- assert_eq "default (opt-in off) keeps everything" \
- "gen/api.pb.go,src/Model.cs,src/app.js,src/migrate.py,src/notes.go,src/widget.js" "$off"
- # Opt-in ON: the three banner-shaped generated files drop; the marker-quoting
- # first-party files (prose, bare DO NOT EDIT) are kept (Morlock M1).
- on=$( cd "$G"; GIT_OK=true; RAW_DIR="$G/raw"; SCAN_ROOTS=(.); CHECKUP_EXCLUDE_GENERATED=1; build_source_inventory
- tr '\0' '\n' < "$SOURCE_LST" | sort | paste -sd',' - )
- assert_eq "opt-in drops banner-shaped generated, keeps marker-quoting first-party" \
+ # Default ON (#117 flip): the three banner-shaped generated files drop; the
+ # marker-quoting first-party files (prose, bare DO NOT EDIT) are kept (Morlock M1).
+ on=$( cd "$G"; GIT_OK=true; RAW_DIR="$G/raw"; SCAN_ROOTS=(.); build_source_inventory; tr '\0' '\n' < "$SOURCE_LST" | sort | paste -sd',' - )
+ assert_eq "default-on drops banner-shaped generated, keeps marker-quoting first-party" \
"src/app.js,src/migrate.py,src/notes.go" "$on"
- cnt=$( cd "$G"; GIT_OK=true; RAW_DIR="$G/raw"; SCAN_ROOTS=(.); CHECKUP_EXCLUDE_GENERATED=1; build_source_inventory; printf '%s' "$GENERATED_EXCLUDED_COUNT" )
- assert_eq "GENERATED_EXCLUDED_COUNT set" "3" "$cnt"
- enum=$( cd "$G"; GIT_OK=true; RAW_DIR="$G/raw"; SCAN_ROOTS=(.); CHECKUP_EXCLUDE_GENERATED=1; build_source_inventory; tr '\0' '\n' < "$RAW_DIR/source-files.lst.generated" | sort | paste -sd',' - )
+ # Kill-switch: CHECKUP_EXCLUDE_GENERATED=0 restores the whole tree.
+ off=$( cd "$G"; GIT_OK=true; RAW_DIR="$G/raw"; SCAN_ROOTS=(.); CHECKUP_EXCLUDE_GENERATED=0; build_source_inventory; tr '\0' '\n' < "$SOURCE_LST" | sort | paste -sd',' - )
+ assert_eq "kill-switch (=0) keeps everything" \
+ "gen/api.pb.go,src/Model.cs,src/app.js,src/migrate.py,src/notes.go,src/widget.js" "$off"
+ cnt=$( cd "$G"; GIT_OK=true; RAW_DIR="$G/raw"; SCAN_ROOTS=(.); build_source_inventory; printf '%s' "$GENERATED_EXCLUDED_COUNT" )
+ assert_eq "GENERATED_EXCLUDED_COUNT set (default-on)" "3" "$cnt"
+ enum=$( cd "$G"; GIT_OK=true; RAW_DIR="$G/raw"; SCAN_ROOTS=(.); build_source_inventory; tr '\0' '\n' < "$RAW_DIR/source-files.lst.generated" | sort | paste -sd',' - )
assert_eq "drops enumerated (loud, not silent)" "gen/api.pb.go,src/Model.cs,src/widget.js" "$enum"
rm -rf "$G"
else
echo " ⊘ skipped — git not installed"
fi
+echo ""
+echo "build_scc_keepset: coverage-by-category counts (convention/author/generated, #117)"
+if command -v git > /dev/null 2>&1 && command -v jq > /dev/null 2>&1; then
+ G="$HOME/.checkup-cov.$$"; rm -rf "$G"; mkdir -p "$G/src" "$G/external" "$G/gen"
+ ( cd "$G"
+ git init -q && git config user.email t@t && git config user.name t
+ # NB: a non-builtin dir name (external/), so the author-declared exclusion is
+ # what catches it — `vendor/` would be caught by the convention stage first.
+ printf 'external/** linguist-vendored\n' > .gitattributes
+ printf 'export const a=1;\n' > src/a.ts # first-party (keep)
+ printf '{"x":1}\n' > src/data.json # first-party config (keep, all-ext)
+ printf 'export const v=1;\n' > external/lib.js # author-declared vendored → excluded
+ printf 'export const m=1;\n' > src/x.min.js # convention suffix → excluded
+ printf '// @generated\nexport const g=1;\n' > gen/api.js # generated marker → excluded
+ git add -A && git commit -qm init >/dev/null 2>&1 )
+ run() { ( cd "$G"; GIT_OK=true; RAW_DIR="$G/raw"; SCAN_ROOTS=(.); build_scc_keepset; printf '%s' "${!1}" ); }
+ assert_eq "SCC_CANDIDATE_COUNT = 6 (all tracked, incl .gitattributes)" "6" "$(run SCC_CANDIDATE_COUNT)"
+ assert_eq "CONVENTION_EXCLUDED_COUNT = 1 (.min.js)" "1" "$(run CONVENTION_EXCLUDED_COUNT)"
+ assert_eq "GITATTR_EXCLUDED_COUNT = 1 (vendor/lib.js)" "1" "$(run GITATTR_EXCLUDED_COUNT)"
+ assert_eq "GENERATED_EXCLUDED_COUNT = 1 (gen/api.js)" "1" "$(run GENERATED_EXCLUDED_COUNT)"
+ assert_eq "SCC_KEEP_COUNT = 3 (a.ts, data.json, .gitattributes)" "3" "$(run SCC_KEEP_COUNT)"
+ # The deltas must partition the candidate set exactly (no double count, no leak).
+ part=$( cd "$G"; GIT_OK=true; RAW_DIR="$G/raw"; SCAN_ROOTS=(.); build_scc_keepset
+ echo $(( SCC_KEEP_COUNT + CONVENTION_EXCLUDED_COUNT + GITATTR_EXCLUDED_COUNT + GENERATED_EXCLUDED_COUNT )) )
+ assert_eq "kept + excluded categories == candidates" "6" "$part"
+ rm -rf "$G"
+else
+ echo " ⊘ skipped — git/jq not installed"
+fi
+
echo ""
echo "coverage helpers: by-area grouping and exclusion-source label"
SOURCE_LST="$TMP/cov.lst"