Skip to content

node_modules in git history bloats the pack to ~30 MiB #32

Description

@melvincarvalho

Symptom

The tracked working tree is only ~444 KB, but `.git` is ~30 MiB. A first `git clone` (or any push to a new remote with a 10 MB body limit, e.g. a typical Solid pod's git endpoint) transfers all of that history.

Cause

`node_modules/` was committed at some point in the past. It's now in `.gitignore`, but the historical blobs are still in the pack.

Top blobs across all history:

Size Path
11.1 MB `node_modules/@esbuild/linux-x64/bin/esbuild`
2.1 MB `node_modules/@rollup/rollup-linux-x64-gnu/rollup.linux-x64-gnu.node`
1.3 MB `node_modules/vite/dist/node/chunks/config.js`
1.2 MB `node_modules/rdflib/dist/rdflib.min.js.map`
930 KB `node_modules/rollup/dist/es/shared/node-entry.js`
929 KB `node_modules/rollup/dist/shared/rollup.js`
792 KB `node_modules/jsonld/dist/jsonld.js`
711 KB `node_modules/@asamuzakjp/css-color/dist/browser/css-color.min.js.map`
623 KB `node_modules/jsonld/dist/jsonld.esm.js`
521 KB `node_modules/rdflib/dist/515.rdflib.min.js.map`

Repro:

```
git rev-list --objects --all
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)'
| awk '$1=="blob"' | sort -k3 -nr | head
```

Impact

  • New contributors download ~30 MiB to clone a 444 KB project
  • Pushing to git remotes with a body-size limit (~10 MB) fails with HTTP 413 — hit this trying to install the app via `git push` to a Solid pod
  • CI fetches and any future `gh-pages` deploys carry the same overhead

Proposed fix

Purge `node_modules/` (and any other historical bloat) from history with `git-filter-repo`:

```
git filter-repo --invert-paths --path node_modules/
```

This rewrites all commit SHAs, so it needs coordination:

  1. Pick a quiet window
  2. Run the filter locally on a fresh clone
  3. `git push --force` to GitHub
  4. All collaborators `git fetch --all && git reset --hard origin/` (or re-clone)
  5. Any open PRs need to be rebased

Expected reduction: ~30 MiB → ~1 MiB (444 KB working tree + a small amount of compressed history).

Optional belt-and-braces

After the purge, add a pre-commit / CI guard to prevent regression. Example with husky:

```yaml

.husky/pre-commit

git diff --cached --name-only | grep -q '^node_modules/' && {
echo "refusing to commit anything under node_modules/"; exit 1; }
```

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions