x-brain

Turn any X account's reposts into a queryable knowledge base + mind graph.

No username or key is persistent. Copy .env.example → .env and fill it. Every secret stays local.

1. Install

git clone <this-repo> && cd x-brain
python3 -m venv .venv && source .venv/bin/activate
pip install -r harness/requirements.txt
ollama serve &  # or your Ollama install

Pull the models you want (see config/models.json for the tested roster; any GGUF registered in Ollama works — edit the JSON and restart):

ollama pull nemotron3-nano   # or: ollama create nemotron3-nano -f Modelfile

2. Configure

cp .env.example .env
# edit .env:
#   X_USER_ID=...            # numeric X user id (https://tweeterid.com/)
#   X_AUTH_TOKEN=...         # from DevTools → Application → Cookies → x.com → auth_token
#   X_CT0=...                # same place → ct0
#   XBRAIN_DIR=./data        # optional, default ~/.xbrain
#   INFERX_API_KEY=...       # optional, for router/vision
#   OPENROUTER_API_KEY=...   # optional, for vision/fallback

Alternatively per-command:

python3 harness/xb.py --brain-dir ./data --user-id 123456 auth --auth-token ... --ct0 ... --user-id 123456

Or via env: X_USER_ID=123 python3 harness/xb.py doctor

3. Run

# verify creds + one live page
python3 harness/xb.py --brain-dir ./data doctor
python3 harness/xb.py --brain-dir ./data doctor --posts-only   # same for posts timeline

# L1: enumerate reposts — or posts with --posts-only (originals only, separate cursor)
python3 harness/xb.py --brain-dir ./data enum --resume
python3 harness/xb.py --brain-dir ./data enum --posts-only --resume

# L2: thread context (Fx fallback built-in)
python3 harness/xb.py --brain-dir ./data enrich

# L3: resolve t.co links
python3 harness/xb.py --brain-dir ./data links-run

# L4: media understanding (optional, needs vision keys)
python3 harness/xb.py --brain-dir ./data vision-run

# L5: routed LLM cards (4 cards/row, quarantines on failure)
python3 harness/xb.py --brain-dir ./data llm-run --route --mode int-ext-int --backends ollama,inferx,or-nemotron

# curation + fusion (run after L3/L4/L5)
python3 harness/xb.py --brain-dir ./data deep-run
python3 harness/xb.py --brain-dir ./data fuse-run

# mind graph
python3 harness/xb.py --brain-dir ./data graph-export --out ./data/vault
# open ./data/vault in Obsidian

./run_all.sh, ./run_text_ox.sh, ./run_vision.sh, ./run_links_deep.sh are resilient wrappers (auto-restart, log to $XBRAIN_DIR/*.log).

4. Dashboard (search your cache)

cd dashboard/web && npm install && npm run build
XBRAIN_DIR=./data python3 dashboard/server.py --port 5173
# open http://127.0.0.1:5173

Keyword (FTS5 BM25) + latent vector + manual-submit agentic search, top-K CSV/Markdown export, per-account drain control (L1–L10, detached), and a first-run demo/tutorial on fictional data for empty caches. Full docs: dashboard/README.md.

Dashboard-first path (recommended — start here, no CLI needed)

You need: Python 3.10+, Node 18+, an X account, and 5 minutes.

1. Install + build (one time):

   git clone <this-repo> && cd x-brain
   pip install -r harness/requirements.txt -r dashboard/requirements.txt
   cd dashboard/web && npm install && npm run build && cd ../..

2. Run: XBRAIN_DIR=./data python3 dashboard/server.py --port 5173, then open http://127.0.0.1:5173. Empty cache → you get a welcome tutorial + 6 fictional sample posts. (Turn it off anytime: Settings → Demo & tutorial.)
3. Connect your X account: click the account card next to the search box (shows Connect X when empty). Fill @handle + auth_token + ct0 (browser DevTools → Application → Cookies → x.com), optional numeric user id, then Save + personalize. Tokens land in ./data/creds.json (0600) and never leave your machine.
4. Start the drain — same panel, top to bottom, each shows pending counts and time estimates before you start it (runs detached, safe to close the tab):
   • L1 enum — your repost history. Wait for this to finish first.
   • L2 enrich → thread context. L3 links → article text.
   • L4 vision → image OCR/describe (needs a vision key, budget-capped).
   • L5 llm-run → topic/entities/summary + reasoning on every row.
   • L6 deep → curation, L7 fuse → unified summaries (run last), L8 vectors → latent search (needs sentence-transformers), L9 index → rebuild search, L10 graph → Obsidian vault.
5. Search: Keyword for exact terms, Latent for semantic recall, Agentic (press Enter — manual submit saves LLM calls). Export top-K to CSV/MD. Click any card for the Detail modal (post, vision, link, reasoning).

Stuck? harness/scripts/status.sh shows drain health; the CLI in §3 below runs every stage manually with the same flags the dashboard uses.

5. Config as code

config/models.json drives all model selection (routing_tiers fast/standard/deep/uncensored). Edit it and restart:

{
  "routing_tiers": {
    "fast":       {"tag_topic": "granite-4.1-3b-q8", "extract_entities": "qwen3.5-4b-super-coder-q4", ...},
    "standard":   {"tag_topic": "nemotron3-nano", ...},
    "deep":       {"tag_topic": "ornith-1.5-9b-q4km", ...},
    "uncensored": {"tag_topic": "small-8b-gaston-q4km", ...}  // or null to disable
  }
}

The harness checks ollama list on start (AVAILABLE/MISSING → falls back to standard) so any Ollama tag can be dropped in.

6. Examples

python3 harness/xb.py --help
python3 harness/xb.py llm-run --help
python3 harness/xb.py graph-export --help
ls examples/   # tombstones, limits, rerun list, vision-log samples (all fictional)
cat config/models.md   # evaluation that chose the defaults (Q4/Q8, 6GB GPU)
cat LEARNINGS-gateway-504.md  # vision gateway engineering notes
python3 harness/scripts/seed_demo.py --out /tmp/xbrain-demo  # fictional demo DB

7. Live demo + deploy

Target	How	Notes
Static demo (tutorial + fictional samples)	.github/workflows/pages.yml → project site https://<user>.github.io/<repo>/	coexists with a user portfolio site; set VITE_BASE=/<repo>/
Full demo (real API, fictional DB)	render.yaml on Render free tier	SQLite seeded at build, FTS-only, sleeps when idle
Your own cache, one command	docker run -p 5000:5000 -v xbrain:/data <image>	Dockerfile; empty brain, drain via dashboard
Zero-install hacking	.devcontainer/ in Codespaces	120 free core-hrs/mo, port 5173 forwarded

X/Twitter ToS note: drains use your own login session against your own account's reposts; credentials never leave your machine. Hosted demos ship fictional data only — never proxy scraping for strangers.

8. Data layout

$XBRAIN_DIR/          # ./data by default
  state.sqlite        # single-writer WAL, single source of truth
  creds.json  (0600)  # auth_token + ct0
  config.json (0600)  # user_id
  limits.json         # rate-limit windows persisted across runs
  cache/              # tid pair cache
  quarantine/         # *.skip.json — never dropped, requeueable
  vault/              # Obsidian vault + graph.json

9. Security

• No default username/id — protocol.py has no hard-coded user. Missing id → prompt or X_USER_ID env/.env.
• No keys shipped — session.py reads X_AUTH_TOKEN/X_CT0/INFERX_API_KEY/OPENROUTER_API_KEY from env/.env or creds.json (0600).
• .gitignore excludes .env, creds.json, state.sqlite*, vault/, *.log.
• See .env.example for every knob.