Skip to content

Development.md

Latest commit

 

History

History
345 lines (274 loc) · 13.5 KB

Development.md

File metadata and controls

345 lines (274 loc) · 13.5 KB
Raw

Development Guide

Covers building, testing, and extending both the backend and the browser extension.

Backend Development

Environment

  • Python 3.12+, uv package manager
  • Running Ollama instance (for AI features)

Setup

cd backend
uv sync                    # Install dependencies
uv run pre-commit install  # Enable git hooks (ruff, mypy)

Configuration

Create backend/.env:

DATABASE_URL=sqlite:///data/mindcache.db
FAISS_INDEX_PATH=data/faiss_index.bin
BM25_INDEX_PATH=data/bm25_index.pkl
EMBEDDING_PROVIDER=ollama
EMBEDDING_MODEL_NAME=embeddinggemma:300m
EMBEDDING_DIMENSION=768
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=qwen3.5:2b
LOG_LEVEL=INFO

Runtime .env (current overrides):

OLLAMA_MODEL=gemma4:31b-cloud
EMBEDDING_PROVIDER=ollama
EMBEDDING_MODEL_NAME=embeddinggemma:300m

The EMBEDDING_DIMENSION is auto-detected by querying Ollama /api/embed on startup, falling back to settings.EMBEDDING_DIMENSION (384) if detection fails.

Run

uv run uvicorn app.main:app --reload --host 127.0.0.1 --port 8000

API docs at http://127.0.0.1:8000/docs (auto-generated by FastAPI).

Docker

docker compose up --build -d

The Docker setup:

  • Builds from python:3.12-slim with uv for fast dependency installation
  • Mounts ./data:/app/data for persistent SQLite/FAISS/BM25 files
  • Connects to host Ollama via host.docker.internal:11434
  • Pre-downloads SentenceTransformer model for zero-network cold start

Database Migrations

MindCache uses Alembic for schema versioning. The initial schema is created programmatically in main.py lifespan, with dynamic PRAGMA-based ALTER TABLE for existing databases.

uv run alembic revision --autogenerate -m "describe_changes"
uv run alembic upgrade head
uv run alembic downgrade -1

The migrations/versions/ directory is intentionally empty — the app creates tables on startup if they don't exist.

Test

uv run pytest              # All tests
uv run pytest -s           # With logs/prints
uv run pytest --cov=app tests/  # With coverage
uv run pytest tests/test_api.py -k "test_visit_and_search_flow"  # Single test

Tests use an in-memory SQLite database (sqlite+aiosqlite:///:memory:) and mock all AI services (embedding, keyword extraction, entity extraction, Ollama). This means tests run in milliseconds with no external dependencies.

Test structure:

  • tests/conftest.py — Fixtures: event loop, in-memory DB, mocked AI services, FAISS/BM25 temp files
  • tests/test_api.py — 9 tests: health, visit+search flow, custom title, time filtering, noise skipping, platform search, all URLs, click analytics, search evaluation dataset
  • tests/test_extractors.py — 14 tests: factory selection, YouTube (success + missing transcript), X (success + partial fallback + fxtwitter), GitHub, Google Search (success + DDG fallback), Reddit (post + fallback + JSON fallback), pipeline integration
  • tests/test_services.py — 9 tests: BS4 fallback extraction, ISO date parsing, URL validation, embedding generation, keyword extraction, vector service (add/search/delete), Ollama summary, BM25 tokenization/search, quality score calculation
  • tests/search_evaluation.json — 4 regression queries: "stop ai slop", "karpathy coding rules", "fastapi web framework", "rust programming language"

Lint

uv run ruff check .              # Lint with auto-fix
uv run ruff format --check .     # Format check
uv run mypy .                    # Static type checking

Pre-commit hooks run ruff lint and format automatically on git commit.

Code style (pyproject.toml): line-length 120, target Python 3.12, selects E/W/F/I/B/C4/UP rules.

Project Structure

backend/
├── app/
│   ├── api/              # FastAPI routers
│   │   ├── __init__.py
│   │   ├── visit.py      # POST /visit
│   │   ├── search.py     # POST /search, POST /search/click
│   │   ├── documents.py  # CRUD /documents, GET /graph
│   │   └── health.py     # GET /health
│   ├── core/
│   │   ├── config.py     # Pydantic settings
│   │   ├── exceptions.py # Custom HTTP exceptions
│   │   └── logging.py    # Logging configuration
│   ├── db/
│   │   ├── base.py       # SQLAlchemy Base
│   │   └── session.py    # async engine + session factory
│   ├── models/
│   │   └── document.py   # ORM models (Document, Keyword, Entity, etc.)
│   ├── repositories/
│   │   └── document_repository.py  # Async CRUD operations
│   ├── schemas/
│   │   └── document.py   # Pydantic V2 request/response schemas
│   ├── services/
│   │   ├── __init__.py
│   │   ├── document_processor.py  # Ingestion orchestration
│   │   ├── embedding_service.py   # Ollama embedding generation
│   │   ├── ollama_service.py      # Ollama LLM communication
│   │   ├── vector_service.py      # FAISS management
│   │   ├── bm25_service.py        # BM25 lexical search
│   │   ├── search_service.py      # Hybrid search pipeline
│   │   ├── keyword_extractor.py   # Keyword extraction (Ollama + TF)
│   │   ├── entity_extractor.py    # NER (Ollama + regex)
│   │   └── extractors/            # Platform-specific extractors
│   │       ├── base.py            # Abstract base + ExtractionResult
│   │       ├── factory.py         # ExtractorFactory registry
│   │       ├── generic.py         # Trafilatura + BS4
│   │       ├── youtube.py         # yt-dlp + captions
│   │       ├── x.py               # fixupx.com syndication API
│   │       ├── github.py          # Repo metadata + README
│   │       ├── reddit.py          # old.reddit.com comments
│   │       ├── pdf.py             # pypdf + pspdfkit
│   │       └── google_search.py   # SERP + DuckDuckGo
│   └── main.py           # App entry, lifespan, CORS
├── data/                 # SQLite DB + FAISS + BM25 (gitignored)
├── migrations/
│   ├── env.py            # Alembic async config
│   └── script.py.mako    # Migration template
├── tests/                # Pytest test suite
├── Dockerfile            # Production image
├── docker-compose.yml    # Container orchestration
└── pyproject.toml        # Dependencies + tool config

Extension Development

Environment

  • Node.js 18+, npm 9+
  • Running backend server

Setup

cd extension
npm install

Dev Server (UI Only)

npm run dev

Serves popup and settings pages in a browser tab at a local Vite dev URL. Background worker cannot be fully tested here since it needs Chrome extension APIs (tabs, storage, contextMenus). For background testing, build and load the extension in a browser.

Build

npm run build

The build.js script runs a three-stage production build:

  1. Popup & Settings — Standard Vite build (ESM, dist/assets/)
  2. content.js (459KB) — Vite IIFE build with Defuddle + Turndown bundled
  3. background.js (28KB) — Vite IIFE build with blacklist constants inlined

All three stages set process.env.NODE_ENV = 'production'.

Test

npm run test          # Run once
npm run test:watch    # Watch mode

Tests use vitest with jsdom environment and mocked Chrome APIs:

  • tests/setup.ts — Mocks chrome.storage.local, chrome.tabs, chrome.runtime, global.fetch
  • tests/service.test.ts — 4 tests: health check (success + offline), visit submission, semantic search
  • tests/store.test.ts — 6 tests: settings defaults, modifications, domain exclusion; search query + recent searches constraints; connection state transitions

Load in Browser

  1. Go to chrome://extensions (or brave://extensions)
  2. Enable Developer mode
  3. Click Load unpacked → select extension/dist/

Package for Release

npm run build
cd dist
zip -r ../mindcache-extension.zip .

Project Structure

extension/
├── public/manifest.json       # Manifest V3: permissions, commands, content scripts
├── src/
│   ├── index.css              # Global Tailwind + custom styles
│   ├── background/index.ts    # Service worker: tab tracking, context menus, messaging
│   ├── content/
│   │   ├── index.ts           # Content script entry: message handlers, overlay init
│   │   ├── extract-page.ts    # Defuddle extraction pipeline + Turndown
│   │   └── SearchOverlay.tsx  # Spotlight-style search overlay (Shadow DOM)
│   ├── popup/
│   │   ├── main.tsx           # Popup React root
│   │   └── App.tsx            # Popup UI: status, preview, capture, diagnostics
│   ├── settings/
│   │   ├── main.tsx           # Settings React root
│   │   └── App.tsx            # Full dashboard: 5 tabs, TanStack Query, filters
│   ├── components/
│   │   ├── DocumentDetail.tsx  # Document detail view with AI summary
│   │   └── InteractiveKnowledgeGraph.tsx  # Force-directed graph (Canvas)
│   ├── services/
│   │   └── BackendClient.ts   # API client with retry + validation
│   ├── store/
│   │   ├── settingsStore.ts   # Persisted settings (Zustand + chrome.storage)
│   │   ├── searchStore.ts     # Search state + recent queries
│   │   └── connectionStore.ts # Backend health status (transient)
│   ├── types/
│   │   ├── index.ts           # All TypeScript interfaces
│   │   └── turndown.d.ts      # Turndown type declarations
│   └── utils/
│       ├── error.ts           # Error message formatting
│       └── storage.ts         # chrome.storage.local adapter
├── tests/                     # Vitest test suite
├── build.js                   # 3-stage Vite build script
├── vite.config.ts             # Dev server config
├── vitest.config.ts           # Test config
├── tailwind.config.js         # Tailwind CSS config
└── tsconfig.json              # TypeScript strict config

Named Constants

All hardcoded numeric values in the settings dashboard are extracted to named constants in src/settings/App.tsx:

Constant Default Purpose
DEFAULT_RESULT_LIMIT 10 Default search result count
MAX_DOCUMENTS_FETCH 100 Max documents in list query
SEARCH_DEBOUNCE_DELAY 250ms Search input debounce
SAVE_SUCCESS_DURATION 1500ms Success toast duration
RECENT_ACTIVITY_COUNT 7 Dashboard recent items
MAX_KEYWORDS_DISPLAY 4 Max keyword tags per result
GRAPH_DISPLAY_MAX 50 Graph node display cap
BEST_MATCH_THRESHOLD 65 "Best Match" badge threshold
STRONG_MATCH_THRESHOLD 55 "Strong Match" badge threshold
DEFAULT_MIN_SCORE 0 Default min match score filter
RESULT_LIMIT_OPTIONS [5, 10, 20, 30] Result limit dropdown options

Background worker inlines:

  • BLACKLISTED_EXTENSIONS — file extensions to skip (images, media, archives, binaries, docs)
  • BLACKLISTED_PATHS — URL path patterns to skip (login, signup, admin, auth, etc.)
  • DWELL_TIME_THRESHOLD_SEC — 10s before auto-indexing
  • MIN_DWELL_TIME_SEC — 5s minimum dwell to index on deactivation
  • EXTRACTION_TIMEOUT_MS — 2s content script timeout

Content script defines:

  • EXCLUDED_TAGS — elements stripped during noise removal
  • BLACKLISTED_PATHS — same path blacklist for URL validation

Adding a Context Menu Item

  1. Add chrome.contextMenus.create() in createContextMenus() in src/background/index.ts.
  2. Add a handler in the chrome.contextMenus.onClicked listener.
  3. Implement capture logic (reuse captureCurrentPage, captureUrl, or captureSelection).
  4. Add message handler in src/content/index.ts if DOM access is needed.

Adding a Keyboard Shortcut

  1. Register the command in extension/public/manifest.json under "commands".
  2. Add a handler in chrome.commands.onCommand in src/background/index.ts.
  3. Update the settings page Shortcuts section.

Ingestion Verification

  1. Start the backend.
  2. Load the extension in the browser.
  3. Visit any webpage, wait 10 seconds.
  4. Check backend logs for POST /visit transaction.
  5. Or check GET http://127.0.0.1:8000/documents.

Testing Manual Capture

  1. Turn off Auto extraction in extension Settings → Privacy.
  2. Visit a page.
  3. Use Ctrl+Shift+S, right-click → "Save this page to MindCache", or click the popup button.
  4. Verify the visit appears in GET /documents.

Testing the Search Overlay

  1. Open any webpage.
  2. Press Ctrl+Shift+K (or MacCtrl+Shift+K on macOS).
  3. The spotlight overlay appears with tabs and memory search modes.
  4. Test Tab mode switch, Ctrl+T/Ctrl+M shortcuts, and Arrow/Enter navigation.
  5. Verify backend connection shows in the footer badge.

Extension Dependencies

Package Purpose
@tanstack/react-query Data fetching (search, documents, dashboard)
defuddle Page content extraction (Obsidian Web Clipper)
turndown HTML-to-Markdown conversion
marked Markdown-to-HTML rendering (DocumentDetail)
lucide-react Icon library (50+ icons used)
zustand State management with chrome.storage persistence
clsx + tailwind-merge Class name utilities
@tailwindcss/typography Prose styling for document content
vitest + @testing-library/react Testing framework