docs: IntakeGateway/QueryGateway consolidation assessment and plan

[badry-dev]

Adds two documents assessing whether IntakeGateway and QueryGateway should be combined under one product, and what that would involve. Both are committed to QueryGateway on the matching branch as well.

docs/tech-stack-comparison.md — should they be combined?

• Stack comparison — backend runtime, frontend, quality gates, and container topology, side by side.
• Shared surface — the ~9 capability areas both projects implement independently (connection registry, credential encryption, scheduling, run history, admin identity, settings/health, Oracle introspection, SPA shell, API client).
• Conflicts ranked by cost — Python 3.11 vs 3.14 (with a queue-on-3.14 blocker underneath), sync vs async SQLAlchemy, SQLite+create_all vs PostgreSQL+Alembic, Ant Design vs Tailwind/shadcn, incompatible auth models, config collision, route namespaces, loguru vs structlog, encrypted-file vs database credential storage.
• Defects found during review (see below).
• Recommendation — converge onto QueryGateway's platform, port IntakeGateway's domain logic and worker topology onto it; monorepo with shared packages, one admin console, two backend deployables. Four costed alternatives.
• Open decisions requiring a human call — queue technology on 3.14, multi-user/RBAC, product naming, destination breadth, snapshot storage.

docs/consolidation-plan.md — what gets adopted from where

Module-by-module verdicts at implementation level, taking the better implementation regardless of which repo it lives in. Neither codebase wins across the board.

Where this repository's implementation wins: the destination write path (batched binds, bulk SELECT → INSERT/UPDATE upsert), the whole ingestion pipeline, url_guard.py (QueryGateway has no SSRF protection at all), the OAuth2 token cache, rate-limit and Retry-After handling, cursor watermarks, the Celery job queue (QueryGateway has none), row-level error logging with bounded staging, schedule auto-pause after N consecutive failures, engine caching, log redaction, rotate_key, Vite 8, and the coverage floor.

Where QueryGateway's implementation is adopted instead: declarative base and PK strategy, the repository layer, global exception handlers, request-correlation middleware, Alembic-only migrations, container topology, JWT identity, per-endpoint auth policies, structlog, the read-side SQL executor, schedule semantics, snapshot caching, health dashboard, and the frontend shell, API client, CRUD-page pattern and wizard architecture.

One fix is required during the port rather than after: oauth_token_service._get_lock returns an asyncio.Lock keyed by task id, which serializes token refresh only within a single process. With multiple workers, concurrent refreshes race — it needs a database advisory lock or a Redis lock.

The document also recalibrates the phase estimates for AI-assisted development, where review throughput rather than code production is the binding constraint, and notes which parts of the work do not compress at all.

Defects surfaced in this repo

Worth fixing regardless of the merge decision:

1. backend/app/services/connection_pool.py:36 hardcodes C:\oracle\instantclient_23_0 as the Oracle thick-mode client path — non-configurable and Windows-only, so thick mode is silently unavailable in the Linux containers this project ships.
2. PostgreSQL and MySQL destinations are advertised in the README and built in connection_pool.py, but neither psycopg2 nor PyMySQL appears in any requirements file or the Dockerfile — both fail at engine creation with ModuleNotFoundError.
3. requirements-minimal.txt and requirements-simple.txt pin FastAPI 0.104.0 / 0.100.0 next to the real requirements.txt at 0.141.1.
4. Schema is managed twice: Base.metadata.create_all() in lifespan alongside four Alembic revisions.
5. No type checking and no formatter gate in CI.
6. Duplicated comment line in backend/app/core/config.py.
7. Both date-fns and dayjs are shipped; antd already depends on dayjs.

Changes

Documentation only. No code, dependency, or configuration changes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01L6oCRzNahV9djmgUtpNe9q

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}