security: Sprint 0 emergency fixes

Dockerfile

@@ -3,11 +3,22 @@ FROM python:3.11-slim
 WORKDIR /app
 COPY pyproject.toml README.md ./
 COPY src/ ./src/
-RUN pip install --no-cache-dir .
+
+# Install the project into an agent-owned venv. Runtime user dependencies
+# (mounted via /workspace/requirements.txt) are installed into the same venv
+# by entrypoint.sh, so they resolve for the non-root `agent` user.
+RUN python -m venv --system-site-packages /venv \
+    && /venv/bin/pip install --no-cache-dir . \
+    && useradd -r -s /bin/false agent \
+    && mkdir -p /workspace /home/agent \
+    && chown -R agent:agent /app /workspace /venv /home/agent
 
 COPY entrypoint.sh /entrypoint.sh
 RUN chmod +x /entrypoint.sh
 
+ENV PATH="/venv/bin:$PATH" HOME="/home/agent"
+USER agent
+
 WORKDIR /workspace
 ENTRYPOINT ["/entrypoint.sh"]
 CMD ["serve", "--config", "/workspace/agents.yml"]

SECURITY.md

@@ -0,0 +1,123 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported |
+|---------|-----------|
+| `main` (beta) | Yes — active development, receives security fixes |
+| < 1.0 releases | No — no stable releases yet |
+
+This project is in **beta**. Security fixes are applied to `main` and shipped in the next container image build (`ghcr.io/extra-org/extra`). There are no long-term support branches.
+
+---
+
+## Reporting a Vulnerability
+
+If you discover a security vulnerability, please report it **privately** — do not open a public GitHub issue.
+
+**Preferred:** [GitHub Security Advisories](https://github.com/extra-org/extra/security/advisories/new)
+
+### What to include
+
+- Description of the vulnerability
+- Steps to reproduce
+- Affected version or commit
+- Potential impact assessment
+- Any suggested fix (if applicable)
+
+### Response timeline
+
+| Step | Target |
+|------|--------|
+| Acknowledgment | 72 hours |
+| Triage and severity assessment | 7 days |
+| Fix or mitigation | 14 days for critical, 30 days for others |
+| Public disclosure | After fix is released, coordinated with reporter |
+
+We follow **coordinated disclosure**. Please do not publicly disclose the vulnerability until a fix is available and you have been notified.
+
+---
+
+## Important: Beta Software — Known Security Gaps
+
+This project is **not production-ready**. The following critical security gaps are known and documented. **Do not expose this software to an untrusted network without addressing them.**
+
+| Gap | Risk | Status |
+|-----|------|--------|
+| **No authentication at the HTTP boundary** — Extra does not validate API keys or JWTs; authentication is expected to be handled by the client gateway, host application, or API middleware. The verified identity/context is then passed into Extra (e.g. via the runtime hooks feature) for access-control decisions. See [docs/SIDECAR_CONTEXT_AUTH.md](docs/SIDECAR_CONTEXT_AUTH.md). | Critical | Design gap — deployer responsibility |
+| **Access control not enforced** — the `protected` node feature is structurally wired but the Security/Context Gate that populates real identity is not implemented | Critical | Open — Sprint 1 planned |
+| **No rate limiting** — no per-IP or per-endpoint request throttling | High | Open — Sprint 1 planned |
+| **No built-in TLS** — must be handled by a reverse proxy | High | Open — Sprint 3 planned |
+| **No security headers** — missing `X-Content-Type-Options`, `X-Frame-Options`, `Strict-Transport-Security` | Medium | Open — Sprint 1 planned |
+| **No SSE disconnect detection** — engine continues processing after client disconnects | Medium | Open — Sprint 2 planned |
+| **No server timeouts** — uvicorn keep-alive and request timeouts not configured | Medium | Open — Sprint 2 planned |
+
+For the full remediation plan, see [docs/SECURITY_SPRINTS.md](docs/SECURITY_SPRINTS.md).
+
+---
+
+## Security Features Already Implemented
+
+Despite the gaps above, the following security practices are in place:
+
+| Practice | Description |
+|----------|-------------|
+| YAML secret scanning | Rejects literal secrets and credential shapes; only variable references are allowed |
+| Non-root Docker user | Container runs as `agent` user, not root |
+| CORS default deny | Empty allowed origins list by default; must be explicitly configured |
+| Request-field length validation | Pydantic `max_length` bounds on request string fields (applied after the body is read/parsed — this is **field-length validation, not a full HTTP body-size limit**). A true HTTP body-size limit at the ASGI/proxy layer is planned in Sprint 2 |
+| Execution cost guardrails | Per-run token and cost limits prevent runaway LLM spend |
+| MCP auth headers never logged | Authentication headers passed to MCP servers are excluded from logs |
+| SQL injection prevention | All database access goes through SQLAlchemy ORM |
+| Safe YAML loading | YAML is loaded without code execution or arbitrary object construction |
+| Sensitive argument masking | HITL approval prompts mask sensitive tool arguments |
+| Request ID sanitization | Request IDs are regex-validated and capped at 64 characters |
+
+---
+
+## Deployment Security Recommendations
+
+Before exposing Extra to any network, you **must**:
+
+1. **Terminate authentication at the client gateway / host application / API middleware** — Extra does not validate API keys or JWTs itself. The verified caller identity and context must be passed into Extra for access-control decisions, for example via the runtime hooks feature (see [docs/SIDECAR_CONTEXT_AUTH.md](docs/SIDECAR_CONTEXT_AUTH.md)).
+
+2. **Run behind a reverse proxy** — use nginx, Caddy, or a cloud load balancer to terminate TLS and add security headers.
+
+3. **Restrict CORS origins** — set `CORS_ORIGINS` in your `.env` to the exact domains that should access the API.
+
+4. **Use Docker network isolation** — bind to `127.0.0.1:port:port` and expose through the container platform's networking.
+
+5. **Monitor logs** — watch for authentication failures, approval rejections, and protected-node access attempts.
+
+6. **Pin container versions** — use `ghcr.io/extra-org/extra:v1.2.3` (specific tag), not `:latest`.
+
+---
+
+## Security Roadmap
+
+| Sprint | Focus | Status |
+|--------|-------|--------|
+| Sprint 0 | Emergency fixes (error responses, non-root Docker, request limits) | Completed |
+| Sprint 1 | Access control & rate limiting (wire identity into `protected` nodes, rate limiting, security headers) | Planned |
+| Sprint 2 | Input hardening (timeouts, SSE cleanup, path validation, log redaction) | Planned |
+| Sprint 3 | Production readiness (TLS docs, health checks, audit logging) | Planned |
+
+Full details: [docs/SECURITY_SPRINTS.md](docs/SECURITY_SPRINTS.md)
+
+---
+
+## Security Update Process
+
+- Security fixes are merged to `main` and released via [release-please](https://github.com/googleapis/release-please) automation.
+- Container images are published to `ghcr.io/extra-org/extra` on each release.
+- Users should subscribe to [GitHub Releases](https://github.com/extra-org/extra/releases) for notifications.
+- For critical vulnerabilities, we will publish a security advisory on GitHub with affected versions and upgrade instructions.
+
+---
+
+## References
+
+- [Security Sprints](docs/SECURITY_SPRINTS.md) — full remediation plan
+- [Architecture](docs/ARCHITECTURE.md) — system design and security boundaries
+- [Runtime Lifecycle](docs/RUNTIME_LIFECYCLE.md) — request flow and security gate
+- [Plugin Context/Auth](docs/SIDECAR_CONTEXT_AUTH.md) — access control design

docker-compose.yml

@@ -0,0 +1,43 @@
+# Local development compose — exposes services on 127.0.0.1 only.
+# Not intended for production deployment.
+#
+# The manager needs a writable database. For local dev it uses a bundled SQLite
+# file on a named volume (writable under the non-root `agent` user); production
+# deployments point DATABASE_URL at a client-owned database instead.
+
+services:
+  engine:
+    build: .
+    ports:
+      - "127.0.0.1:8090:8090"
+    volumes:
+      - ./examples:/workspace:ro
+    env_file:
+      - .env
+    command: ["serve", "--host", "0.0.0.0", "--config", "/workspace/enterprise-knowledge-assistant/agents.yaml"]
+    healthcheck:
+      test: ["CMD", "python", "-c", "import httpx; httpx.get('http://127.0.0.1:8090/health')"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+
+  manager:
+    build: .
+    ports:
+      - "127.0.0.1:8100:8100"
+    volumes:
+      - ./examples:/workspace:ro
+      - manager-data:/data
+    environment:
+      - DATABASE_URL=sqlite+aiosqlite:////data/chat.db
+    env_file:
+      - .env
+    command: ["agent-manager", "--host", "0.0.0.0", "--config", "/workspace/enterprise-knowledge-assistant/agents.yaml"]
+    healthcheck:
+      test: ["CMD", "python", "-c", "import httpx; httpx.get('http://127.0.0.1:8100/health')"]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+
+volumes:
+  manager-data:

docs/SECURITY_SPRINTS.md

@@ -0,0 +1,83 @@
+# Security Sprints
+
+Actionable remediation plan from the security audit (July 2026).
+Each sprint is ordered — do them in sequence.
+
+---
+
+## Sprint 0 — Emergency Fixes (1-2 days)
+
+Quick wins, low effort, high impact.
+
+| # | Task | Files | Effort |
+|---|------|-------|--------|
+| 1 | **Generic error responses** — replace `str(exc)` with sanitized messages in all API error returns. Log full exceptions server-side only. | `agent_engine/api/app.py:243,297,332` `agent_manager/api/routes.py:64,115` | 2h |
+| 2 | **Non-root Docker user** — add `RUN useradd -r -s /bin/false agent` and `USER agent` before entrypoint. | `Dockerfile` | 30m |
+| 3 | **Request size limits** — add `max_length=4096` (or similar) to all `str` fields in request schemas. | `agent_manager/api/schemas.py` `agent_engine/api/app.py` | 1h |
+| 4 | **Default host 127.0.0.1** — change default `--host` from `0.0.0.0` to `127.0.0.1` in both servers. | `agentctl/main.py:190` `agent_manager/config.py:29` | 30m |
+| 5 | **Docker Compose** — add `docker-compose.yml` with `engine` (8090) and `manager` (8100) services, SQLite volume, health checks, `.env` passthrough. | `docker-compose.yml` | 30m |
+
+**Definition of done:** `make check` passes. No `str(exc)` in HTTP responses. Docker runs as non-root. `docker compose up` starts both services.
+
+---
+
+## Sprint 1 — Auth & Access Control (1-2 weeks)
+
+The critical missing piece.
+
+| # | Task | Files | Effort |
+|---|------|-------|--------|
+| 1 | **Auth middleware** — implement a FastAPI dependency that validates API key or JWT bearer token. Apply to all routes in both APIs. | `agent_engine/api/app.py` `agent_manager/api/app.py` `agent_manager/api/deps.py` | 3-5d |
+| 2 | **Wire auth context to AccessFilter** — populate `AuthContext` from the verified identity so `protected` nodes are actually enforced. | `engine/langgraph/engine.py:669-674` `engine/langgraph/filters.py:42-67` `docs/SIDECAR_CONTEXT_AUTH.md` | 2-3d |
+| 3 | **Rate limiting** — add `slowapi` or equivalent. Per-IP and per-endpoint limits, especially on `/invoke` and `/stream`. | `agent_engine/api/app.py` `agent_manager/api/app.py` | 1d |
+| 4 | **Security headers** — add middleware for `X-Content-Type-Options`, `X-Frame-Options`, `Strict-Transport-Security`, `Referrer-Policy`. | Both API app files | 2h |
+| 5 | **Tighten CORS** — restrict `allow_methods` and `allow_headers` to specific values instead of `*`. | `agent_manager/api/web.py:24-25` | 30m |
+
+**Definition of done:** All endpoints require valid credentials. `protected: true` nodes reject unauthenticated callers. `make check` passes.
+
+---
+
+## Sprint 2 — Input Hardening (1 week)
+
+Defense-in-depth.
+
+| # | Task | Files | Effort |
+|---|------|-------|--------|
+| 1 | **Server timeouts** — configure uvicorn `timeout_keep_alive` and request timeouts. | `agentctl/main.py:208` `agent_manager/cli.py:38` | 2h |
+| 2 | **SSE disconnect detection** — abort engine processing when client disconnects mid-stream. | `agent_engine/api/app.py:264-303` `agent_manager/api/routes.py:91-119` | 1d |
+| 3 | **Path validation** — verify `import_from_path()` resolves within the expected `plugins/` directory. | `loaders/_import.py:22-29` | 2h |
+| 4 | **Dependency lockfile** — generate `uv.lock` or `pip-compile` output for reproducible builds. | `pyproject.toml` | 1h |
+| 5 | **Log redaction** — add a redaction pass in `StructuredFormatter` for values matching secret patterns at DEBUG level. | `logging_config.py` `observability/providers/logging/provider.py` | 4h |
+
+**Definition of done:** Timeouts configured. SSE cleans up on disconnect. Lockfile committed. `make check` passes.
+
+---
+
+## Sprint 3 — Production Readiness (ongoing)
+
+Operational security.
+
+| # | Task | Files | Effort |
+|---|------|-------|--------|
+| 1 | **TLS docs** — document that TLS must be handled by reverse proxy. Add `--ssl-keyfile`/`--ssl-certfile` options to `agentctl serve`. | `agentctl/main.py` `docs/` | 1d |
+| 2 | **Health check** — add `GET /healthz` that returns 200 when engine is ready. Add `HEALTHCHECK` to Dockerfile. | `agent_engine/api/app.py` `Dockerfile` | 2h |
+| 3 | **Audit logging** — log all authentication failures, approval decisions, and protected-node access attempts. | `agent_engine/api/app.py` `engine/langgraph/filters.py` | 1d |
+| 4 | **Pentest checklist** — create a doc with test cases for an external security review. | `docs/SECURITY_PENTEST.md` | 4h |
+
+**Definition of done:** Deployment docs include TLS guidance. Docker image has health check. Audit log covers sensitive operations.
+
+---
+
+## Appendix: What's Already Done Well
+
+| Practice | Location |
+|----------|----------|
+| YAML secret scanning (keys + credential shapes) | `parsers/yaml/parser.py:483-498` |
+| Sensitive argument masking in HITL approvals | `approvals/sanitization.py` |
+| CORS default deny (empty origins list) | `agent_manager/config.py:33` |
+| Request ID sanitization (regex + 64-char cap) | `agent_engine/api/app.py:44-56` |
+| MCP auth headers never logged | `runtime/hooks/mcp.py:81-85` |
+| Execution limits per run (cost guardrails) | `runtime/execution.py` |
+| Access filter fail-closed on exceptions | `engine/langgraph/filters.py:51-67` |
+| `.env` files gitignored | `.gitignore` |
+| SQL injection prevention via ORM | `infrastructure/persistence/` |

entrypoint.sh

@@ -1,6 +1,9 @@
 #!/bin/sh
 set -e
 
+# PATH already points at the project's agent-owned venv (/venv/bin), so `pip`
+# below installs workspace dependencies into a writable, non-root location.
+
 if { [ "$1" = "serve" ] || [ "$1" = "agent-manager" ]; } && [ -f /workspace/requirements.txt ]; then
     echo "Installing user dependencies..." >&2
     pip install -q -r /workspace/requirements.txt

src/agent_engine/api/app.py

@@ -14,7 +14,7 @@
 
 from fastapi import FastAPI, Header, HTTPException, Response
 from fastapi.responses import StreamingResponse
-from pydantic import BaseModel
+from pydantic import BaseModel, Field
 
 from agent_engine.approvals.decision import ApprovalDecision, parse_decision
 from agent_engine.approvals.errors import (
@@ -106,7 +106,7 @@ def _map_approval_error(exc: ApprovalError) -> HTTPException:
 
 
 class InvokeRequest(BaseModel):
-    message: str
+    message: str = Field(max_length=65536)
 
 
 class ToolRecord(BaseModel):
@@ -145,15 +145,15 @@ class ApprovalDecisionRequest(BaseModel):
     """Decide a pending tool call. ``user_id`` must match the run's authorized
     approver when one was recorded."""
 
-    user_id: str | None = None
+    user_id: str | None = Field(default=None, max_length=128)
 
 
 class ApprovalDecisionBody(ApprovalDecisionRequest):
     """A free-text decision from a UI/CLI, parsed to a typed decision at this
     boundary. Accepts values like ``allow``, ``allow for this session``, or
     ``deny``; an unrecognized value yields a 400."""
 
-    decision: str
+    decision: str = Field(max_length=128)
 
 
 class RunStatusResponse(BaseModel):
@@ -240,7 +240,7 @@ async def invoke(
             )
         except Exception as exc:
             log(logger, logging.ERROR, "request end", status="error", error=str(exc))
-            raise HTTPException(status_code=500, detail=str(exc)) from exc
+            raise HTTPException(status_code=500, detail="internal server error") from exc
         log(
             logger,
             logging.INFO,
@@ -294,7 +294,7 @@ async def event_stream():
                 )
             except Exception as exc:
                 log(logger, logging.ERROR, "request end", status="error", error=str(exc))
-                yield f"data: {json.dumps({'type': 'error', 'detail': str(exc)})}\n\n"
+                yield f"data: {json.dumps({'type': 'error', 'detail': 'internal server error'})}\n\n"
 
         return StreamingResponse(
             event_stream(),
@@ -329,7 +329,7 @@ async def _decide(
         except ApprovalError as exc:
             raise _map_approval_error(exc) from exc
         except Exception as exc:
-            raise HTTPException(status_code=500, detail=str(exc)) from exc
+            raise HTTPException(status_code=500, detail="internal server error") from exc
         return InvokeResponse(
             system_name=result.system_name,
             answer=result.answer,

src/agent_manager/api/routes.py

@@ -4,6 +4,7 @@
 
 import dataclasses
 import json
+import logging
 from collections.abc import AsyncIterator
 from typing import Annotated
 
@@ -28,6 +29,7 @@
 )
 
 router = APIRouter()
+logger = logging.getLogger(__name__)
 
 Service = Annotated[ConversationService, Depends(get_service)]
 
@@ -61,7 +63,8 @@ async def send_message(
     except ConversationTokenBudgetExceeded:
         raise HTTPException(status_code=429, detail="conversation token budget exceeded") from None
     except Exception as exc:  # engine failure
-        raise HTTPException(status_code=500, detail=str(exc)) from exc
+        logger.exception("send_message failed: conversation_id=%s", conversation_id)
+        raise HTTPException(status_code=500, detail="internal server error") from exc
     return SendMessageResponse(
         answer=result.answer,
         visited=list(result.visited),
@@ -112,7 +115,8 @@ async def event_source() -> AsyncIterator[str]:
                 payload = _to_stream_event(event).model_dump(exclude_none=True)
                 yield f"event: {event.type}\ndata: {json.dumps(payload)}\n\n"
         except Exception as exc:
-            yield f"event: error\ndata: {json.dumps({'type': 'error', 'error': str(exc)})}\n\n"
+            logger.exception("stream_message failed: conversation_id=%s", conversation_id)
+            yield f"event: error\ndata: {json.dumps({'type': 'error', 'error': 'internal server error'})}\n\n"
         finally:
             yield "event: done\ndata: [DONE]\n\n"