Skip to content

Add embedding service for the portfolio assistant - #7

Merged
B-Blarr merged 4 commits into
mainfrom
assistant-embedding
Sep 24, 2026
Merged

B-Blarr merged 4 commits into
mainfrom
assistant-embedding

Conversation

@B-Blarr

@B-Blarr B-Blarr commented Sep 24, 2026

Copy link
Copy Markdown
Owner

Stage 2 of the portfolio assistant. A small FastAPI service that turns texts
into 768-dimensional vectors with intfloat/multilingual-e5-base. Nothing
calls it yet, and it is not deployed.

Why a separate service

The model needs about 1.8 GB of memory. Inside Django, each of the three
Gunicorn workers would load its own copy. As its own process it loads once,
and PyTorch stays out of the Django dependencies and the CI.

Changes

  • embedding_service/main.py: POST /embed and GET /health
  • embedding_service/requirements.txt: pinned dependencies, CPU-only PyTorch
    through --extra-index-url
  • embedding_service/smoke_test.py: checks a running service, standard
    library only
  • embedding_service/README.md: setup, API and design notes

Design decisions

  • The service adds the e5 prefix. Callers must send kind (query or
    passage). A missing prefix does not raise an error, it only lowers
    retrieval quality, so the field is required.
  • Texts over 512 tokens are rejected instead of being cut off silently.
  • def instead of async def for /embed. Encoding is CPU-bound and
    would block the event loop, including /health.
  • setuptools from PyPI, not from the PyTorch index. The PyTorch index
    only offers 78.1.0, which has known vulnerabilities.

Verification

  • /embed returns 768 values per text, each vector with length 1.0
  • Invalid kind, blank texts, an empty list, more than 64 texts and texts
    over 512 tokens return 422
  • Smoke test: related German passage 0.817, related English passage 0.770,
    unrelated passage 0.754
  • pip install --dry-run --ignore-installed -r requirements.txt resolves to
    torch-2.14.0+cpu and setuptools-84.0.0
  • flake8 and isort pass

@B-Blarr
B-Blarr merged commit c255482 into main Sep 24, 2026
3 checks passed
@B-Blarr
B-Blarr deleted the assistant-embedding branch September 24, 2026 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant