Add retrieval for the portfolio assistant

.env.example

@@ -15,7 +15,7 @@ FORCE_SCRIPT_NAME=
 
 # PostgreSQL. These values match the local container from compose.yml.
 # On the server use the real credentials and port 5432.
-# Without DB_NAME the project falls back to local SQLite.
+# DB_NAME is required, Django refuses to start without it.
 DB_NAME=coderr
 DB_USER=coderr
 DB_PASSWORD=coderr
@@ -35,3 +35,10 @@ DB_PORT=5433
 # EMAIL_HOST_PASSWORD=
 # DEFAULT_FROM_EMAIL=
 # CONTACT_RECIPIENT=
+
+# Embedding service for the portfolio assistant, see embedding_service/.
+# EMBEDDING_SERVICE_URL=http://127.0.0.1:8002
+
+# Portfolio assistant. Off unless set. Keep it off on the server until the
+# embedding service runs there.
+# ASSISTANT_ENABLED=True

.github/workflows/ci.yml

@@ -64,7 +64,7 @@ jobs:
           --health-retries 5
 
     env:
-      # A set DB_NAME switches settings.py from SQLite to PostgreSQL.
+      # settings.py reads the database connection from these variables.
       DB_NAME: coderr
       DB_USER: coderr
       DB_PASSWORD: coderr

DEPLOYMENT.md

@@ -241,8 +241,8 @@ der Kernel bei einer Speicherspitze den größten Prozess beendet, also
 ### 3.7 Pakete
 
 ```bash
-sudo apt install -y nginx postgresql postgresql-contrib python3-venv \
-  python3-dev libpq-dev git certbot python3-certbot-nginx fail2ban
+sudo apt install -y nginx postgresql postgresql-contrib postgresql-16-pgvector \
+  python3-venv python3-dev libpq-dev git certbot python3-certbot-nginx fail2ban
 ```
 
 ### 3.8 Datenbank
@@ -271,6 +271,22 @@ der Nutzer falsch angelegt, es fehlt aber nur dieses Recht.
 `CREATEDB` braucht Django, um für Tests eine temporäre Datenbank
 anzulegen.
 
+**pgvector** braucht der Portfolio-Assistent. Die Extension ist nicht
+`trusted`, nur ein Superuser darf sie anlegen. Deshalb legt `postgres` sie
+einmal an, bevor die erste Migration läuft:
+
+```bash
+sudo -u postgres psql -d coderr -c 'CREATE EXTENSION vector'
+sudo -u postgres psql -d template1 -c 'CREATE EXTENSION vector'
+```
+
+Die Migration `assistant_app.0001` findet die Extension dann vor und
+überspringt `CREATE EXTENSION`, `coderr` braucht keine Superuser-Rechte.
+`template1` ist die Vorlage für jede neue Datenbank. Liegt die Extension
+dort, haben auch die Wegwerf-Datenbanken aus 5.5 sie von Anfang an. Ohne
+sie scheitert das Einspielen einer Sicherung als `coderr` an
+`permission denied to create extension "vector"`.
+
 Passwörter erzeugen:
 
 ```bash
@@ -516,8 +532,9 @@ CONTACT_RECIPIENT=<Zieladresse der Formularnachrichten>
 ```
 
 Ohne gesetzte Variablen verhält sich das Projekt wie in der Entwicklung:
-`DEBUG=True`, SQLite, CORS auf `localhost:5500`. Der Produktionsmodus
-entsteht ausschließlich durch diese Datei.
+`DEBUG=True`, CORS auf `localhost:5500`. Pflicht sind nur `SECRET_KEY` und
+`DB_NAME`, ohne sie startet Django nicht. Der Produktionsmodus entsteht
+ausschließlich durch diese Datei.
 
 `benjaminblarr.de` muss in `ALLOWED_HOSTS` und den
 `CSRF_TRUSTED_ORIGINS` stehen bleiben, obwohl Coderr dort nicht mehr
@@ -982,8 +999,13 @@ export PGPASSWORD="$(grep -m1 '^DB_PASSWORD=' .env | cut -d= -f2- | tr -d "\r\"'
 
 ```bash
 createdb coderr_neu
-gunzip -c ~/backups/coderr/<datei>.sql.gz | psql -X -q -v ON_ERROR_STOP=1 -d coderr_neu
+gunzip -c ~/backups/coderr/<datei>.sql.gz | grep -v '^COMMENT ON EXTENSION' \
+  | psql -X -q -v ON_ERROR_STOP=1 -d coderr_neu
 ```
+Das `grep -v` lässt die Zeile `COMMENT ON EXTENSION` weg. Sie darf nur der
+Eigentümer der Extension ausführen, und das ist `postgres`, nicht `coderr`.
+Ohne den Filter bricht das Einspielen an `must be owner of extension vector`
+ab. `restore_probe.sh` filtert genauso.
 
 Bricht das ab: `dropdb coderr_neu`. Die echte Datenbank ist unberührt.
 
@@ -1131,6 +1153,12 @@ gewinnt, nicht der letzte.
 
 **Schema-Rechte ab PostgreSQL 15.** Siehe Abschnitt 3.8.
 
+**pgvector und das Einspielen als `coderr`.** Siehe Abschnitte 3.8 und
+5.5. Vor dem ersten Deploy mit pgvector lokal nachgestellt: Ohne die
+Extension in `template1` scheitert jede Wiederherstellung am
+`CREATE EXTENSION`, mit ihr am `COMMENT ON EXTENSION`. Aufgefallen, bevor
+eine Sicherung gebraucht wurde, nicht danach.
+
 **GitHub-Passwort funktioniert nicht.** Passwort-Anmeldung für
 Git-Operationen ist seit 2021 abgeschaltet. Für private Repos wird ein
 Deploy Key oder ein Personal Access Token benötigt. Bei öffentlichen
@@ -1251,6 +1279,13 @@ Für die Zukunft: Secret Key ab dem ersten Commit in die `.env`.
       Server entfernen. Sonst versucht certbot weiter, ein Zertifikat
       für eine Domain zu erneuern, die es nicht mehr gibt
 
+### Erledigt am 26.09.2026
+
+- [x] `postgresql-16-pgvector` 0.6.0 installiert, ohne Neustart von
+      PostgreSQL. Extension als `postgres` in `coderr` und `template1`
+      angelegt
+
+
 ### Erledigt am 15.09.2026
 
 - [x] Einspielen beider Sicherungsarten in eine Kopie der Datenbank

README.md

@@ -68,17 +68,25 @@ Akademie; everything behind `/api/` is this project.
 
 ```
 coderr_backend/
-├── core/          # Project settings, root URL config, WSGI/ASGI
-├── auth_app/      # Registration, login, profiles
-│   └── api/       # serializers.py, views.py, urls.py, permissions.py
-├── offers_app/    # Offers and offer details
-│   └── api/       # serializers.py, views.py, urls.py, permissions.py, pagination.py
-├── orders_app/    # Orders and order statistics
-│   └── api/       # serializers.py, views.py, urls.py, permissions.py
-├── reviews_app/   # Reviews
-│   └── api/       # serializers.py, views.py, urls.py, permissions.py
-├── base_app/      # Aggregated platform statistics (base-info)
-│   └── api/       # views.py, urls.py
+├── core/               # Project settings, root URL config, WSGI/ASGI
+├── auth_app/           # Registration, login, profiles
+│   └── api/            # serializers.py, views.py, urls.py, permissions.py
+├── offers_app/         # Offers and offer details
+│   └── api/            # serializers.py, views.py, urls.py, permissions.py, pagination.py
+├── orders_app/         # Orders and order statistics
+│   └── api/            # serializers.py, views.py, urls.py, permissions.py
+├── reviews_app/        # Reviews
+│   └── api/            # serializers.py, views.py, urls.py, permissions.py
+├── base_app/           # Aggregated platform statistics (base-info)
+│   └── api/            # views.py, urls.py
+├── contact_app/        # Contact form of the portfolio site
+├── assistant_app/      # Retrieval for the portfolio assistant
+│   ├── api/            # serializers.py, views.py, urls.py
+│   ├── knowledge/      # Knowledge base, one Markdown file per topic
+│   └── management/     # build_index, evaluate_retrieval
+├── embedding_service/  # Standalone FastAPI service that embeds texts
+├── deploy/             # Deployment script and Nginx configuration
+├── compose.yml         # Local PostgreSQL with pgvector
 ├── manage.py
 └── requirements.txt
 ```
@@ -187,10 +195,10 @@ coderr_backend/
 
    The API is now available at `http://127.0.0.1:8000/`.
 
-Without any environment variables set, the project runs in development mode:
-`DEBUG=True` and a local SQLite file. The SQLite fallback cannot hold vector
-data, so the setup above uses PostgreSQL. Production settings are switched on
-purely through the `.env` file.
+The project needs PostgreSQL: `settings.py` stops with an error when
+`DB_NAME` is missing, because the portfolio assistant stores its vectors with
+pgvector. Everything else defaults to development mode (`DEBUG=True`), and
+production settings are switched on purely through the `.env` file.
 
 ---
 
@@ -205,7 +213,7 @@ python manage.py test
 Measure test coverage (target: **≥ 95 %**):
 
 ```bash
-coverage run --source=auth_app,offers_app,orders_app,reviews_app,base_app --omit='*/migrations/*,*/tests/*' manage.py test
+coverage run --source=auth_app,offers_app,orders_app,reviews_app,base_app,assistant_app --omit='*/migrations/*,*/tests/*' manage.py test
 coverage report
 ```
 
@@ -329,6 +337,31 @@ Base path: `/api/`
 
 ---
 
+## Portfolio Assistant
+
+Besides the Coderr API, this backend serves the assistant on my portfolio
+site: visitors ask questions about me and my projects, and the answer comes
+from a knowledge base I maintain instead of being made up. The feature is
+under construction and not live yet.
+
+The current stage covers retrieval only, without a language model:
+
+1. `assistant_app/knowledge/*.md` holds the knowledge base, cut into sections
+   at every `##` heading.
+2. `python manage.py build_index` embeds every section through the
+   [embedding service](embedding_service/README.md) and stores the vectors in
+   PostgreSQL with pgvector.
+3. `POST /api/assistant/` with `{"question": "..."}` returns the five closest
+   sections with their cosine similarity.
+4. `python manage.py evaluate_retrieval` checks a fixed list of questions, in
+   German and English, against the sections they should find.
+
+The endpoint answers `503` unless `ASSISTANT_ENABLED=True` is set, so a server
+without the embedding service stays safe. Locally the embedding service runs
+in a second terminal, see its README.
+
+---
+
 ## Deployment
 
 The live instance runs on a VPS I set up and maintain myself: Ubuntu 24.04,

assistant_app/admin.py

@@ -0,0 +1,20 @@
+"""Admin configuration for the assistant app."""
+
+from django.contrib import admin
+
+from .models import KnowledgeChunk
+
+
+@admin.register(KnowledgeChunk)
+class KnowledgeChunkAdmin(admin.ModelAdmin):
+    """Read-only view of the indexed knowledge base."""
+
+    list_display = ('source', 'position', 'heading')
+    list_filter = ('source',)
+    search_fields = ('heading', 'content')
+    fields = ('source', 'position', 'heading', 'content')
+    readonly_fields = fields
+
+    def has_add_permission(self, request):
+        """Chunks are only created by the build_index command."""
+        return False

assistant_app/api/serializers.py

@@ -0,0 +1,25 @@
+"""Serializers for the assistant app."""
+
+from rest_framework import serializers
+
+from assistant_app.models import KnowledgeChunk
+
+
+class QuestionSerializer(serializers.Serializer):
+    """Validates a visitor question."""
+
+    question = serializers.CharField(max_length=500)
+
+
+class ChunkResultSerializer(serializers.ModelSerializer):
+    """One retrieved knowledge section with its similarity score."""
+
+    similarity = serializers.SerializerMethodField()
+
+    class Meta:
+        model = KnowledgeChunk
+        fields = ('source', 'heading', 'content', 'similarity')
+
+    def get_similarity(self, chunk):
+        """Turn the cosine distance into a similarity between -1 and 1."""
+        return round(1 - chunk.distance, 4)

assistant_app/api/urls.py

@@ -0,0 +1,9 @@
+"""URL configuration for the assistant app."""
+
+from django.urls import path
+
+from .views import AssistantView
+
+urlpatterns = [
+    path('assistant/', AssistantView.as_view(), name='assistant'),
+]

assistant_app/api/views.py

@@ -0,0 +1,54 @@
+"""API views for the assistant app."""
+
+import logging
+
+from django.conf import settings
+from drf_spectacular.utils import extend_schema
+from rest_framework import status
+from rest_framework.permissions import AllowAny
+from rest_framework.response import Response
+from rest_framework.views import APIView
+
+from assistant_app.embedding_client import EmbeddingServiceError
+from assistant_app.retrieval import search
+
+from .serializers import ChunkResultSerializer, QuestionSerializer
+
+logger = logging.getLogger(__name__)
+
+
+class AssistantView(APIView):
+    """Returns the knowledge sections that best match a question.
+
+    No language model is involved yet: the response shows exactly what
+    retrieval found, so its quality can be measured on its own.
+    """
+
+    permission_classes = [AllowAny]
+    authentication_classes = []
+
+    @extend_schema(exclude=True)
+    def post(self, request):
+        """Validate the question and return the closest sections.
+
+        Excluded from the generated API schema: this endpoint belongs to
+        the portfolio and is not part of the Coderr API.
+        """
+        if not settings.ASSISTANT_ENABLED:
+            return self._unavailable()
+        serializer = QuestionSerializer(data=request.data)
+        serializer.is_valid(raise_exception=True)
+        try:
+            chunks = search(serializer.validated_data['question'])
+        except EmbeddingServiceError as error:
+            logger.error("Assistent: %s", error)
+            return self._unavailable()
+        results = ChunkResultSerializer(chunks, many=True).data
+        return Response({'results': results})
+
+    def _unavailable(self):
+        """Answer 503 when the assistant is switched off or broken."""
+        return Response(
+            {'detail': 'Der Assistent ist gerade nicht verfügbar.'},
+            status=status.HTTP_503_SERVICE_UNAVAILABLE,
+        )

assistant_app/apps.py

@@ -0,0 +1,10 @@
+"""App configuration for the assistant app."""
+
+from django.apps import AppConfig
+
+
+class AssistantAppConfig(AppConfig):
+    """Default configuration for the assistant app."""
+
+    name = 'assistant_app'
+    verbose_name = "Assistent"

assistant_app/embedding_client.py

@@ -0,0 +1,50 @@
+"""Client for the local embedding service in embedding_service/."""
+
+import httpx
+from django.conf import settings
+
+MAX_TEXTS_PER_REQUEST = 64
+QUERY_TIMEOUT = 5
+PASSAGE_TIMEOUT = 120
+
+
+class EmbeddingServiceError(Exception):
+    """The embedding service is unreachable or rejected the request."""
+
+
+def embed_query(text):
+    """Return the vector for one visitor question."""
+    return _post('query', [text], QUERY_TIMEOUT)[0]
+
+
+def embed_passages(texts):
+    """Return one vector per knowledge passage, in input order.
+
+    The service accepts at most 64 texts per request, so longer lists
+    are sent in batches.
+    """
+    vectors = []
+    for start in range(0, len(texts), MAX_TEXTS_PER_REQUEST):
+        batch = texts[start:start + MAX_TEXTS_PER_REQUEST]
+        vectors.extend(_post('passage', batch, PASSAGE_TIMEOUT))
+    return vectors
+
+
+def _post(kind, texts, timeout):
+    """Send one request to /embed and return its vectors."""
+    try:
+        response = httpx.post(
+            f"{settings.EMBEDDING_SERVICE_URL}/embed",
+            json={'kind': kind, 'texts': texts},
+            timeout=timeout,
+        )
+    except httpx.HTTPError as error:
+        raise EmbeddingServiceError(
+            f"Embedding service not reachable: {error!r}"
+        ) from error
+    if response.status_code != 200:
+        raise EmbeddingServiceError(
+            f"Embedding service answered {response.status_code}: "
+            f"{response.text}"
+        )
+    return response.json()['vectors']