Document intelligence for course requirements, deadlines and obligations — with versioned source evidence behind the results.
CourseFlow turns assignment briefs, handbooks, slides and announcements into structured requirements and a cross-course study plan. Users can inspect the supporting excerpt, see conflicting deadlines, and track what changes when a source is updated.
Python · FastAPI · PostgreSQL / pgvector · NVIDIA Nemotron · Next.js · Docker
Run the demo · Architecture · Evaluation · Code walkthrough · Limitations
The default demo uses synthetic course documents and recorded model outputs. Replay passes through the application’s evidence gates and deterministic rules; it does not demonstrate fresh inference on an arbitrary uploaded PDF.
A deadline can disagree across documents. A handbook can contain a mandatory review that an assignment brief omits. A revised announcement can invalidate a previously correct plan.
CourseFlow makes those relationships inspectable:
| Capability | Result |
|---|---|
| Structured extraction | Assessments, deadlines, requirements and obligations linked to source evidence |
| Material discovery | Requirements mapped to retrieved teaching material, with source locations |
| Evidence inspection | Stored excerpts with document version and page, slide or section |
| Conflict detection | Disagreeing deadlines shown together rather than silently resolved |
| Change propagation | A simulated LMS sync updates affected intelligence and the study plan |
| Study planning | Cross-course next steps with dependencies and a reason for their order |
This is a working prototype, not a production LMS integration.
| Area | Current implementation |
|---|---|
| Reproducible demo | Two fictional Northbridge University courses; recorded model outputs and embeddings |
| Live inference | Evaluation runners and an optional Analyze live with Nemotron action for one demo assessment; credentials required |
| Document ingestion | Backend import path; see the development guide. This is separate from an end-to-end upload UI |
| LMS connector | Synthetic DemoConnector; no live Moodle connection |
| Access | One demo learner; no authentication or user-level authorization |
| Deployment | Docker demo stack; no public hosted URL included |
A modular monolith keeps document provenance, application state, vector search and full-text search in one PostgreSQL-backed system.
flowchart TD
A["Sources and versions"] --> B["Source spans and chunks"]
B --> C["LLM structured extraction"]
B --> D["Vector and full-text retrieval"]
C --> E["Evidence validation"]
D --> F["LLM material verification"]
F --> E
E --> G["Structured course state"]
G --> H["Dependency, conflict and planning rules"]
H --> I["Next.js interface"]
G --> I
Retrieval supports requirement-to-material mapping. CourseFlow presents structured state and evidence rather than a conversational question-answering interface.
| Component | Responsibility |
|---|---|
| FastAPI, async SQLAlchemy, Pydantic, Alembic | API, validated contracts, persistence and migrations |
| PostgreSQL + pgvector | Relational state, embeddings and full-text search |
| NVIDIA Nemotron through Nebius Token Factory | Structured extraction and material verification |
| Qwen3-Embedding-8B | Document and query embeddings |
| Next.js, React, TypeScript | Requirements, source evidence, conflicts and study-plan views |
| LLM responsibilities | Deterministic application responsibilities |
|---|---|
| Propose assessments, requirements and obligations | Validate schemas, evidence references and supported values |
| Select relevant material from retrieved candidates | Reject invented candidate IDs and invalid evidence |
| Return structured model output | Compute dependencies, conflicts, supersession and the plan |
The orchestration is a fixed, traced workflow, not an autonomous tool-using agent loop. Each course has a WorkflowRun; failed steps stop processing and record an error category. Source changes scope reprocessing to affected versions.
System context · Data contract · Workflow implementation
Results below come from committed reports on frozen synthetic datasets. They measure specific tasks, not overall product accuracy or performance on real university documents.
The retrieval/v1 benchmark contains 12 positive queries and 2 invariant-only probes, covering keyword, paraphrase and mixed queries, stale versions and cross-course results. It replays recorded embeddings through the application search service; the scored runs make no live provider calls.
| Mode | Recall@1 | Recall@3 | Recall@5 | MRR@10 |
|---|---|---|---|---|
| Vector | 9/12 | 11/12 | 12/12 | 0.8403 |
| Full-text | 9/12 | 11/12 | 12/12 | 0.8500 |
| Hybrid / RRF | 9/12 | 12/12 | 12/12 | 0.8611 |
All three modes passed the two invariant probes. A predefined selection rule chose hybrid as the default. Two replay runs returned identical metrics and hit lists.
Interpretation: hybrid recovered one additional positive case in the top three on this corpus. This small benchmark was used for default selection; it does not establish statistical significance or independent generalization performance.
Per-case results · Dataset and reproduction instructions · Selection decision
Recorded live Nebius experiments on the frozen Northbridge golden dataset report:
| Task | Reported result | Evidence |
|---|---|---|
| Deadline extraction | Exact match 6/6; zero false verified assessments in the reported run | Assessment report |
| Requirement extraction | Verified labelled recall 11/11 | Claims report |
| Obligation extraction | Verified labelled recall 2/3; one labelled obligation missed | Claims report |
Requirement and obligation labels are incomplete, so these reports do not establish precision or F1. Extraction experiments use live model calls; offline demo replay and retrieval replay are different execution modes. Reports include the model, prompt/gate versions and usage details.
Vector similarity captures semantic matches; PostgreSQL full-text search complements it for exact terminology. Reciprocal Rank Fusion combines their ranks, avoiding direct addition of incompatible raw scores. Search scores are ranking signals, not probabilities of correctness.
Search is scoped to current resource versions within a course and a compatible embedding space. The vector path uses exact cosine search, with no ANN/HNSW index. Course scoping is a data boundary, not user authorization.
The span-pack/v1 chunker preserves source offsets, keeps consecutive slide text and notes together, and attaches headings to following content. Oversized spans are split at preferred text boundaries.
Its target is 800 estimated tokens, with a hard maximum of 1,000 and overlap up to 120. The estimate is ceil(characters / 4), not a tokenizer measurement. These are versioned prototype settings, not universal optimal values.
Evidence links resolve through stored source spans and resource versions. The application checks citation location and applicable value constraints rather than treating model confidence as verification.
Quote presence is not a general semantic entailment test: an authentic quote can still be misinterpreted. Model quality, validation coverage and human review remain relevant.
Conflicting deadlines remain visible with their sources. The planner uses the earlier date conservatively and does not present that choice as resolution of the conflict. Dependencies, conflict handling, supersession and planning use versioned deterministic rules.
Retrieval design · Ingestion design · Architecture decisions
| Condition | Handling |
|---|---|
| Unknown evidence reference or missing quote | Reject the unsupported result |
| Multiple matching quote locations | Treat the evidence as ambiguous rather than choosing silently |
| Permissive wording or unsupported obligation category | Route affected claims to needs_review |
| Invented material candidate ID | Reject through the candidate subset gate |
| Inconsistent retrieval index | Refuse search rather than return a partial, inconsistent corpus |
| Workflow step failure | Stop the workflow and retain the recorded failure category |
Source text is passed as delimited data, with per-call nonces and no model tools in the extraction flow. The committed injection example and control are useful regression checks, not proof of general prompt-injection immunity.
The application does not submit coursework or write to an LMS. Demo reset and sync controls do modify local demo state.
Prerequisite: Docker with Compose v2. Run from a fresh clone:
git clone https://github.com/eseeyuh/courseflow.git
cd courseflow
docker compose -f infra/demo/compose.yml up -d --build --waitOpen http://localhost:3000. Startup migrates the database and initializes the synthetic demo state. The first build requires network access to fetch dependencies; the default model replay needs no API key or live inference calls.
Try the three-minute walkthrough: inspect a requirement and its evidence, compare conflicting deadlines, then trigger the simulated sync and inspect the changes.
Stop the stack:
docker compose -f infra/demo/compose.yml downOptional live analysis: set NEBIUS_API_KEY, NEBIUS_BASE_URL, MODEL_FAST and MODEL_STRONG in your local root .env, then start with:
docker compose --env-file .env -f infra/demo/compose.yml up -d --build --waitUse Analyze live with Nemotron on Applied Machine Learning’s Coursework 1. This makes a provider call and incurs usage charges; it is not a complete live LMS workflow. Keep credentials out of git.
Demo controls are unauthenticated. Keep the stack local unless you deliberately provide an appropriate deployment boundary. Configuration: development and demo environment template.
The development guide covers Python/uv and Node prerequisites, document import, migrations, API docs and troubleshooting. With dependencies installed and the test database available:
docker compose up -d db
cd backend
uv sync --locked
uv run pytest
uv run ruff check .
uv run ruff format --check .From frontend/:
npm ci
npm run lint
npm run typecheck
npm run buildBackend provider tests use scripted responses, not real model calls. With the development app running, npx playwright test exercises the demo golden path. Test outcomes are distinct from live-model evaluation; this README does not imply a fresh test run on every checkout.
| Start here | What to inspect |
|---|---|
| Ingestion | Source import and versioning |
| Chunking | Structural boundaries, size estimates and offsets |
| Search | Course/version scope, index verification and search modes |
| Rank fusion | Deterministic RRF and tie-breaking |
| Claim evidence location | Quote matching and original source offsets |
| Workflow | Fixed steps, traces and failure propagation |
| Evaluation | Frozen datasets, replay, metrics and reproduction |
- Synthetic data and simulated LMS: no validated real-university deployment or live Moodle connector.
- Replay-first demo: arbitrary new documents do not have a complete live upload-to-result UI.
- No authentication: course-scoped retrieval does not provide multi-user access control.
- Small evaluations: incomplete extraction labels and no broad generalization claim.
- Material mapping is broad: several slides can be selected; stored mapping ranks do not reliably distinguish the best slide.
- Incomplete recall: one of three labelled obligations was missed in the reported extraction run.
- Desktop-first interface: mobile presentation is not tuned.
Next priorities are a complete live document-processing path, independent evaluation on additional documents, and server-enforced access control. These are planned improvements, not current capabilities. Detailed gaps: post-hackathon accuracy work.
Developed for the Nebius × NVIDIA Global AI Hackathon, Best Apps & Agents. Repository owner: eseeyuh. See contributors and commit history for recorded contributions.
