Skip to content

Feat/working branch - #159

Open
simonvanlierde wants to merge 1085 commits into
mainfrom
feat/working-branch
Open

Feat/working branch#159
simonvanlierde wants to merge 1085 commits into
mainfrom
feat/working-branch

Conversation

@simonvanlierde

Copy link
Copy Markdown
Contributor

feat/working-branch → main: security hardening, MFA, brand refresh, and repo restructure

TL;DR

A ~2.5-month working branch (728 commits) intended as a single squash-merge. It hardens the auth/account-security stack, adds TOTP MFA with recovery codes, removes dead data-model surface (organizations, newsletter), rebuilds the public homepage around live stats, rebrands toward R9lab / ReLab, and restructures the two frontend subrepos.

Reading the diff: the raw stat (1507 files, +115k/−170k) is misleading. Most of it is two directory renames (frontend-webwww, frontend-appapp), regenerated files (api.generated.ts, OpenAPI schema), and brand assets. The real logic change surface is a fraction of that — the themes below are where it lives.

Why squash

The branch grew organically over ~2.5 months, so its intermediate history is noisy. Squashing collapses it into one clean, coherent commit on main. The changes were checked against main across multiple angles — correctness, security, test coverage, performance, and over-engineering — before landing.


Highlights by area

🔐 Auth & account security (largest area)

The auth module was substantially reworked — service layer split into focused units (lifecycle, login_flow, mfa_service, session_flow, token_store, email/*, oauth/*) and hardened end to end.

  • TOTP MFA — enrollment + challenge flow, MFA account management in the profile, and MFA-completion enforcement on OAuth login handoffs.
  • Recovery codes — accept recovery codes for MFA login and for MFA disable; replaced the old email-based MFA reset; hardened code lifecycle.
  • OAuth link/unlink — step-up re-auth required to unlink a social login (password prompt in-app), provider-bound OAuth state JWTs, email notifications on link/unlink and welcome mail for social signups.
  • Non-enumerable registration — registration no longer reveals whether an email exists, and auto-login after signup was removed to keep the flow non-enumerable.
  • Session/token lifecycle — cookie names single-sourced, sessions revoked before account deletion, refresh tokens keyed by fingerprint, closed token-lifecycle gaps, caches bypassed for credentialed/no-store responses.
  • Passwords — common-password blocklist (3000 entries) via a dedicated checker; password-hashing service extracted.
  • Audit logging — authentication, session, authorization-denial, and rate-limit events logged with sanitization.

🗄️ Data model & migrations

17 new migrations. Notable removals and reshapes:

  • Removed organizations and newsletter subscribers (dead/unused surface).
  • Split product roles; normalized rows before the role CHECK constraint.
  • Circularity columns → JSONB; removed product dismantling notes/times.
  • Upload quota: per-user quota ledger + upload size metadata, enforced on product uploads.
  • Email canonicalization (email_canonical), has_usable_password, MFA/recovery tables.
  • Ops: autovacuum tuning for high-churn tables, FK indexes on material↔product links.

📁 File storage hardening

Paginated media lists, real zip-size enforcement, malware scanning extended to device thumbnails, quota owner keying fixed, sensitive-key rejection in local storage, JSON-recursion guard, dotted filenames accepted, uploaded filenames validated against MIME type.

🌐 Public site (www, formerly frontend-web)

Homepage rebuilt: new Hero, SiteFooter, privacy page, and a StatsPanel fed by new public stats API (/stats/totals, /stats/categories, /stats/series) — monthly activity chart with per-category part counts. New brand/token CSS, theme handling, 404 page, and a Vitest suite for the new components.

📷 RPi camera plugin

WebSocket relay bounded against unresponsive devices, camera-flapping-to-offline fixed, livestream/recording lifecycle corrected, device key kept on the LAN, response ownership verified, device-assertion lifetime capped, circuit breaker made Redis-only and atomic.

📱 App (React Native, formerly frontend-app)

MFA challenge screen + pending-login routing, OAuth callback via URL fragment, new-product drafts can set type/material before first save, "already live" stream dialog, single-flight auth/MFA submits, gallery no longer silently deletes unrenderable images, static-background refactor (parallax scaffolding removed), plus a large sweep of review-driven bug fixes.

🎨 Brand → R9lab / ReLab

R9lab logo/wordmark/flask marks wired into www, docs, and app UIs; email templates rebranded with hosted wordmark; README wordmark. Asset generators live in assets/logo-src/.

Copy pass across the app UI, docs, and www: unified sign-in/sign-out terminology, sentence-case titles and buttons, plainer error and empty-state messages, and less boilerplate — with tests and e2e specs updated to match.

Note: this is the incremental brand pass. A full code/name rename (siteMeta, package names, app name) is deliberately deferred.

🏗️ Infra / CI / deploy

  • Opt-in restic backups and Google/GitHub OAuth; secret inventory split into required/optional; startup warning when secrets still hold deploy placeholders.
  • ClamAV service added to the deploy stack (topology documented).
  • CI: required-jobs gate derived from needs, OpenAPI/app-codegen freshness gates, JUnit → Codecov Test Analytics, per-PR RN a11y lint, Playwright browser caching.
  • Supply-chain: reject Python deps newer than 3 days; hardened HTTP client for auth integrations; DB/Redis TLS hardening; trusted-proxy CIDR validation; rightmost X-Forwarded-For.
  • OpenSSF Scorecard badge; Cloudflare edge rules for the public stats endpoint.

🧭 Repo restructure

  • frontend-webwww
  • frontend-appapp

Renames (with history preserved) touch devcontainers, CI, docs, and configs — the bulk of the file count.


Verification

Each subrepo's loop (just fixjust checkjust test) was run as changes landed. Beyond the test suites, the diff was reviewed across correctness, security, test-coverage, performance, and simplification angles. New migrations include a downgrade/upgrade round-trip test, and new logic (stats, auth flows, storage, MFA) ships with regression tests.

@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

Comment thread backend/Dockerfile.backups Fixed
Comment thread app/src/services/api/auth/authUser.ts Fixed
@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

Comment thread app/scripts/redact_api.mjs Fixed
Comment thread app/src/features/auth/useOAuthLogin.ts Fixed
Comment thread docs/e2e/api-reference.spec.ts Fixed
Comment thread app/src/services/api/auth/authUser.ts Fixed

@github-advanced-security github-advanced-security AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

Comment thread backend/app/core/lifecycle.py Fixed
Comment thread backend/app/api/plugins/rpi_cam/websocket/connection_manager.py Fixed
Comment thread backend/app/api/plugins/rpi_cam/websocket/connection_manager.py Fixed
Comment thread backend/app/api/auth/services/email/providers.py Fixed
Comment thread backend/app/api/auth/services/email/providers.py Fixed
Comment thread backend/app/api/auth/services/email/providers.py Fixed
Comment thread backend/Dockerfile.backups Fixed
simonvanlierde added a commit that referenced this pull request Jul 12, 2026
- guard the getUser slot by sequence instead of promise identity (CodeQL missing-await)
- read api.generated.ts directly instead of existsSync-then-read (CodeQL TOCTOU)
- drop redundant location.hash guard and document server-side OAuth enforcement (CodeQL bypass FP)
- anchor the Scalar request-blocking regex in docs e2e (CodeQL regex anchor)
- catch Exception instead of BaseException in shutdown steps (code-quality)
- add stamp-freshness HEALTHCHECK to the backups image (Trivy DS-0026)
- use docstring bodies for protocol stubs instead of bare ellipses (code-quality)
- remove unused Union import from alembic migration (code-quality)
simonvanlierde added a commit that referenced this pull request Jul 12, 2026
- break auth-services<->core import cycle: inline redis bool coercion in blocklist_store
- break auth models<->user_database cycle: move get_user_db/get_auth_async_session to user_manager
- rpi-cam ws auth: hash camera rate-limit bucket via rate_limit_bucket_key like all other callers
- rpi-cam models: replace non-exhaustive match with dict lookup in to_http_error
- connection_manager: sanitize camera_id in logs; pairing: drop unused _STATUS_WAITING
- support_services/test_upload_security: replace del-parameter idiom (noqa B027 / pass)
- test_email_providers: match provider URLs with startswith instead of substring
- codeql config: enable AlertSuppression packs; exclude py/unused-import (ruff owns it) and informational py/cyclic-import
- suppress verified false positives inline (token fingerprint sha256, secret-name logging, masked email logging)
Comment thread backend/app/api/auth/services/email/service.py Dismissed
Comment thread backend/app/api/auth/services/email/service.py Dismissed
Comment thread backend/app/api/auth/services/rate_limiter.py Fixed
Comment thread backend/app/api/auth/services/token_store.py Dismissed
Comment thread backend/app/core/secrets.py Dismissed
Comment thread scripts/env_policy.py Dismissed
Comment thread scripts/env_policy.py Dismissed
Comment thread backend/app/api/auth/services/blocklist_store.py Dismissed
Comment thread backend/tests/unit/plugins/rpi_cam/conftest.py Dismissed
Comment thread backend/alembic/env.py Dismissed
Comment thread backend/app/api/auth/services/oauth/login.py Dismissed
simonvanlierde added a commit that referenced this pull request Jul 14, 2026
- guard the getUser slot by sequence instead of promise identity (CodeQL missing-await)
- read api.generated.ts directly instead of existsSync-then-read (CodeQL TOCTOU)
- drop redundant location.hash guard and document server-side OAuth enforcement (CodeQL bypass FP)
- anchor the Scalar request-blocking regex in docs e2e (CodeQL regex anchor)
- catch Exception instead of BaseException in shutdown steps (code-quality)
- add stamp-freshness HEALTHCHECK to the backups image (Trivy DS-0026)
- use docstring bodies for protocol stubs instead of bare ellipses (code-quality)
- remove unused Union import from alembic migration (code-quality)
simonvanlierde added a commit that referenced this pull request Jul 14, 2026
- break auth-services<->core import cycle: inline redis bool coercion in blocklist_store
- break auth models<->user_database cycle: move get_user_db/get_auth_async_session to user_manager
- rpi-cam ws auth: hash camera rate-limit bucket via rate_limit_bucket_key like all other callers
- rpi-cam models: replace non-exhaustive match with dict lookup in to_http_error
- connection_manager: sanitize camera_id in logs; pairing: drop unused _STATUS_WAITING
- support_services/test_upload_security: replace del-parameter idiom (noqa B027 / pass)
- test_email_providers: match provider URLs with startswith instead of substring
- codeql config: enable AlertSuppression packs; exclude py/unused-import (ruff owns it) and informational py/cyclic-import
- suppress verified false positives inline (token fingerprint sha256, secret-name logging, masked email logging)
- gate the smoke harness itself (justfile, docker-smoke action) on the
  orchestration filter, and backend/pyproject.toml + .python-version on the
  backend filter, so harness and image-input edits still smoke something
- add merge_group to the security workflow and treat it like a push to main, so
  an enabled merge queue cannot stall on a required security check
- drop the now-dead run-containers output and guard the smoke matrix against an
  empty smoke-legs value
- restore cancel-in-progress for push runs while keeping schedule and
  merge-queue runs uncancellable
The suite covered these steps as fragments, each from a fresh login as the
seeded superuser — which never proved they compose, and never proved a normal
contributor can do them at all, since a superuser passes every ownership check
by construction.

- add a journey spec running as `alice`, a seeded verified non-superuser:
  create, add a dimension, upload an image, add a child component, then read
  the result back as an anonymous visitor
- assert the guest cannot see Edit Product, so the block cannot quietly pass if
  the context ever inherits a session
- let loginAndReachProducts take credentials, defaulting to the admin
- wait for the picked file to reach the form before saving; setFiles returns
  early, so both upload tests could save a product with no images and never
  issue the upload

Registration is not chained in: a password signup is unverified, product
creation requires verification, and the E2E stack has no mail transport, no
stored token, and no admin route to set is_verified. The spec header records
this; auth.spec.ts still covers signup to the verify-email prompt.
Nothing clears the database between tests in a run — only teardown does — so
every test that creates a product leaves it behind. Reading a row straight off
the rendered list worked until enough accumulated to push the target off the
first page, at which point unrelated tests started failing.

- search by name before picking a row, in both product-open helpers
- fix openGalleryLightbox, which looked for "View image N"; the trigger is
  labelled `View ${altText}`, so both lightbox tests had been failing outright
- assert the capture screen is reached before filling it. A swallowed press
  otherwise surfaced 60s later as "Create component not found", pointing at a
  screen the run never reached
- record the swallowed press in ProductComponents: reproduced ~2 in 8 under
  4-way parallel load, recovers on a second press every time, and not the id
  guard — `Product ID: N` renders from that same object at the time of the
  lost click

Also add the three Track E items the cutover runbook did not carry: rehearsing
on staging, confirming validate-result is a required check, and reading the
release-please proposal before merging it.
…age logs

- raise RelayCommandRejectedError for explicit 4xx relay error payloads and map it at the requester; other errors keep the 503 contract
- log relay-listener redis failures once until recovery instead of once per second
- replace the full ProductRead nested in MaterialProductLinkReadWithinMaterial with a ProductSummary (id, name, thumbnail_url)
- regenerate OpenAPI schemas and the app client
- replace inline JSX closures with useCallback or extracted list components
- render numeric button children inside Text to avoid a native crash
- keep targeted biome ignores only where the rule is genuinely inapplicable
process_image_for_storage(path)

assert path.read_bytes() == before, "GIF was re-encoded despite needing no processing"
assert PILImage.open(path).n_frames == 3
process_image_for_storage(path)

assert path.read_bytes() == before, "animated original was re-encoded despite carrying EXIF"
assert PILImage.open(path).n_frames == 3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants