Helm charts and install scripts for the services running on my home k3s
cluster (10 nodes: 3 control-plane, 7 workers). External exposure goes
through infra/traefik/ — the single ingress choke point,
its ACME DNS-01 certresolver holding a *.zachd.duckdns.org wildcard cert
(no cert-manager involved), and its access log — with the DuckDNS side of
that credential in infra/duckdns/. Some services are
additionally published on diemer.codes via
infra/cloudflared/.
Each subfolder is a standalone project with its own chart, docs, and
install/upgrade scripts. Per-project secrets live in a gitignored
values.local.yaml alongside the tracked values.yaml.
The exception is scripts/k3s/, which isn't a project: it
operates on the nodes themselves (health, disk cleanup, rolling restarts, OS
and k3s upgrades) over tailscale ssh, so it lives at the repo root.
plans/ is the other non-project folder: designs that are decided but
not yet deployed, usually blocked on hardware or a maintenance window. A plan is
deleted once its work ships and the reasoning has moved into the READMEs it
touched.
Some projects live in their own repo and are tracked here as submodules, so this
repo stays the full index of what runs on the cluster. Clone with
git clone --recurse-submodules; an existing clone catches up with
git submodule update --init. Then enable the tracked hooks, once per clone —
git config core.hooksPath .githooks — they keep the submodule worktrees on
the recorded pins after every pull, checkout and rebase
(scripts/submodules-refresh.sh) and refuse commits that would move a pin
backwards or to a commit GitHub has never seen (.githooks/pre-commit). The
pins mean "what shipped", and only scripts/sync-submodules.sh moves them.
Every chart in this repo, deployed:
Real command output, not mockups —
docs/capture/termshot.py runs the command, parses
the ANSI it prints, and paints it into the window above.
| Folder | What it is | Docs |
|---|---|---|
life/archiveteam-warrior/ |
Archive Team volunteer worker using spare capacity; lowest scheduling priority with a bounded graceful drain on preemption. | README |
minecraft/ |
Prominence II: Hasturian Era (Fabric 1.20.1) via the itzg/minecraft chart. BlueMap + Discord Integration add-ons, mc-backup sidecar. |
README bootstrap in values.yaml, ADDON_SETUP, CLIENT_SETUP |
minecraft/claude-bridge/ |
Sandboxed Claude Code in a pod — players type /claude <q>, the bridge tails the server log + replies via RCON, can teleport on request, logs feature requests to FEEDBACK.md. |
minecraft/claude-bridge/README |
minecraft/claude-mod/ |
Tiny server-side Fabric mod that registers /claude <prompt> via Brigadier and prints a recognizable line for claude-bridge to pick up. Sideloaded into the PVC via install.sh. |
minecraft/claude-mod/README |
games/terraria/ |
Calamity adventure on tModLoader, with Magic Storage, Recipe Browser, Boss Checklist and Ore Excavator. In-game Claude reads progression, gear and recipes and can teleport the caller. Large Classic world; LAN/tailnet TCP 7777. | games/terraria/README |
discord/vocard/ |
Vocard music bot + Lavalink + MongoDB — slash-command music player for voice channels. Bot-only (no dashboard). | discord/vocard/README |
discord/smitele-bot/ → submodule |
Smite-le — a Wordle-shaped Smite guessing game bot, plus the daily Hi-Rez match data collector that was previously a Windows Task Scheduler job. The corpus lives on an SMB share from the NAS (ReadWriteMany: the CronJob writes it, the bot reads it); the bot's own caches sit on a small PVC. Chart + source live in zdiemer/smitele-bot. | zdiemer/smitele-bot README |
games/steam-headless/ |
Steam + Sunshine on the Radeon 680M node, streamed to Moonlight (iOS, TV, laptop). LAN/tailnet only. | games/steam-headless/README |
games/cloud-game/ |
CloudRetro — server-side libretro emulation (NES/SNES/GBA/N64/PS1) streamed to the browser over WebRTC; shared room links give each friend a controller. Curated ROMs unzipped from the NAS share. LAN/tailnet only (UDP). | games/cloud-game/README |
games/romm/ |
RomM — self-hosted ROM manager + in-browser EmulatorJS player, library mounted read-only over SMB from the NAS. | games/romm/README |
games/smt-imagine/ |
Shin Megami Tensei IMAGINE — the 2007 MMO, shut down 2016, back as a solo server: COMP_hack 4.12.2 (lobby/world/channel in one pod, SQLite) built from the last surviving source tarball, since upstream's GitHub org is empty. Client data staged onto a PVC from a ReIMAGINE client, never in git. Three game ports on the node IPs like Minecraft; account manager behind Authelia at smt.zachd.duckdns.org. Tailnet only. |
games/smt-imagine/README |
games/ra2/ |
The owner's Red Alert 2 and Yuri's Revenge static recompilations to WebAssembly, built from NAS executables and served with read-only NAS assets at ra2.zachd.duckdns.org and yuri.zachd.duckdns.org. Tailnet only. |
games/ra2/README |
games/ror2/ |
Experimental unofficial Risk of Rain 2 server. Windows game files come from the NAS over the existing read-only SMB claim; Proton and the companion plugin run in one pod. UDP 7777 is reachable at rain.zachd.duckdns.org from the tailnet. |
games/ror2/README |
games/ffxiv-1x/ |
FINAL FANTASY XIV 1.23b — the original 2010 game, erased by A Realm Reborn — as a private server: Garlemald Server (the Rust port of Project Meteor), four binaries on one SQLite file, built from upstream git at a pinned commit. Three game ports on the node IPs, no Ingress: the client needs an IP. 75 of 524 quests scripted upstream; CONTENT.md is the inventory. Tailnet only. |
games/ffxiv-1x/README |
games/gamedex/ → submodule |
Gamedex — searchable browser for the Games Master List spreadsheet, mirrored live from a Dropbox shared link. Faceted search, no auth, PII columns stripped. Chart + source live in zdiemer/gamedex; the pin here is the deployed commit. | zdiemer/gamedex README |
finance/money/ → submodule |
money — self-hosted personal finance dashboard: a Monte Carlo retirement/FI planning engine, interactive scenarios, and an auto-syncing monthly net-worth tracker (SimpleFIN + real home/vehicle valuations + collectibles). Gated behind Authelia at money.zachd.duckdns.org; single user, DuckDNS-only, personal numbers never committed to the (public) image. |
zdiemer/money README |
media/jellyfin/ |
Jellyfin — media server for /mnt/vault/media (NFS, read-write), with VAAPI transcoding able to fail over between the Vega 8 and Radeon 680M nodes. Not behind Authelia (TV/phone apps need the raw API); LAN NodePort :30096 for smart TVs. |
media/jellyfin/README |
media/jellyseerr/ |
Jellyseerr — request/discovery frontend; signs users in with their Jellyfin account and hands approved requests to Radarr/Sonarr. | media/jellyseerr/README |
media/arr/ |
Prowlarr + Sonarr + Radarr + qBittorrent-inside-gluetun (PIA, port forwarding, real killswitch), with a decluttarr watchdog that blocklists dead downloads so the *arrs re-search instead of stalling. Acquisition automation behind Jellyfin: imports are hardlinks into the shared NFS media volume. Admin UIs behind Authelia forward-auth. The VPN pod deliberately bypasses infra/egress-proxy. |
media/arr/README |
media/bitmagnet/ |
bitmagnet — a self-hosted BitTorrent DHT crawler that builds its own Torznab index in Postgres and serves it to Prowlarr. No tracker site, account, or Cloudflare in the path, which is why it survives the blocks that kill public indexers. Index PVC is deliberately excluded from k8up (a re-crawlable cache). UI behind Authelia forward-auth — bitmagnet has no login of its own. | media/bitmagnet/README |
auth/authelia/ |
Authelia — OIDC provider + (future) Traefik forward-auth. Shared login for every service in the cluster. | auth/authelia/README |
docs/paperless-ngx/ |
Paperless-ngx — self-hosted document management with OCR + full-text search. Bundles Postgres, Redis, Tika, and Gotenberg inline. | docs/paperless-ngx/README |
docs/stirling-pdf/ |
Stirling PDF — locally-processed toolkit for ~50 PDF operations (merge/convert/OCR/sign/redact). Gated behind Authelia forward-auth at the Traefik ingress. | docs/stirling-pdf/README |
web/kelsey-green/ |
kelsey.green — static Astro site, no image of our own: git-sync pulls the CI-built deploy branch and nginx serves it. Public via a Cloudflare tunnel (outbound-only) as well as the usual DuckDNS ingress. |
web/kelsey-green/README |
web/rachel-freeman/ |
rachelfreeman.zachd.duckdns.org — an online art store for originals and prints: Payload CMS admin + Next.js storefront in one image, Postgres, and Stripe Checkout. Prices live in Postgres, not in Stripe, so there is one catalogue and nothing to keep in sync. The only chart here whose GHCR package is private. | web/rachel-freeman/README |
web/old-diemer-codes/ → submodule |
old.diemer.codes — the 2019 Create React App personal site, kept exactly as it was. Inverts the usual submodule shape: the app repo is a frozen archive we don't modify, so the chart + Dockerfile live here and the source is the submodule under site/. Public via the shared Cloudflare tunnel. |
web/old-diemer-codes/README |
web/fsu/ → submodules |
fsu.zachd.duckdns.org — nine pieces of 2013–16 Florida State coursework, running rather than screenshotted: a C shell (Cloysta), a C++ hash-table REPL, and a menu of 2013 C++ console programs and 2014 MIPS under spim, each behind its own ttyd; a Swing Breakout compiled to Java 8 bytecode and run client-side on CheerpJ's WebAssembly JVM; a Django 1.9.5 bank still on Python 2.7, because mixed tabs and an implicit relative import mean it cannot be anything else; and Verilog sims, Python 2 scrapers (against the Wayback Machine) and a Notepad++ diff, run at image build time. Five of the nine are private repos recovered from Dropbox. Inverts the submodule shape nine times over — see Conventions. Three images from one context. Tailnet only, with Authelia in front of the three terminals because one of them is a real fork/exec shell. | web/fsu/README |
web/talaria/ → submodule |
talaria — auction watch platform (search, tracking, listing alerts): a Python backend, scrapers, and a frontend, with Postgres, Elasticsearch, Redis and Logstash in its own chart at helm/talaria/. The biggest thing on the cluster, and the only one whose secrets are sops-encrypted in-git rather than in a values.local.yaml. Chart + source live in zdiemer/talaria. |
zdiemer/talaria README |
web/talaria-deals/ |
talaria.deals — a single Ingress publishing the sibling talaria project's existing service through the shared Cloudflare tunnel. Lives here rather than in talaria's chart because that chart is in another repo; additive, so talaria keeps answering on DuckDNS. |
web/talaria-deals/README |
web/apartment-watch/ → deprecated |
SF rental scraper → SMS, retired 2026-08-13 when the flat search it served ended. A CronJob scraped Craigslist and Dahlia (plain HTTP) plus Zumper/Apartments.com/Zillow (Camoufox, to clear Akamai and PerimeterX), filtered on rent/laundry/parking/neighborhood with a scored scam filter, texted new matches through infra/sms-relay, and served the run pages at homes.diemer.codes. Uninstalled from the cluster; the chart and source stay here as the worked example for infra/egress-proxy's browser tier and for private-GHCR image pulls. |
web/apartment-watch/README |
life/laundry/ |
Laundry-done detector. An ESP32 + MPU-6050 stuck to the washer measures vibration and posts it every 5 s; the cluster runs the state machine and texts through infra/sms-relay when a load finishes. The device decides nothing on purpose — thresholds have to be tunable by helm upgrade, because by the time you know the right one the sensor is glued to the back of an appliance. Sensors reach the cluster over a LoadBalancer on the node IPs: every DuckDNS name here resolves to a Tailscale address and a microcontroller can't run tailscaled. Quiet is only ever counted from readings that arrived, so a dropped wifi link delays a text but can never fake one. |
life/laundry/README |
life/early-internet-archive/ |
Private, provenance-first browser for recovered forum posts, reviews, conversations, and images from 2005–2016. Source captures live on a NAS-backed RWX claim; two rootless readers copy the immutable SQLite search projection locally and stream media from the read-only claim. Tailnet-only DuckDNS ingress with Authelia forward-auth. | life/early-internet-archive/README |
infra/actions-runner/ |
Self-hosted GitHub Actions runners (ARC) for the private repos only — one scale set per repo, scale-to-zero, dind sidecar for image builds. runs-on: arc. Private repos are the only ones GitHub meters, and the only ones that should ever run on our hardware. |
infra/actions-runner/README |
infra/cloudflared/ |
Shared, domain-agnostic Cloudflare Tunnel connector. Publishes services on diemer.codes (auth/pdf/games/romm/old/smite) and talaria.deals through Traefik over an outbound-only tunnel; each app also keeps its DuckDNS ingress via an ingress.cloudflareHosts list. One tunnel, any number of zones. |
infra/cloudflared/README |
infra/traefik/ |
Load-bearing for the whole cluster. The Traefik config overlay: the duckdns ACME DNS-01 certresolver every ingress here names, the http→https redirect, the Recreate rollout, and JSON access logging. Doesn't install Traefik — k3s does — but every change is a cluster-wide ingress outage. Split out of infra/duckdns. |
infra/traefik/README |
infra/duckdns/ |
Load-bearing for the whole cluster. Keeps zachd.duckdns.org pointed at the house (updater CronJob) and owns the DuckDNS token that Traefik's certresolver reads — copied into kube-system because a secretKeyRef can't cross namespaces. Moved out of the sibling talaria project. |
infra/duckdns/README |
infra/alloy/ |
The telemetry pipeline. Two releases from one directory: a DaemonSet shipping logs (Traefik access log, cloudflared, Authelia, CrowdSec detections, the egress proxy, plus pods labelled from their own ingress.enabled) and nine Prometheus scrape jobs, and a one-replica Deployment running blackbox probes of the public hostnames. Values-only against the upstream chart; no PVC, so nothing is pinned to a node. Every target is filtered to the local node so each is handled exactly once — which is why the DaemonSet must tolerate every taint. |
infra/alloy/README |
infra/grafana-dashboards/ |
Four Grafana dashboards and thirteen alert rules as code, pushed to Grafana Cloud over its HTTP API by upgrade.sh. Deploys nothing to the cluster. Datasource UIDs are resolved at push time rather than committed; --verify runs every panel query against live data, because a dashboard of empty panels reads as "nothing is happening" rather than "this query is wrong". Alerts evaluate but route to a deliberate dead end until wired to sms-relay. |
infra/grafana-dashboards/README |
infra/node-exporter/ |
Host CPU/memory/disk/NIC metrics for every node. Values-only, DaemonSet, hostNetwork — which is not optional, since /proc/net resolves inside the reading process's netns and a cluster-network pod would report its own veth. The only source of visibility for Minecraft :25565 and Jellyfin :30096, neither of which traverses Traefik. Collectors are opt-in to hold the series budget. |
infra/node-exporter/README |
infra/postgres-storage/ |
Talaria PostgreSQL data and WAL on node3’s second NVMe. A static Retain volume and filesystem identity guard keep database I/O separate from the OS and etcd disk. The original copy remains retained. | infra/postgres-storage/README |
infra/kube-state-metrics/ |
Kubernetes object state as metrics: CronJob last-success, backup freshness, PVC fill denominators, pod restarts, CrashLoop reasons. Unfiltered it is ~15–20k series against a 10k free tier, so the collector and metric allowlists are the whole design — upgrade.sh fails the deploy above 3k. |
infra/kube-state-metrics/README |
infra/ingress-policy/ |
Load-bearing for the whole cluster. A ValidatingAdmissionPolicy requiring every Ingress in every namespace to name ingressClassName: traefik, pin router.entrypoints: websecure, and either name a certresolver or declare that TLS ends at the Cloudflare edge. No pods — the API server evaluates it. Exists because tenants from other repos (whatnowgg, talaria, money) can't see this repo's conventions. Advisory (Warn) until the warnings go quiet. |
infra/ingress-policy/README |
infra/cluster-status/ |
Tailnet-only dashboard at status.zachd.duckdns.org. Opens on a catalog of every service in the house as a grid of favicon tiles, each linking to its public address where it has one (Cloudflare tunnel or VPS relay) and its DuckDNS name where that is the only way in — favicons fetched by the collector from each ClusterIP, so a service behind Authelia still gets its own mark. Below it: nodes, CPU/RAM/disk broken down pods-vs-k3s-vs-system, pod tables, deployment health, warnings. A read-only collector sidecar writes JSON; nginx serves it as a static page, so traffic never touches the k8s API. Was public at status.diemer.codes until the page's content — node names, pod inventory, raw event text — was weighed rather than just its API safety. Ported from talaria's authed /admin/cluster. |
infra/cluster-status/README |
infra/hatch/ |
Tailnet-only REST/JSON cluster API at hatch.zachd.duckdns.org, for an external AI agent that has no kubeconfig and no vault session. The other half of the story infra/cluster-status tells: that one renders the cluster for a human, this one answers questions about it for a machine, and it proxies the same collector rather than scraping again — which is why it needs no nodes/proxy. Bearer key with read/act scopes, checked in constant time; pod logs and remediations are each bounded by a deny-list, and a refusal names the rule that matched rather than returning a bare 403. Actions (rollout restart, delete a pod, scale) ship off, and their RBAC is not rendered until they are on. Every mutation, denial and log read is one JSON line to Loki. No tailnet ipAllowList: klipper-lb SNATs the source before Traefik sees it, so the gate is DNS plus the key. |
infra/hatch/README |
infra/ntfy/ |
ntfy — push notifications at ntfy.zachd.duckdns.org. The cheap tier next to infra/sms-relay: unlimited, structured (title/priority/tap-through/attachment), per-topic, for everything that doesn't need to arrive as a real text message. Deliberately not behind Authelia — every client is an API client — so the gate is ntfy's own deny-all ACL, with a write-only alerts publisher and an admin subscriber seeded from the vault on every start. |
infra/ntfy/README |
infra/priority-classes/ |
Load-bearing for the whole cluster. The three cluster-wide scheduling priorities, including the platform-app globalDefault that every pod here inherits without naming it. Pure policy, no workload. Moved out of talaria. |
infra/priority-classes/README |
infra/renovate/ |
Self-hosted Renovate as a weekly CronJob — opens version-bump PRs across the zdiemer repos (helm values images, CHART_VERSION pins, Dockerfile ARGs, Chart.yaml appVersion, plus normal npm/pip/actions in the app repos). Shared behavior preset in renovate/default.json. PRs propose, a human merges, upgrade.sh deploys. |
infra/renovate/README |
infra/egress-proxy/ |
Load-bearing for the whole cluster. The single egress choke point — the counterpart to infra/traefik. A CONNECT-only squid that services opt into with egress.proxy.enabled, so outbound traffic is attributable (auth username = chart name, in every log line), counted, and movable off the home address by naming a different lane. lane: direct means "measured, but unchanged", which is what makes it safe to onboard one service at a time. Never a TLS-terminating proxy: smitele-bot's Cloudflare clearance and talaria's JA3/JA4 both depend on the client's own handshake reaching the origin. |
infra/egress-proxy/README |
infra/coredns-config/ |
Load-bearing for the whole cluster. The CoreDNS config overlay, via the coredns-custom ConfigMap k3s's Corefile already imports. Today it owns one thing: a gated DNS query log that produces the cluster's egress inventory — every outbound connection starts with a lookup, so this sees all of it, including from repos this one can't read. Default off, and meant to be: it's an audit instrument for a measurement window, not a pipeline. Applying it restarts cluster DNS, so upgrade.sh canaries the stanza on the real image first and auto-rolls-back. Collected by scripts/egress-audit.sh. |
infra/coredns-config/README |
-
One namespace per project (
minecraft,discord). Created manually once beforehelm install; never managed by a chart. -
Secrets never hit git. Tokens, passwords, and any user-identifying config live only in
values.local.yaml. The.gitignoreglob**/values.local.yamlcovers every project. Watch out for the neighbouring*-secret.yamlglob: it is meant for rendered manifests, but it also swallows chart templates named that way, silently and without an error (templates/ghcr-secret.yamlnever gets added). Name pull-secret templates something else — seeweb/apartment-watch/templates/imagepullsecret.yaml. -
1Password is where those secrets live, and they are never written to a disk. Each project tracks a
values.local.tpl.yamlholding nothing butop://references. At deploy timeupgrade.shsourcesscripts/lib/secret-values.sh, which resolves that template into a shell variable and hands helm a pipe:. "${HERE}/../../scripts/lib/secret-values.sh" sv_load "$HERE" || exit 1 helm upgrade --install "$REL" "$HERE" -n "$NS" -f "$VALUES" -f <(sv_fd)There is no
values.local.yamlany more. It used to be materialized next to each chart, which meant every secret in the fleet sat on ext4 between deploys — and on the claude-workspace PVC, unpacked there on every pod start. The vault was already the source of truth; the files were a cache nothing needed.Each helm call writes its own
<(sv_fd)rather than sharing one through aVALUE_ARGSarray. That fd is a pipe, readable exactly once, and eight scripts here read their values twice or more. A shared one renders as though the secret were not there — measured: fiverender()calls ininfra/traefikgive a stable 163 lines each, and hoisting the substitution into the array gives 0 on the first.The
.examplefiles stay, and matter more than before: they are now the only human-readable record of shape and provenance, which a template of references cannot carry.infra/coredns-configis a deliberate exception — itsvalues.local.yamlis a mode switch, not a secret, and its absence is what closes the DNS audit window. -
Nothing reconciles, because nothing can drift. With no second copy there is no push/pull to arbitrate, so the old
secrets.sh synctimer is gone. Asecrets.sh checktimer (every 15 min,scripts/systemd/) runs in its place and guards what can still go wrong: a vault item renamed or deleted, a field emptied, a reference that no longer resolves. None of that is visible until a deploy renders a Secret with a blank value — which installs perfectly and fails at runtime. It runs unattended becausescripts/op-session.shobtains a session without a terminal, which is also why the weekly backup carries a vault dump.Editing a secret is
secrets.sh edit <chart>: the vault's copy is opened in$EDITORon tmpfs and pushed back on save.secrets.sh new <chart>creates one, seeded from the.example. Both take--from-stdinfor anything non-interactive. -
Each project ships an
upgrade.shthat does the right pre-flight (e.g. Minecraft flushes the world to disk and triggers a backup before the helm upgrade). Prefer it over rawhelm upgrade. -
Availability conventions. Written down once here because they repeat in nearly every chart, and because most of them are the kind of thing that looks optional until a node drain proves otherwise.
A pod behind a Service gets a
preStopsleep. Removing a pod from a Service's Endpoints and sending it SIGTERM are concurrent, not ordered, so Traefik can still be routing to a pod that has already begun shutting down and those requests become 502s.maxUnavailable: 0does not help — it guarantees a Ready replacement, not that this pod stopped receiving traffic. 3s for singletons, 5s for the multi-replica tier. Use the nativelifecycle.preStop.sleepaction rather thanexec: sh -c sleep: most images here have read-only roots and several have no shell. It needs k8s ≥ 1.30.terminationGracePeriodSecondsis set explicitly wherever it matters. The 30s default is not enough for anything that flushes on exit — databases, SQLite-backed apps, qBittorrent's resume data, and above all Minecraft, where a world save cut in half is real corruption. It must exceed thepreStopsleep plus the app's own shutdown.PDBs exist only where
replicas >= 2, and are gated on it in the template. AminAvailable: 1budget against a single-replica Deployment can never be satisfied by evicting that replica, sokubectl drainblocks forever rather than failing. Singletons are therefore deliberately unprotected by a budget;scripts/k3s/drain-preflight.shreports what a drain will gap instead of pretending to prevent it.Spread is
DoNotSchedule, not a preference. ApreferredpodAntiAffinity is a hint the scheduler may decline, so "the replicas are spread" becomes a prediction rather than a property — and it fails silently. Spreading 2–3 replicas across this many nodes remains satisfiable. Always includematchLabelKeys: [pod-template-hash], or a rollout measures its surge pod against the pods it is replacing, finds every node full, and stalls.RecreatevsRollingUpdatefollows the volume, not taste. A single-writer RWO volume meansRecreate— and ontruenas-iscsithat is not a preference: RWO there is single-node attach, so a surge pod on another node blocks onFailedAttachVolumeand the rollout wedges rather than fails. Charts with no volume, or with genuinely shared state, roll withmaxUnavailable: 0— which is only meaningful if the container has a readiness probe, so add one before relying on it.Downtime claims need a number.
scripts/measure-gap.shpolls a host while an upgrade or drain runs and reports the outage windows. Two "zero-downtime" rollouts in this repo were silently broken before it existed. One caveat it now warns about: on an Authelia-gated host the forward-auth middleware answers before Traefik routes anywhere, so an unauthenticated probe measures Authelia rather than the service behind it. -
Ingresses do not name a certresolver. They set
traefik.ingress.kubernetes.io/router.tls: "true"and pick up the cluster-wide default certificate from Traefik's TLSStore — the wildcard thatinfra/traefik-certs/renews into a Secret. Traefik itself no longer runs ACME at all.The constraint that shaped all of this still holds and is worth knowing before adding a host: DuckDNS can only write a TXT record at the account's own subdomain, so a per-host cert for
foo.zachd.duckdns.orgcannot answer its own DNS-01 challenge. Every sub-subdomain rides the wildcard SAN instead. That is why the wildcard is not optional, and why per-Ingressrouter.tls.domains.*annotations used to exist — they are gone now because one default certificate covers every host. -
Apps we write ourselves live in their own repo, added back here as a submodule so this repo still lists everything on the cluster. The app repo owns its chart and its source together —
Chart.yamlappVersiontracksvalues.yamlimage.tag, so a release is one commit in one place. It builds toregistry.zachd.duckdns.org/zdiemer/<name>— the in-cluster registry,infra/registry, so a pod can reschedule while GitHub is down; every node holds the pull credential — and shipsbuild.sh+upgrade.sh. zdiemer/gamedex is the reference shape. Work in the app's own checkout and deploy from there — that checkout is the source for itsvalues.local.yaml, and the submodule copy here is a materialize-on-demand artifact, never a second source.The submodule worktrees are read-only (
scripts/submodules-lock.sh:chmod -R a-wplus ano-pushremote, with matching deny rules in the tracked.claude/settings.json). An edit made inside one looks like it worked and can never ship, which is the kind of mistake worth making impossible rather than memorable. Unlock deliberately if you ever need to.Pins move on their own:
scripts/sync-submodules.sh(daily timer) advances one only once the chart'simage.tagat that commit matches the image the cluster is actually running, so the pin keeps meaning what shipped rather than what is on the branch. It commits locally and never pushes.web/whatnowggdeploys to a VPS instead of this cluster, so it tracks its newestv*tag;web/talariapins every image to:latest, so its pin follows the branch head with that stated plainly;web/old-diemer-codes/siteis a frozen archive and is skipped.A pulled pin does not check itself out, and that is where the "why is
web/whatnowggmodified again?" state comes from. The timer runs in one clone (~/Code/selfhosted); every other clone pulls the pin move into its index and leaves the worktree on the old commit, whichgit statusthen reports as a dirty gitlink forever. It is not a change — it is that clone being behind — and committing it with a broadgit add -Awalks the pin backwards, so the repo claims the cluster runs older code than it does. Exactly what the verified sync exists to prevent, arriving through the back door..githooks/post-mergeandpost-checkoutclose it: after any pull or branch switch,scripts/submodules-refresh.shputs the worktrees back on the recorded commits (unlocking and re-locking around the checkout, and doing nothing at all when there is no drift). Both needcore.hooksPathset — same one-liner as the pre-push hook below. As a belt,sync-submodules.shnow reads the current pin from the index rather than the worktree, so a stale checkout cannot talk it into re-pinning old code.There are two exceptions, both for the same reason: the app repo is a frozen archive that is deliberately not being modified, so the chart and the Dockerfile live in this repo and the app is the submodule underneath, rather than the other way round. The submodule has to sit inside the chart directory because
docker buildcannot COPY from outside its context — which also means the build only ever reads it, so the submodule worktree is never dirtied. Everything else — the registry,build.sh,upgrade.sh, appVersion trackingimage.tag— is the same shape.web/old-diemer-codes/is the 2019 site, one submodule atsite/.web/fsu/is the same idea at nine: 2013–16 university coursework underprojects/, none of which has a chart, a build or a deploy of its own. Those nine are the strongest case for the shape — running them unmodified is the entire point, so every fix a decade-old program needs in 2026 lives outside it inweb/fsu/, andsubmodules-lock.shplus the deny rules in.claude/settings.jsonmake that a filesystem property rather than a promise. -
The lint checks run before the push, not just after it.
.github/workflows/lint.ymlhas always runci-lint-charts.sh,check-appversion-drift.sh,ci-lint-availability.shandci-lint-dashboards.shon every push tomain— and nothing about this repo reports a red run back to anyone, sodev/claude-workspacesat atappVersion: v14againstimage.tag: v15through three consecutive failing runs..githooks/pre-pushruns the same scripts locally (~15s, quiet unless something fails). Enable it once per clone — this is also what turns on the submodule refresh hooks above, so a clone without it drifts in both directions at once:git config core.hooksPath .githooks
git push --no-verifybypasses it deliberately. -
Anything generated from
scripts/says so when it goes stale. ThesecretsCLI is a bundle ofscripts/secrets.shcut at build time (scripts/build-secrets-cli.sh), living both in~/.local/binand in the claude-workspace image — so it freezes discovery and every fix at that moment. Both copies once drifted behind a chart deprecation and spent days failing on vault items that no longer existed. The bundle now carries the sha256 of its sources and warns on stderr when a checkout it can see has moved past it. Rebuild withscripts/build-secrets-cli.sh --install, and rebuild the workspace image (dev/claude-workspace/build.sh) for the pod's copy.
Production whatnow.gg droplet configuration is managed by
infra/whatnow-droplet: Ansible desired state,
Cloudflare origin firewall, pinned releases and 1Password-backed secret delivery.
Production egress-sfo3 relay configuration is managed by
infra/egress-droplet, with preserved vault
credentials, private-destination isolation and verified firewall maintenance.


