Skip to content

About

Helm charts and install scripts for the services running on my home k3s cluster

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

selfhosted

lint License k3s Helm Traefik Renovate

selfhosted

Helm charts and install scripts for the services running on my home k3s cluster (10 nodes: 3 control-plane, 7 workers). External exposure goes through infra/traefik/ — the single ingress choke point, its ACME DNS-01 certresolver holding a *.zachd.duckdns.org wildcard cert (no cert-manager involved), and its access log — with the DuckDNS side of that credential in infra/duckdns/. Some services are additionally published on diemer.codes via infra/cloudflared/.

Each subfolder is a standalone project with its own chart, docs, and install/upgrade scripts. Per-project secrets live in a gitignored values.local.yaml alongside the tracked values.yaml.

The exception is scripts/k3s/, which isn't a project: it operates on the nodes themselves (health, disk cleanup, rolling restarts, OS and k3s upgrades) over tailscale ssh, so it lives at the repo root.

plans/ is the other non-project folder: designs that are decided but not yet deployed, usually blocked on hardware or a maintenance window. A plan is deleted once its work ships and the reasoning has moved into the READMEs it touched.

Some projects live in their own repo and are tracked here as submodules, so this repo stays the full index of what runs on the cluster. Clone with git clone --recurse-submodules; an existing clone catches up with git submodule update --init. Then enable the tracked hooks, once per clone — git config core.hooksPath .githooks — they keep the submodule worktrees on the recorded pins after every pull, checkout and rebase (scripts/submodules-refresh.sh) and refuse commits that would move a pin backwards or to a commit GitHub has never seen (.githooks/pre-commit). The pins mean "what shipped", and only scripts/sync-submodules.sh moves them.

What that looks like

kubectl get nodes

Every chart in this repo, deployed:

helm list -A

Real command output, not mockups — docs/capture/termshot.py runs the command, parses the ANSI it prints, and paints it into the window above.

Projects

Folder What it is Docs
life/archiveteam-warrior/ Archive Team volunteer worker using spare capacity; lowest scheduling priority with a bounded graceful drain on preemption. README
minecraft/ Prominence II: Hasturian Era (Fabric 1.20.1) via the itzg/minecraft chart. BlueMap + Discord Integration add-ons, mc-backup sidecar. README bootstrap in values.yaml, ADDON_SETUP, CLIENT_SETUP
minecraft/claude-bridge/ Sandboxed Claude Code in a pod — players type /claude <q>, the bridge tails the server log + replies via RCON, can teleport on request, logs feature requests to FEEDBACK.md. minecraft/claude-bridge/README
minecraft/claude-mod/ Tiny server-side Fabric mod that registers /claude <prompt> via Brigadier and prints a recognizable line for claude-bridge to pick up. Sideloaded into the PVC via install.sh. minecraft/claude-mod/README
games/terraria/ Calamity adventure on tModLoader, with Magic Storage, Recipe Browser, Boss Checklist and Ore Excavator. In-game Claude reads progression, gear and recipes and can teleport the caller. Large Classic world; LAN/tailnet TCP 7777. games/terraria/README
discord/vocard/ Vocard music bot + Lavalink + MongoDB — slash-command music player for voice channels. Bot-only (no dashboard). discord/vocard/README
discord/smitele-bot/ → submodule Smite-le — a Wordle-shaped Smite guessing game bot, plus the daily Hi-Rez match data collector that was previously a Windows Task Scheduler job. The corpus lives on an SMB share from the NAS (ReadWriteMany: the CronJob writes it, the bot reads it); the bot's own caches sit on a small PVC. Chart + source live in zdiemer/smitele-bot. zdiemer/smitele-bot README
games/steam-headless/ Steam + Sunshine on the Radeon 680M node, streamed to Moonlight (iOS, TV, laptop). LAN/tailnet only. games/steam-headless/README
games/cloud-game/ CloudRetro — server-side libretro emulation (NES/SNES/GBA/N64/PS1) streamed to the browser over WebRTC; shared room links give each friend a controller. Curated ROMs unzipped from the NAS share. LAN/tailnet only (UDP). games/cloud-game/README
games/romm/ RomM — self-hosted ROM manager + in-browser EmulatorJS player, library mounted read-only over SMB from the NAS. games/romm/README
games/smt-imagine/ Shin Megami Tensei IMAGINE — the 2007 MMO, shut down 2016, back as a solo server: COMP_hack 4.12.2 (lobby/world/channel in one pod, SQLite) built from the last surviving source tarball, since upstream's GitHub org is empty. Client data staged onto a PVC from a ReIMAGINE client, never in git. Three game ports on the node IPs like Minecraft; account manager behind Authelia at smt.zachd.duckdns.org. Tailnet only. games/smt-imagine/README
games/ra2/ The owner's Red Alert 2 and Yuri's Revenge static recompilations to WebAssembly, built from NAS executables and served with read-only NAS assets at ra2.zachd.duckdns.org and yuri.zachd.duckdns.org. Tailnet only. games/ra2/README
games/ror2/ Experimental unofficial Risk of Rain 2 server. Windows game files come from the NAS over the existing read-only SMB claim; Proton and the companion plugin run in one pod. UDP 7777 is reachable at rain.zachd.duckdns.org from the tailnet. games/ror2/README
games/ffxiv-1x/ FINAL FANTASY XIV 1.23b — the original 2010 game, erased by A Realm Reborn — as a private server: Garlemald Server (the Rust port of Project Meteor), four binaries on one SQLite file, built from upstream git at a pinned commit. Three game ports on the node IPs, no Ingress: the client needs an IP. 75 of 524 quests scripted upstream; CONTENT.md is the inventory. Tailnet only. games/ffxiv-1x/README
games/gamedex/ → submodule Gamedex — searchable browser for the Games Master List spreadsheet, mirrored live from a Dropbox shared link. Faceted search, no auth, PII columns stripped. Chart + source live in zdiemer/gamedex; the pin here is the deployed commit. zdiemer/gamedex README
finance/money/ → submodule money — self-hosted personal finance dashboard: a Monte Carlo retirement/FI planning engine, interactive scenarios, and an auto-syncing monthly net-worth tracker (SimpleFIN + real home/vehicle valuations + collectibles). Gated behind Authelia at money.zachd.duckdns.org; single user, DuckDNS-only, personal numbers never committed to the (public) image. zdiemer/money README
media/jellyfin/ Jellyfin — media server for /mnt/vault/media (NFS, read-write), with VAAPI transcoding able to fail over between the Vega 8 and Radeon 680M nodes. Not behind Authelia (TV/phone apps need the raw API); LAN NodePort :30096 for smart TVs. media/jellyfin/README
media/jellyseerr/ Jellyseerr — request/discovery frontend; signs users in with their Jellyfin account and hands approved requests to Radarr/Sonarr. media/jellyseerr/README
media/arr/ Prowlarr + Sonarr + Radarr + qBittorrent-inside-gluetun (PIA, port forwarding, real killswitch), with a decluttarr watchdog that blocklists dead downloads so the *arrs re-search instead of stalling. Acquisition automation behind Jellyfin: imports are hardlinks into the shared NFS media volume. Admin UIs behind Authelia forward-auth. The VPN pod deliberately bypasses infra/egress-proxy. media/arr/README
media/bitmagnet/ bitmagnet — a self-hosted BitTorrent DHT crawler that builds its own Torznab index in Postgres and serves it to Prowlarr. No tracker site, account, or Cloudflare in the path, which is why it survives the blocks that kill public indexers. Index PVC is deliberately excluded from k8up (a re-crawlable cache). UI behind Authelia forward-auth — bitmagnet has no login of its own. media/bitmagnet/README
auth/authelia/ Authelia — OIDC provider + (future) Traefik forward-auth. Shared login for every service in the cluster. auth/authelia/README
docs/paperless-ngx/ Paperless-ngx — self-hosted document management with OCR + full-text search. Bundles Postgres, Redis, Tika, and Gotenberg inline. docs/paperless-ngx/README
docs/stirling-pdf/ Stirling PDF — locally-processed toolkit for ~50 PDF operations (merge/convert/OCR/sign/redact). Gated behind Authelia forward-auth at the Traefik ingress. docs/stirling-pdf/README
web/kelsey-green/ kelsey.green — static Astro site, no image of our own: git-sync pulls the CI-built deploy branch and nginx serves it. Public via a Cloudflare tunnel (outbound-only) as well as the usual DuckDNS ingress. web/kelsey-green/README
web/rachel-freeman/ rachelfreeman.zachd.duckdns.org — an online art store for originals and prints: Payload CMS admin + Next.js storefront in one image, Postgres, and Stripe Checkout. Prices live in Postgres, not in Stripe, so there is one catalogue and nothing to keep in sync. The only chart here whose GHCR package is private. web/rachel-freeman/README
web/old-diemer-codes/ → submodule old.diemer.codes — the 2019 Create React App personal site, kept exactly as it was. Inverts the usual submodule shape: the app repo is a frozen archive we don't modify, so the chart + Dockerfile live here and the source is the submodule under site/. Public via the shared Cloudflare tunnel. web/old-diemer-codes/README
web/fsu/ → submodules fsu.zachd.duckdns.org — nine pieces of 2013–16 Florida State coursework, running rather than screenshotted: a C shell (Cloysta), a C++ hash-table REPL, and a menu of 2013 C++ console programs and 2014 MIPS under spim, each behind its own ttyd; a Swing Breakout compiled to Java 8 bytecode and run client-side on CheerpJ's WebAssembly JVM; a Django 1.9.5 bank still on Python 2.7, because mixed tabs and an implicit relative import mean it cannot be anything else; and Verilog sims, Python 2 scrapers (against the Wayback Machine) and a Notepad++ diff, run at image build time. Five of the nine are private repos recovered from Dropbox. Inverts the submodule shape nine times over — see Conventions. Three images from one context. Tailnet only, with Authelia in front of the three terminals because one of them is a real fork/exec shell. web/fsu/README
web/talaria/ → submodule talaria — auction watch platform (search, tracking, listing alerts): a Python backend, scrapers, and a frontend, with Postgres, Elasticsearch, Redis and Logstash in its own chart at helm/talaria/. The biggest thing on the cluster, and the only one whose secrets are sops-encrypted in-git rather than in a values.local.yaml. Chart + source live in zdiemer/talaria. zdiemer/talaria README
web/talaria-deals/ talaria.deals — a single Ingress publishing the sibling talaria project's existing service through the shared Cloudflare tunnel. Lives here rather than in talaria's chart because that chart is in another repo; additive, so talaria keeps answering on DuckDNS. web/talaria-deals/README
web/apartment-watch/ → deprecated SF rental scraper → SMS, retired 2026-08-13 when the flat search it served ended. A CronJob scraped Craigslist and Dahlia (plain HTTP) plus Zumper/Apartments.com/Zillow (Camoufox, to clear Akamai and PerimeterX), filtered on rent/laundry/parking/neighborhood with a scored scam filter, texted new matches through infra/sms-relay, and served the run pages at homes.diemer.codes. Uninstalled from the cluster; the chart and source stay here as the worked example for infra/egress-proxy's browser tier and for private-GHCR image pulls. web/apartment-watch/README
life/laundry/ Laundry-done detector. An ESP32 + MPU-6050 stuck to the washer measures vibration and posts it every 5 s; the cluster runs the state machine and texts through infra/sms-relay when a load finishes. The device decides nothing on purpose — thresholds have to be tunable by helm upgrade, because by the time you know the right one the sensor is glued to the back of an appliance. Sensors reach the cluster over a LoadBalancer on the node IPs: every DuckDNS name here resolves to a Tailscale address and a microcontroller can't run tailscaled. Quiet is only ever counted from readings that arrived, so a dropped wifi link delays a text but can never fake one. life/laundry/README
life/early-internet-archive/ Private, provenance-first browser for recovered forum posts, reviews, conversations, and images from 2005–2016. Source captures live on a NAS-backed RWX claim; two rootless readers copy the immutable SQLite search projection locally and stream media from the read-only claim. Tailnet-only DuckDNS ingress with Authelia forward-auth. life/early-internet-archive/README
infra/actions-runner/ Self-hosted GitHub Actions runners (ARC) for the private repos only — one scale set per repo, scale-to-zero, dind sidecar for image builds. runs-on: arc. Private repos are the only ones GitHub meters, and the only ones that should ever run on our hardware. infra/actions-runner/README
infra/cloudflared/ Shared, domain-agnostic Cloudflare Tunnel connector. Publishes services on diemer.codes (auth/pdf/games/romm/old/smite) and talaria.deals through Traefik over an outbound-only tunnel; each app also keeps its DuckDNS ingress via an ingress.cloudflareHosts list. One tunnel, any number of zones. infra/cloudflared/README
infra/traefik/ Load-bearing for the whole cluster. The Traefik config overlay: the duckdns ACME DNS-01 certresolver every ingress here names, the http→https redirect, the Recreate rollout, and JSON access logging. Doesn't install Traefik — k3s does — but every change is a cluster-wide ingress outage. Split out of infra/duckdns. infra/traefik/README
infra/duckdns/ Load-bearing for the whole cluster. Keeps zachd.duckdns.org pointed at the house (updater CronJob) and owns the DuckDNS token that Traefik's certresolver reads — copied into kube-system because a secretKeyRef can't cross namespaces. Moved out of the sibling talaria project. infra/duckdns/README
infra/alloy/ The telemetry pipeline. Two releases from one directory: a DaemonSet shipping logs (Traefik access log, cloudflared, Authelia, CrowdSec detections, the egress proxy, plus pods labelled from their own ingress.enabled) and nine Prometheus scrape jobs, and a one-replica Deployment running blackbox probes of the public hostnames. Values-only against the upstream chart; no PVC, so nothing is pinned to a node. Every target is filtered to the local node so each is handled exactly once — which is why the DaemonSet must tolerate every taint. infra/alloy/README
infra/grafana-dashboards/ Four Grafana dashboards and thirteen alert rules as code, pushed to Grafana Cloud over its HTTP API by upgrade.sh. Deploys nothing to the cluster. Datasource UIDs are resolved at push time rather than committed; --verify runs every panel query against live data, because a dashboard of empty panels reads as "nothing is happening" rather than "this query is wrong". Alerts evaluate but route to a deliberate dead end until wired to sms-relay. infra/grafana-dashboards/README
infra/node-exporter/ Host CPU/memory/disk/NIC metrics for every node. Values-only, DaemonSet, hostNetwork — which is not optional, since /proc/net resolves inside the reading process's netns and a cluster-network pod would report its own veth. The only source of visibility for Minecraft :25565 and Jellyfin :30096, neither of which traverses Traefik. Collectors are opt-in to hold the series budget. infra/node-exporter/README
infra/postgres-storage/ Talaria PostgreSQL data and WAL on node3’s second NVMe. A static Retain volume and filesystem identity guard keep database I/O separate from the OS and etcd disk. The original copy remains retained. infra/postgres-storage/README
infra/kube-state-metrics/ Kubernetes object state as metrics: CronJob last-success, backup freshness, PVC fill denominators, pod restarts, CrashLoop reasons. Unfiltered it is ~15–20k series against a 10k free tier, so the collector and metric allowlists are the whole design — upgrade.sh fails the deploy above 3k. infra/kube-state-metrics/README
infra/ingress-policy/ Load-bearing for the whole cluster. A ValidatingAdmissionPolicy requiring every Ingress in every namespace to name ingressClassName: traefik, pin router.entrypoints: websecure, and either name a certresolver or declare that TLS ends at the Cloudflare edge. No pods — the API server evaluates it. Exists because tenants from other repos (whatnowgg, talaria, money) can't see this repo's conventions. Advisory (Warn) until the warnings go quiet. infra/ingress-policy/README
infra/cluster-status/ Tailnet-only dashboard at status.zachd.duckdns.org. Opens on a catalog of every service in the house as a grid of favicon tiles, each linking to its public address where it has one (Cloudflare tunnel or VPS relay) and its DuckDNS name where that is the only way in — favicons fetched by the collector from each ClusterIP, so a service behind Authelia still gets its own mark. Below it: nodes, CPU/RAM/disk broken down pods-vs-k3s-vs-system, pod tables, deployment health, warnings. A read-only collector sidecar writes JSON; nginx serves it as a static page, so traffic never touches the k8s API. Was public at status.diemer.codes until the page's content — node names, pod inventory, raw event text — was weighed rather than just its API safety. Ported from talaria's authed /admin/cluster. infra/cluster-status/README
infra/hatch/ Tailnet-only REST/JSON cluster API at hatch.zachd.duckdns.org, for an external AI agent that has no kubeconfig and no vault session. The other half of the story infra/cluster-status tells: that one renders the cluster for a human, this one answers questions about it for a machine, and it proxies the same collector rather than scraping again — which is why it needs no nodes/proxy. Bearer key with read/act scopes, checked in constant time; pod logs and remediations are each bounded by a deny-list, and a refusal names the rule that matched rather than returning a bare 403. Actions (rollout restart, delete a pod, scale) ship off, and their RBAC is not rendered until they are on. Every mutation, denial and log read is one JSON line to Loki. No tailnet ipAllowList: klipper-lb SNATs the source before Traefik sees it, so the gate is DNS plus the key. infra/hatch/README
infra/ntfy/ ntfy — push notifications at ntfy.zachd.duckdns.org. The cheap tier next to infra/sms-relay: unlimited, structured (title/priority/tap-through/attachment), per-topic, for everything that doesn't need to arrive as a real text message. Deliberately not behind Authelia — every client is an API client — so the gate is ntfy's own deny-all ACL, with a write-only alerts publisher and an admin subscriber seeded from the vault on every start. infra/ntfy/README
infra/priority-classes/ Load-bearing for the whole cluster. The three cluster-wide scheduling priorities, including the platform-app globalDefault that every pod here inherits without naming it. Pure policy, no workload. Moved out of talaria. infra/priority-classes/README
infra/renovate/ Self-hosted Renovate as a weekly CronJob — opens version-bump PRs across the zdiemer repos (helm values images, CHART_VERSION pins, Dockerfile ARGs, Chart.yaml appVersion, plus normal npm/pip/actions in the app repos). Shared behavior preset in renovate/default.json. PRs propose, a human merges, upgrade.sh deploys. infra/renovate/README
infra/egress-proxy/ Load-bearing for the whole cluster. The single egress choke point — the counterpart to infra/traefik. A CONNECT-only squid that services opt into with egress.proxy.enabled, so outbound traffic is attributable (auth username = chart name, in every log line), counted, and movable off the home address by naming a different lane. lane: direct means "measured, but unchanged", which is what makes it safe to onboard one service at a time. Never a TLS-terminating proxy: smitele-bot's Cloudflare clearance and talaria's JA3/JA4 both depend on the client's own handshake reaching the origin. infra/egress-proxy/README
infra/coredns-config/ Load-bearing for the whole cluster. The CoreDNS config overlay, via the coredns-custom ConfigMap k3s's Corefile already imports. Today it owns one thing: a gated DNS query log that produces the cluster's egress inventory — every outbound connection starts with a lookup, so this sees all of it, including from repos this one can't read. Default off, and meant to be: it's an audit instrument for a measurement window, not a pipeline. Applying it restarts cluster DNS, so upgrade.sh canaries the stanza on the real image first and auto-rolls-back. Collected by scripts/egress-audit.sh. infra/coredns-config/README

Conventions

  • One namespace per project (minecraft, discord). Created manually once before helm install; never managed by a chart.

  • Secrets never hit git. Tokens, passwords, and any user-identifying config live only in values.local.yaml. The .gitignore glob **/values.local.yaml covers every project. Watch out for the neighbouring *-secret.yaml glob: it is meant for rendered manifests, but it also swallows chart templates named that way, silently and without an error (templates/ghcr-secret.yaml never gets added). Name pull-secret templates something else — see web/apartment-watch/templates/imagepullsecret.yaml.

  • 1Password is where those secrets live, and they are never written to a disk. Each project tracks a values.local.tpl.yaml holding nothing but op:// references. At deploy time upgrade.sh sources scripts/lib/secret-values.sh, which resolves that template into a shell variable and hands helm a pipe:

    . "${HERE}/../../scripts/lib/secret-values.sh"
    sv_load "$HERE" || exit 1
    helm upgrade --install "$REL" "$HERE" -n "$NS" -f "$VALUES" -f <(sv_fd)
    

    There is no values.local.yaml any more. It used to be materialized next to each chart, which meant every secret in the fleet sat on ext4 between deploys — and on the claude-workspace PVC, unpacked there on every pod start. The vault was already the source of truth; the files were a cache nothing needed.

    Each helm call writes its own <(sv_fd) rather than sharing one through a VALUE_ARGS array. That fd is a pipe, readable exactly once, and eight scripts here read their values twice or more. A shared one renders as though the secret were not there — measured: five render() calls in infra/traefik give a stable 163 lines each, and hoisting the substitution into the array gives 0 on the first.

    The .example files stay, and matter more than before: they are now the only human-readable record of shape and provenance, which a template of references cannot carry. infra/coredns-config is a deliberate exception — its values.local.yaml is a mode switch, not a secret, and its absence is what closes the DNS audit window.

  • Nothing reconciles, because nothing can drift. With no second copy there is no push/pull to arbitrate, so the old secrets.sh sync timer is gone. A secrets.sh check timer (every 15 min, scripts/systemd/) runs in its place and guards what can still go wrong: a vault item renamed or deleted, a field emptied, a reference that no longer resolves. None of that is visible until a deploy renders a Secret with a blank value — which installs perfectly and fails at runtime. It runs unattended because scripts/op-session.sh obtains a session without a terminal, which is also why the weekly backup carries a vault dump.

    Editing a secret is secrets.sh edit <chart>: the vault's copy is opened in $EDITOR on tmpfs and pushed back on save. secrets.sh new <chart> creates one, seeded from the .example. Both take --from-stdin for anything non-interactive.

  • Each project ships an upgrade.sh that does the right pre-flight (e.g. Minecraft flushes the world to disk and triggers a backup before the helm upgrade). Prefer it over raw helm upgrade.

  • Availability conventions. Written down once here because they repeat in nearly every chart, and because most of them are the kind of thing that looks optional until a node drain proves otherwise.

    A pod behind a Service gets a preStop sleep. Removing a pod from a Service's Endpoints and sending it SIGTERM are concurrent, not ordered, so Traefik can still be routing to a pod that has already begun shutting down and those requests become 502s. maxUnavailable: 0 does not help — it guarantees a Ready replacement, not that this pod stopped receiving traffic. 3s for singletons, 5s for the multi-replica tier. Use the native lifecycle.preStop.sleep action rather than exec: sh -c sleep: most images here have read-only roots and several have no shell. It needs k8s ≥ 1.30.

    terminationGracePeriodSeconds is set explicitly wherever it matters. The 30s default is not enough for anything that flushes on exit — databases, SQLite-backed apps, qBittorrent's resume data, and above all Minecraft, where a world save cut in half is real corruption. It must exceed the preStop sleep plus the app's own shutdown.

    PDBs exist only where replicas >= 2, and are gated on it in the template. A minAvailable: 1 budget against a single-replica Deployment can never be satisfied by evicting that replica, so kubectl drain blocks forever rather than failing. Singletons are therefore deliberately unprotected by a budget; scripts/k3s/drain-preflight.sh reports what a drain will gap instead of pretending to prevent it.

    Spread is DoNotSchedule, not a preference. A preferred podAntiAffinity is a hint the scheduler may decline, so "the replicas are spread" becomes a prediction rather than a property — and it fails silently. Spreading 2–3 replicas across this many nodes remains satisfiable. Always include matchLabelKeys: [pod-template-hash], or a rollout measures its surge pod against the pods it is replacing, finds every node full, and stalls.

    Recreate vs RollingUpdate follows the volume, not taste. A single-writer RWO volume means Recreate — and on truenas-iscsi that is not a preference: RWO there is single-node attach, so a surge pod on another node blocks on FailedAttachVolume and the rollout wedges rather than fails. Charts with no volume, or with genuinely shared state, roll with maxUnavailable: 0 — which is only meaningful if the container has a readiness probe, so add one before relying on it.

    Downtime claims need a number. scripts/measure-gap.sh polls a host while an upgrade or drain runs and reports the outage windows. Two "zero-downtime" rollouts in this repo were silently broken before it existed. One caveat it now warns about: on an Authelia-gated host the forward-auth middleware answers before Traefik routes anywhere, so an unauthenticated probe measures Authelia rather than the service behind it.

  • Ingresses do not name a certresolver. They set traefik.ingress.kubernetes.io/router.tls: "true" and pick up the cluster-wide default certificate from Traefik's TLSStore — the wildcard that infra/traefik-certs/ renews into a Secret. Traefik itself no longer runs ACME at all.

    The constraint that shaped all of this still holds and is worth knowing before adding a host: DuckDNS can only write a TXT record at the account's own subdomain, so a per-host cert for foo.zachd.duckdns.org cannot answer its own DNS-01 challenge. Every sub-subdomain rides the wildcard SAN instead. That is why the wildcard is not optional, and why per-Ingress router.tls.domains.* annotations used to exist — they are gone now because one default certificate covers every host.

  • Apps we write ourselves live in their own repo, added back here as a submodule so this repo still lists everything on the cluster. The app repo owns its chart and its source together — Chart.yaml appVersion tracks values.yaml image.tag, so a release is one commit in one place. It builds to registry.zachd.duckdns.org/zdiemer/<name> — the in-cluster registry, infra/registry, so a pod can reschedule while GitHub is down; every node holds the pull credential — and ships build.sh + upgrade.sh. zdiemer/gamedex is the reference shape. Work in the app's own checkout and deploy from there — that checkout is the source for its values.local.yaml, and the submodule copy here is a materialize-on-demand artifact, never a second source.

    The submodule worktrees are read-only (scripts/submodules-lock.sh: chmod -R a-w plus a no-push remote, with matching deny rules in the tracked .claude/settings.json). An edit made inside one looks like it worked and can never ship, which is the kind of mistake worth making impossible rather than memorable. Unlock deliberately if you ever need to.

    Pins move on their own: scripts/sync-submodules.sh (daily timer) advances one only once the chart's image.tag at that commit matches the image the cluster is actually running, so the pin keeps meaning what shipped rather than what is on the branch. It commits locally and never pushes. web/whatnowgg deploys to a VPS instead of this cluster, so it tracks its newest v* tag; web/talaria pins every image to :latest, so its pin follows the branch head with that stated plainly; web/old-diemer-codes/site is a frozen archive and is skipped.

    A pulled pin does not check itself out, and that is where the "why is web/whatnowgg modified again?" state comes from. The timer runs in one clone (~/Code/selfhosted); every other clone pulls the pin move into its index and leaves the worktree on the old commit, which git status then reports as a dirty gitlink forever. It is not a change — it is that clone being behind — and committing it with a broad git add -A walks the pin backwards, so the repo claims the cluster runs older code than it does. Exactly what the verified sync exists to prevent, arriving through the back door. .githooks/post-merge and post-checkout close it: after any pull or branch switch, scripts/submodules-refresh.sh puts the worktrees back on the recorded commits (unlocking and re-locking around the checkout, and doing nothing at all when there is no drift). Both need core.hooksPath set — same one-liner as the pre-push hook below. As a belt, sync-submodules.sh now reads the current pin from the index rather than the worktree, so a stale checkout cannot talk it into re-pinning old code.

    There are two exceptions, both for the same reason: the app repo is a frozen archive that is deliberately not being modified, so the chart and the Dockerfile live in this repo and the app is the submodule underneath, rather than the other way round. The submodule has to sit inside the chart directory because docker build cannot COPY from outside its context — which also means the build only ever reads it, so the submodule worktree is never dirtied. Everything else — the registry, build.sh, upgrade.sh, appVersion tracking image.tag — is the same shape.

    web/old-diemer-codes/ is the 2019 site, one submodule at site/. web/fsu/ is the same idea at nine: 2013–16 university coursework under projects/, none of which has a chart, a build or a deploy of its own. Those nine are the strongest case for the shape — running them unmodified is the entire point, so every fix a decade-old program needs in 2026 lives outside it in web/fsu/, and submodules-lock.sh plus the deny rules in .claude/settings.json make that a filesystem property rather than a promise.

  • The lint checks run before the push, not just after it. .github/workflows/lint.yml has always run ci-lint-charts.sh, check-appversion-drift.sh, ci-lint-availability.sh and ci-lint-dashboards.sh on every push to main — and nothing about this repo reports a red run back to anyone, so dev/claude-workspace sat at appVersion: v14 against image.tag: v15 through three consecutive failing runs. .githooks/pre-push runs the same scripts locally (~15s, quiet unless something fails). Enable it once per clone — this is also what turns on the submodule refresh hooks above, so a clone without it drifts in both directions at once:

    git config core.hooksPath .githooks

    git push --no-verify bypasses it deliberately.

  • Anything generated from scripts/ says so when it goes stale. The secrets CLI is a bundle of scripts/secrets.sh cut at build time (scripts/build-secrets-cli.sh), living both in ~/.local/bin and in the claude-workspace image — so it freezes discovery and every fix at that moment. Both copies once drifted behind a chart deprecation and spent days failing on vault items that no longer existed. The bundle now carries the sha256 of its sources and warns on stderr when a checkout it can see has moved past it. Rebuild with scripts/build-secrets-cli.sh --install, and rebuild the workspace image (dev/claude-workspace/build.sh) for the pod's copy.

Production whatnow.gg droplet configuration is managed by infra/whatnow-droplet: Ansible desired state, Cloudflare origin firewall, pinned releases and 1Password-backed secret delivery. Production egress-sfo3 relay configuration is managed by infra/egress-droplet, with preserved vault credentials, private-destination isolation and verified firewall maintenance.

About

Helm charts and install scripts for the services running on my home k3s cluster

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages