Skip to content

security.md

Latest commit

 

History

History
241 lines (195 loc) · 14 KB

security.md

File metadata and controls

241 lines (195 loc) · 14 KB
Raw

Security

The lab is a security project, so the interesting question is not "is it secure" but "what is it defending against, and what is it knowingly not."

Threat model

What this network is actually built to survive:

Threat Control
A compromised IoT device pivoting to a workstation VLAN 20 is terminal — no route to any other segment
A guest on the Wi-Fi enumerating the LAN VLAN 10 is terminal, client isolation on
A smart TV's firmware phoning somewhere unexpected VLAN 40 is terminal, egress only
A corporate laptop carrying something in from outside Sits on VLAN 50 but has no management access
A lab VM escaping into the house VLAN 30 reachable only from trusted, never to it
Losing visibility of a failure 35 alert rules, 30 days of metrics and logs
Mains power loss Not currently defended. mjolnir has no battery installed — see below

What it explicitly does not defend against: a determined attacker with physical access to the rack, a supply-chain compromise in an upstream container image, or a vulnerability in pfSense itself. There is no egress filtering by domain, and no MFA on the internal services.

Intrusion detection is running as of 2026-08-21, on the Skids (VLAN 20) interface only. Suricata sits on morpheus rather than the hypervisor because it is the only device that sees the IoT and guest segments, per ADR-0006. Alerts reach Loki through the firewall's syslog pipe, with classification and priority parsed into labels; SuricataHighPriorityAlert and SuricataAlertStorm are armed against them. runbooks/enable-suricata.md covers the setup and the tuning.

Three limits, stated rather than implied:

  • It is alert-only. Block Offenders is off and stays off until a fortnight of understood alerts, and probably not on VLAN 20 even then — an auto-block there can take out a camera or the alarm hub.
  • It watches one segment. Guest (VLAN 10) is next; the rest are unwatched. WAN deliberately never will be.
  • It sees plaintext only. Suricata cannot inspect inside TLS, so the useful signal is DNS, SNI, JA3 and the diminishing share of traffic still in the clear.

There is still no way to detect Suricata itself dying — a quiet IDS and a stopped one produce identical log output, so no log rule can separate them. That needs a process metric and is tracked in roadmap.md. Until it exists, a silent ids alert group is not evidence that anything is watching.

Segmentation

Default deny between every segment. Three exceptions:

  1. Specific hosts on Hicks (50) may reach Winterfell (99) on management ports. Without this there is no way to administer anything.
  2. Hicks (50) may reach ImaginationLAN (30) so the lab is usable.
  3. Winterfell (99) may reach 10.0.30.10 on ImaginationLAN (30) for SNMP. This is how snmp-exporter polls the ProLiant's iLO, and it has been in place — and answering — for as long as that target has existed.

ADR-0002 records two rules, which was accurate when the decision was made. The third arrived with the monitoring stack and was never written down. It is the safe direction — management initiating into the lab, never the reverse — but an undocumented rule is still an undocumented rule, and a document that overstates a control is worse than one that admits the exception.

Everything else — IoT, media, guest — gets internet and nothing more.

The IoT segment is the one that justifies the whole exercise. It holds cameras, a doorbell, an alarm hub, smart speakers, a baby monitor and a $20 Tuya white-noise machine. Every one of those is a network-connected computer running firmware nobody outside its vendor has audited, several with no update mechanism at all. Treating them as untrusted is not paranoia; it is the only assumption consistent with what they are.

Secrets

  • Credentials are encrypted with SOPS + age and committed in encrypted form. See secrets/README.md.
  • The private key lives at ~/.config/sops/age/keys.txt on the deployment host and is never in the repository. That host's disk is not encrypted — see below.
  • That key is the single point of failure for every encrypted secret here, so it is copied off the host and the copy is proven to decrypt with make secrets-verify-backup KEY=<copy> — which refuses to run against the live key and blanks the environment first, because the obvious hand-typed equivalent passes even for an unrelated keypair. See runbooks/back-up-the-age-key.md.
  • scripts/render-config.sh decrypts at deploy time into gitignored files. Nothing writes a plaintext secret into a tracked path.
  • make secrets-edit hardens $EDITOR before handing it the decrypted file, so the editor cannot persist the plaintext in an undo file, swap file or backup that sops does not shred. See secrets/README.md.
  • CI runs gitleaks with rules specifically for SNMP communities, inline Grafana passwords, PEM private keys and age secret keys, and separately asserts that every secrets/*.sops.yaml is genuinely encrypted.

Known historical exposure

This repository previously committed real credentials. Removing them from HEAD does not remove them from history, and anything ever pushed to a public repository must be treated as compromised:

What Where Status
SNMP community shared across all four devices snmp.yaml, from commit ee3d443 (now rewritten) Purged from history. Replaced with four distinct per-device values, SOPS-encrypted. Rotated on all four. morpheus, mjolnir and shiva verified answering the new community and refusing the old; neo answers the new one but still accepts its previous community — accepted risk, see SECURITY.md and the runbook
Grafana admin / admin with anonymous Admin access compose file Fixed: password from SOPS, anonymous auth disabled
Decrypted secrets in editor undo files ~/.local/state/nvim/undodir/, written by make secrets-edit Found 2026-08-20: three files holding the live pfSense, APC and iLO SNMP communities in plaintext, mode 664, on an unencrypted disk. Shredded. make secrets-edit now hardens the editor first, so it cannot recur. Never committed, never left the host, so the communities were not rotated on that basis
Alertmanager webhook URL and the MokerLink SNMP community a local Claude Code session transcript under ~/.claude/projects/ Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook was rotated because it is a one-line regenerate; the switch community was not, because rotating it means the neo residual below all over again
Passphrase-encrypted TLS private keys certificates/, added in efb2632, deleted in 647d90a Purged from history, and the CA replaced — see runbook. Anything that trusted the old CA must be re-pointed at the new one

CI scans both the working tree and the full history, with no ignore file. Both must be clean unconditionally.

There was a .gitleaksignore listing nine historical findings, each annotated with what it was and why it was still there. It was an acknowledgement, not a fix, and it existed because a CI job that is permanently red for a known reason gets ignored — and then a genuinely new leak goes unnoticed alongside it. The purge removed what it acknowledged, so the file was deleted. A history scan that passes with no exceptions is the evidence the purge worked.

Everything above sits on an unencrypted disk

Every entry in that table is something that happened once. This one is a standing property of the host, which is why it is stated separately rather than added as a sixth row.

Measured on prometheus (10.0.99.20):

  • /dev/mapper holds control and ubuntu--vg-ubuntu--lv and nothing else — no LUKS anywhere. The root filesystem is plain ext4 on LVM.
  • /boot and the EFI partition are likewise plain.
  • /swap.img is 4 GiB, unencrypted, on that same root filesystem, and in use. Anything the stack has held in memory can have been paged into it.

So the age private key at ~/.config/sops/age/keys.txt, the rendered artefacts under snmp-exporter/.rendered/ and alertmanager/.rendered/, and stacks/observability/.env — the last three hold plaintext by design, because something has to hand the containers a usable credential — are protected by nothing but file permissions. They are all mode 600 and owned by robo, which is the right setting and is also the entire control. Permissions are enforced by the running kernel; they mean nothing to a disk read on another machine.

This is accepted, not scheduled. The threat model above already excludes an attacker with physical access to the rack, and this is that exclusion restated where it actually bites. Full-disk encryption on a headless host has its own failure mode — either a passphrase nobody is present to type after a power cut, or a key stored on the same machine, which is most of the way back to where this started.

It is recorded because it changes the severity of things that would otherwise look minor. The undo-file leak above is the worked example: three community strings at mode 664 in ~/.local/state/nvim/undodir/ were a real finding because the disk beneath them is readable. On an encrypted disk that is a much smaller problem. Neither fact is interesting alone.

The UPS reports a battery it does not have

mjolnir has no battery installed. Its Network Management Card nonetheless reports 100% state of charge, 48.0 VDC, a battery temperature, an hour of runtime, a 2030 replacement date, and upsAlarmsPresent = 0. Every one of those values is derived rather than measured.

The single honest signal it emits is the self-test result. The management card renders it as Refused — internal fault; over SNMP it is upsTestResultsSummary = 4 (aborted), from the standard UPS-MIB the apc_ups module already walks. No extra OIDs were needed to see it.

Any alert rule keyed on charge, runtime or alarm count will therefore never fire, no matter how bad things get. UpsSelfTestFailed and UpsBatteryUnproven in ups.rules.yaml key on the self-test instead, and are the only two rules in that file that can detect this condition.

This is worth stating carefully: the monitoring did not fail, and neither did the rules. The device lied, and the rules trusted it.

Why SNMPv2c is still a weak point

The devices are polled with SNMPv2c, which transmits the community string in cleartext. Anyone with a port on the management VLAN can read it off a single packet. Two mitigations are in place, one only partly, and one is not:

  • Done: each device has its own community, confirmed live on all four, so one captured packet no longer grants read access to the whole fleet. The switch does still accept its own previous community as well — an accepted residual, recorded in SECURITY.md.
  • Done: SNMP is reachable only on the management VLAN and the switch-management LAN, neither of which anything but specific trusted hosts can enter.
  • Not done: SNMPv3 with authPriv. The MokerLink switch does not support it. Tracked in roadmap.

These communities are read-only, but "read-only" on a firewall means the complete state table and interface topology. They are credentials.

Hardening applied to the stack

  • Anonymous Grafana access disabled; sign-up disabled; admin password from SOPS.
  • Prometheus, Alertmanager and snmp-exporter run as nobody (65534); Loki as its own unprivileged UID.
  • snmp-exporter is never published to a host interface — it is reachable only on the compose network.
  • The Alloy debug UI binds to 127.0.0.1 only.
  • The Docker socket is mounted read-only into Alloy.
  • All images are pinned to explicit versions, so an upstream compromise cannot arrive silently via :latest. Dependabot proposes the bumps; CI validates them.
  • Grafana telemetry and update checks disabled.

Alloy still runs privileged: true, which it needs for host-level metric collection. That is a real tradeoff and is noted rather than hidden.

What this repository deliberately does not publish

Being able to describe a network precisely is useful; publishing a complete fingerprint of a house is not. Withheld on purpose:

  • Full MAC addresses. Truncated to the OUI, which keeps the useful information (vendor, and therefore what the device is) and drops the unique identifier. Full MACs enable device tracking and, on some networks, MAC-based access control bypass.
  • Owner-linked device names. Personal devices are listed by role (laptop-01) rather than by person, and a child's bedroom is not labelled.
  • Camera-to-room mapping. Knowing there are seven cameras is fine. Knowing which one covers which door is a physical-security detail.
  • The WAN address, firewall rule bodies, and Wi-Fi configuration.

The public IP was already redacted in the original inventory — the rest of this is the same instinct applied consistently.

Rack patch-cable colours are published, and that is a deliberate exception worth defending rather than an oversight. It is the same shape of information as the camera-to-room mapping above — a physical-security detail — but not the same reach. A room mapping is useful remotely: you learn which camera covers which door straight off this page, without ever approaching the house. A cable colour is useful only to someone already standing at the rack, who can see the cables, read the switch port labels and reach the firewall's console port regardless. It tells an attacker nothing their position has not already given them, and it tells a maintainer a great deal. See ADR-0009.