You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up from #63 / #113. Not a continuation of that fix — ContainerHighMemory
now measures against machine_memory_bytes and works whether or not this is ever
done. This is the other half of #63's "two ways out", kept separate on purpose:
a limit is an enforcement control, the alert is a detection control, and
coupling them is what made #63 unfireable in the first place.
Why
No service in compose.yaml sets mem_limit, cpus, deploy.resources, pids_limit or ulimits. A leak in any one container can take the host down,
and the host is a 2012 MacBook Pro with 8 GB soldered (docs/hardware.md:29).
Setting limits also makes the existing ContainerOomKilled rule meaningful
per-service rather than only firing on a host-wide OOM.
The blocker: there is not enough data to choose numbers yet
This is the part worth recording before anyone starts.
cAdvisor only began reporting per-container metrics correctly after #62 fixed cgroup: host. Measured just now, container_memory_working_set_bytes has ~6 hours of history, not the 30 days --storage.tsdb.retention.time
suggests. Peaks over that window (MiB):
service
current
peak (~6h)
2x peak
3x peak
grafana
159
585
1169
1754
alloy
185
187
375
562
prometheus
129
168
337
505
loki
107
152
305
458
alertmanager
21
25
51
76
snmp-exporter
17
19
37
56
total
617
1137
Host is 7816 MiB. 3x every peak is 3410 MiB — it fits comfortably, so the
constraint is not "is there room", it is "is the number right".
Two reasons the window is too short to trust:
grafana already swings 3.7x inside six hours (159 → 585). A limit picked
from a quiet sample would OOM-kill it during whatever caused that.
Prometheus is the genuinely hard one. Its working set grows with the TSDB
as retention fills, and it spikes during compaction and wide range queries.
Six hours of a mostly-empty TSDB says nothing about steady state at day 30.
It is also the worst thing on the host to OOM-kill, since it is what would
otherwise tell you the kill happened.
Suggested approach: let the stack run undisturbed for a couple of weeks now that
collection is fixed, then size from max_over_time(...[14d]) with generous
headroom — 3x peak, not 2x — and start with the low-risk services
(snmp-exporter, alertmanager, loki, alloy) before touching prometheus
or grafana.
Two things to decide, not just numbers
Swap.mem_limit without memswap_limit lets a constrained container
spill to swap rather than be killed. /swap.img is 4 GiB and unencrypted
(docs/security.md:134, SECURITY.md:79), and the repo already treats it as
a data-at-rest exposure. Pushing container memory toward swap makes that
exposure more likely, so memswap_limit should be set deliberately either
way rather than left to default.
A limit-relative alert is a possible complement, not a replacement. Once
limits exist, container_spec_memory_limit_bytes becomes non-zero and a
"near its own limit" rule becomes possible alongside the host-relative one.
If that gets added, it needs its own promtool test rules case — ContainerHighMemory cannot fire — no service sets a memory limit #63 is
precisely what happens when a memory rule is added without one.
Done when
At least 14 days of container memory history exists to size from
mem_limit (and a deliberate memswap_limit) on all six services, each
with the comment style compose.yaml uses — what the number is derived
from, not just what it is
Stack restarted and run through make validate, with no container
OOM-killed and ContainerOomKilled quiet
docs/security.md mitigations table updated if the swap posture changes
Follow-up from #63 / #113. Not a continuation of that fix —
ContainerHighMemorynow measures against
machine_memory_bytesand works whether or not this is everdone. This is the other half of #63's "two ways out", kept separate on purpose:
a limit is an enforcement control, the alert is a detection control, and
coupling them is what made #63 unfireable in the first place.
Why
No service in
compose.yamlsetsmem_limit,cpus,deploy.resources,pids_limitorulimits. A leak in any one container can take the host down,and the host is a 2012 MacBook Pro with 8 GB soldered (
docs/hardware.md:29).Setting limits also makes the existing
ContainerOomKilledrule meaningfulper-service rather than only firing on a host-wide OOM.
The blocker: there is not enough data to choose numbers yet
This is the part worth recording before anyone starts.
cAdvisor only began reporting per-container metrics correctly after #62 fixed
cgroup: host. Measured just now,container_memory_working_set_byteshas~6 hours of history, not the 30 days
--storage.tsdb.retention.timesuggests. Peaks over that window (MiB):
Host is 7816 MiB. 3x every peak is 3410 MiB — it fits comfortably, so the
constraint is not "is there room", it is "is the number right".
Two reasons the window is too short to trust:
from a quiet sample would OOM-kill it during whatever caused that.
as retention fills, and it spikes during compaction and wide range queries.
Six hours of a mostly-empty TSDB says nothing about steady state at day 30.
It is also the worst thing on the host to OOM-kill, since it is what would
otherwise tell you the kill happened.
Suggested approach: let the stack run undisturbed for a couple of weeks now that
collection is fixed, then size from
max_over_time(...[14d])with generousheadroom — 3x peak, not 2x — and start with the low-risk services
(
snmp-exporter,alertmanager,loki,alloy) before touchingprometheusor
grafana.Two things to decide, not just numbers
mem_limitwithoutmemswap_limitlets a constrained containerspill to swap rather than be killed.
/swap.imgis 4 GiB and unencrypted(
docs/security.md:134,SECURITY.md:79), and the repo already treats it asa data-at-rest exposure. Pushing container memory toward swap makes that
exposure more likely, so
memswap_limitshould be set deliberately eitherway rather than left to default.
limits exist,
container_spec_memory_limit_bytesbecomes non-zero and a"near its own limit" rule becomes possible alongside the host-relative one.
If that gets added, it needs its own
promtool test rulescase — ContainerHighMemory cannot fire — no service sets a memory limit #63 isprecisely what happens when a memory rule is added without one.
Done when
mem_limit(and a deliberatememswap_limit) on all six services, eachwith the comment style
compose.yamluses — what the number is derivedfrom, not just what it is
make validate, with no containerOOM-killed and
ContainerOomKilledquietdocs/security.mdmitigations table updated if the swap posture changes