Repository navigation
Preserve client identity through ingress and verify Email access #5448
Description
🤖 Generated by the Agentic Engineer
Problem
Email administration remains unavailable after workload readiness recovered. The ingress investigation also establishes that the application receives a shared intermediary address instead of the original client address. Preserve the original identity through the transport so application access controls can make their intended decisions. The remaining HTTP 503 cause is not yet proven to be this identity loss; retain separate acceptance evidence for the availability repair.
Part of #5412; supports #5030.
Delivery and acceptance
- First declare explicit disabled transport settings on both sides and verify that revision in production. This is the rollback baseline, not activation or completed Email delivery.
- Use the existing declarative transport owners for a coordinated forwarding change. Preserve the existing private backend boundary, TLS validation, listener restrictions and per-client security controls.
- Prove original client identity at the application with native traffic. Forged forwarding metadata and direct untrusted transport requests must not acquire another client's identity.
- Verify unchanged public routes, working authenticated Email administration and sustained readiness after the protected rollout.
- Prove rollback through explicit disabled settings. Removing the provider annotation alone is not sufficient: upstream controller issue 876 records that omission can leave the previous setting active.
Full request, transport and runtime evidence remains in the private operator record. Do not remove stored access rules or exempt shared intermediaries as a substitute for proving client identity.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
devantler commentedon Oct 27, 2026
Additional evidence for this repair: in the pinned native server v0.16.27, the connection accept path checks the remote address against retained bans before parsing HTTP. The HTTP forwarding option separately reads the first Forwarded or X-Forwarded-For value. An HTTP toggle alone therefore does not establish recovery or a safe client identity boundary.
Please retain native controls for two distinct clients, forged/missing/ambiguous forwarding metadata, unchanged retained bans and loopback self-checks, followed by public and authenticated browser acceptance after deployment. The self-check repair is merged and ready, but the external route still returns HTTP 503 in the latest observation.
The overlapping #5449 is being consolidated here before source work. This issue remains the owner of the ingress repair; #5443 and #5412 retain their outstanding external-access acceptance.
Source: pinned connection accept path, pinned HTTP request handling.
chore: agent-claim #5448 nonce=a5a9b6f4e41dca37bd95a9e33672f1c8
chore: agent-claim #5448 nonce=68ccd217b3e2f4eee9c624b351c482c0
devantler commentedon Oct 27, 2026
A bounded review of the pinned sources narrows the transport options:
AllowedIpoverrides a retainedBlockedIp, and the automatic-ban paths skip allowed addresses. This exception is address-wide across listeners. Retaining the blocked record while allowing the shared peer would still exempt that intermediary and would not satisfy this issue's acceptance. See native block/allow precedence.An HTTP forwarding design still needs native proof of a single verified client value, refusal of forged/missing/ambiguous metadata and direct untrusted connections, and a producer boundary that does not rely on a shared-address exemption. Source inspection establishes neither those runtime properties nor recovery. This review made no source or production changes; the current owner and rollback baseline remain in place.
chore: agent-claim #5448 nonce=23825fb70193cf0e7f835a4a0fe1a73d
chore: agent-claim #5448 nonce=037c0d019af04b36f9b490e85da67ab3
chore: agent-claim #5448 nonce=f7eb056055bac94107dbe0809143c66a
chore: agent-claim #5448 nonce=f9e66ba84e558dd3a1db645d17d3db88
chore: agent-claim #5448 nonce=aba1aacaf022b03b14bba4c1470c438a
20 remaining items
devantler commentedon Nov 1, 2026
The probe has run: the fence refuses another peer. Run 41299341438 on
85671b538made 10 of 10 TLS attempts from a peer other than the load balancer: refused 10, accepted 0, other 0, verdictREFUSED. Its cleanup step ran and the run succeeded. Stage 3 can use this as its first answer.chore: agent-claim #5448 nonce=58930741677417027115a70ade36506c
test(gitops): retain RBAC coverage across infrastructure layers
chore: agent-claim #5448 nonce=4ffb131b54059cceb4a8fb841e06d6c4
devantler commentedon Nov 1, 2026
Stage 3 is up as a draft: #5696. It turns on both PROXY settings, production only. Nothing has changed in production yet.
A rollout hazard the plan did not size. The HelmRelease sets
rollOutCiliumPods: true, so this change restarts the Cilium agents one node at a time across 9 nodes, and the load balancer setting follows a layer later. Every public route can fail for that whole window. I estimate 5 to 15 minutes, not measured, and a failed attempt costs a second window on the way back. No ordering avoids it, because the listener cannot accept both kinds of traffic.Checked from source and tests only: the deploy steps follow the rendered active profile, and the peer fence stays selected.
devantler commentedon Nov 2, 2026
Stage 3 is live: #5696 merged at 01:43 UTC after its deploy succeeded.
enable-gateway-api-proxy-protocol=trueGATEWAY_RECEIVER_SOURCE=VERIFIED public_probes=4 fence_selected=trueNot measured: the outage window the earlier comment estimated at 5 to 15 minutes. These are single samples after the rollout, so they say nothing about stability yet. Original client identity at Envoy has not been read either; that is stage 4's first proof.
Email's 503 is unchanged, as expected before stage 4. This does not show whether the identity work will clear it.
devantler commentedon Nov 2, 2026
Stage 3 deployed: #5696 merged at 01:43 UTC on 2 November, and PROXY is on both sides in production. Deploy run 41371279947 passed, and the Heal job was skipped.
ipMode: Proxyenable-gateway-api-proxy-protocol=true, Programmed, Cilium agents 9 of 9 updated and readyGATEWAY_RECEIVER_FENCE=VERIFIED selected=true wrote=falseandGATEWAY_RECEIVER_SOURCE=VERIFIED public_probes=4HTTPRoutestep passed. Of 21 hostnames: 17 answered with 2xx or 3xx, 2 are.lannames that do not resolve from this host, 1 returned 404, 1 returned 503The outage window was likely shorter than my 5 to 15 minute estimate, but it is not measured directly. Cilium agents restarted between 01:33:50 and 01:35:22 UTC, and the load balancer's
EnsuredLoadBalancerevent came at about 01:35:35. That puts the mismatch at roughly 2 minutes. This comes from those timestamps, not from a probe outside the cluster, so the real failed-request window is unknown. I did not read Coroot for it.Not shown by this.
chore: agent-claim #5448 nonce=45d6ddc931886c6fd080a29af6ad0c03
fix(email): publish the client proxy configuration with its bundle
devantler commentedon Nov 2, 2026
Stage 4 is ready in #5711 but not queued. It is evaluated, promoted and clean at
e97505f6f. This run's merge was refused by its permission layer, recorded on devantler-tech/monorepo#3902. Production has not changed.The bundle as prepared could not have reached production: KSail publishes only YAML and JSON, so its separate
Caddyfilewould have been missing. #5711 fixes that. Stage 5 follows only after stage 4 deploys and its identity checks pass.devantler commentedon Nov 2, 2026
Stage 4 is live, and Email's sign-in page now loads. #5711 merged at 04:24 UTC on 2 November. This was read from outside at about 04:36 UTC, so it is one short window, not sustained readiness.
/redirects to/account/, which returned 200 in 15 of 15 samplesclient-proxyandprivate-mailready, 0 restarts since the Pod started at 04:15 UTCcloudflare-proxied: "false". Two public resolvers return the Gateway's own address, and TLS is validX-Forwarded-Foralone, a two-address chain,ForwardedandX-Real-IPall got a normal 200. The adapter answers 400 to a chain, so the gateway did replace the headerThis revises the plan on one point. The plan said the page would only clear at stage 5. It cleared at stage 4, because the mail server no longer sees visitors as the shared address that holds the stored block.
Stage 5 is no longer needed to restore the page, and I am not starting it. The stored rule is still there, and I did not read or change it. Proxied traffic should no longer arrive from that address, and the mail server's native ports are denied to the gateway. Removing the rule would take signed-in access for a tidy-up that fixes nothing a visitor sees. If a signed-in look shows the rule still affects anyone, the narrow recovery in the plan still applies.
Still not proven:
chore: agent-claim #5448 nonce=f7ce804843bd1789f39ac80138ca1314