Repository navigation
fix(backend): keep stick-tables through a reload that races the peers resync - #381
Merged
Merged
Conversation
… resync Applying a configuration change could still lift a ban, measured on the beta.8 -> beta.9 upgrade path: 3 of 8 applies lost it, and an Apply issued right after a ban was created lost it every time (21 of 21 on a stack whose running worker had never received the tables). The generated config's `peers` section is what carries the stick-tables across the reload, but HAProxy only starts that handover when the outgoing worker's own resync is over (`__process_stopping_peer_sync` in `peers.c`). A worker that received nothing from its predecessor — the first worker that generates the peers section after an upgrade, or one whose predecessor could not teach it — reaches that state only when its resync window expires, `PEER_RESYNC_TIMEOUT`, five seconds later. A reload that arrives before then is not served at all: the outgoing worker never connects, nothing is logged, and the new worker starts with empty tables. Because that new worker never learns anything either, every Apply that follows within the same five seconds fails the same way. `config_apply` now waits for that state before asking the master to reload, reading it from `show peers` through the admin Runtime API. The wait is bounded by HAProxy's own five-second window so a stuck worker cannot hang "Apply configuration", and a worker that did carry the tables over reports as ready immediately, so reloads that are not racing it pay nothing. Measured on `haproxy:3.0.29-alpine` with a backend built from this branch: the 21-iteration rapid-apply sequence that lost the ban 21 out of 21 times before kept it 21 out of 21 times after, waiting about four seconds on its first reload only, and a fresh install kept it 20 out of 20 times. The first Apply after upgrading from a version without the peers section still clears the tables — the worker it replaces cannot teach at all, and waiting does not change that — as does a restarted or recreated HAProxy container. `docs/architecture.md` and the release guide now say exactly that instead of attributing the losses to the upgrade alone.
bihius
added a commit
that referenced
this pull request
Oct 15, 2026
Bumps the three package versions, the release kit's pinned tag and the changelog for the follow-up to v0.1.0-beta.9, whose only content change is the stick-table handover wait in Apply configuration (#381). The upgrade notes state what that fixes, the two remaining cases where the tables still start empty (the first Apply after coming from v0.1.0-beta.8 or older, and a restarted or recreated haproxy container), and that no migration or new environment variable comes with it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Apply configuration could still lift bans on the upgrade path: beta.9's peers section only carries stick-tables across a reload when the outgoing worker's own resync has finished, and a worker that never received anything from its predecessor takes HAProxy's full resync window (5s) to reach that state. Reloading before then is not served at all — the outgoing worker never even connects, so nothing is logged — and because the new worker does not learn anything either, every Apply that follows within the same five seconds fails the same way.
config_apply._reload_haproxynow waits (bounded, via_wait_for_stick_table_handover, readingshow peerson the admin Runtime API) until the worker it is about to replace can hand its tables over, instead of racing it.ban_list_service._send_runtime_commandbecomes public (send_runtime_command) — the same admin socket is what reads the resync state.docs/architecture.md,release/README.md) now describe how the handover really works, what the backend does about its timing, and the two cases that still start empty (the first Apply after upgrading from a version without the peers section, and a recreated HAProxy container).Related to #378, which added the peers section this builds on.
Evidence
Measured on the test host against
haproxy:3.0.29-alpine, both in the same stack (same ban, same traffic, only the backend image swapped):In the failing case the new worker's
show peersreportedflags=0x0(resync unfinished) andlast_status=NONE— no peer connection was ever attempted; with the fix it reportsflags=0x11000003andlast_status=ESTA.Testing
cd src/backend && uv run pytest --cov=app— 891 passed, 6 skipped (run against the worktree's venv; new tests intests/unit/test_config_apply.py)cd src/backend && uv run mypy app/cd src/backend && uv run ruff check app/ tests/haproxy -cclean (the template and therefore the generated config are unchanged; the existing renderer/validation tests still cover them)New tests cover the parser (finished/unfinished/gave-up flags, non-section
flagslines ignored, no peers section), the wait (polls, bounded, logs), and the sequencing that is the point of the fix: the wait runs beforereloadreaches the master socket.