Custom datanode backoff monitor

[anthonyjdam]

Summary

Add a custom DatanodeAdminAdaptiveBackoffMonitor that will stop datanode decommissioning whenever namenode RPC queue length and avg processing time are unhealthy, and when under-replicated block count is high.

Requesting review on:

• Nothing, I feel confident with the changes 🚢
• High-level approach, how do you think this fits in with the larger picture?
• Specific parts of this change:

Testing

• None, trivial change
• Automated tests
• Will manually test end-to-end

Context for reviewer agents

Intent: Give HDFS decommission a load-aware pace on our HBase clusters. The pluggable NameNode decommission monitor (dfs.namenode.decommission.monitor.class) ships upstream as either DatanodeAdminDefaultMonitor (fixed blocks-per-interval, holds the FS write lock the whole tick) or DatanodeAdminBackoffMonitor (HDFS-14854, paces off its own in-flight pendingRep queue up to a fixed pendingRepLimit, default 10000). Neither reacts to NameNode health, so decommission-driven replication competes with foreground traffic when the NN is busy and runs needlessly slowly when it's idle. This PR adds a third monitor, DatanodeAdminAdaptiveBackoffMonitor, which makes that pending limit dynamic via a closed-loop feedback controller driven by a smoothed NameNode-load signal (average RPC queue time). It ships dark behind a flag (default off) — identical to the stock backoff monitor until enabled — and every knob is runtime-reconfigurable. Implements https://github.com/HubSpotEngineering/HBasePlanning/issues/2591. Base branch hubspot-3.3.6.

Changed files (11, +1417/−6):

• ipc/metrics/RpcMetrics.java (+17) — expose the rolling RPC queue-time mean (hadoop-common).
• hdfs/DFSConfigKeys.java (+65) — the config keys + defaults.
• blockmanagement/DatanodeAdminAdaptiveBackoffMonitor.java (+442, new) — the monitor / controller.
• blockmanagement/DatanodeAdminManager.java (+156) — reconfiguration refresh/get methods.
• namenode/FSNamesystem.java (+45) — RPC load-signal accessors.
• namenode/NameNode.java (+127/−6) — RPC-server injection + reconfig wiring.
• resources/hdfs-default.xml (+134) — docs for every new key.
• test/.../blockmanagement/TestDatanodeAdminAdaptiveBackoffMonitor.java (+246, new) — pure-unit tests.
• test/.../hdfs/TestDecommissionWithAdaptiveBackoffMonitor.java (+55, new) — MiniDFSCluster e2e.
• test/.../namenode/TestNameNodeReconfigure.java (+105) — runtime-reconfig tests.
• test/.../tools/TestDFSAdmin.java (+26/−5) — updates the reconfigurable-property assertion.

Diff shape:

• RpcMetrics — adds getQueueMean() / getQueueSampleCount() (rpcQueueTime.lastStat().mean() / .numSamples()), mirroring the pre-existing getProcessingMean() / getProcessingSampleCount(). The rpcQueueTime MutableRate previously had no public getter.
• FSNamesystem — a private volatile Server clientRpcServer (set from NameNode.initialize()), setClientRpcServer(Server), and two null-safe readers: getAvgRpcQueueTimeMs() (→ getQueueMean(), the primary signal) and getAvgRpcProcessingTimeMs() (→ getProcessingMean(), the optional processing-time override). Both return -1 when the server is unwired or no samples exist in the interval.
• DatanodeAdminAdaptiveBackoffMonitor — extends DatanodeAdminBackoffMonitor; reuses all of the parent's tracking/backoff machinery and only changes how pendingRepLimit is chosen. Overrides processConf() (reads the keys, validateAndFixup(), derives the EWMA alpha) and run() (if enabled: setPendingRepLimit(computeAdaptivePendingLimit()), then super.run()). The control math is factored into two pure, @VisibleForTesting helpers — nextControllerLimit(current, smoothedSignal, forceMin) and smoothSignal(prevEma, sample) — so it's testable without the lock-holding run().
• NameNode — injects namesystem.setClientRpcServer(rpcServer.getClientRpcServer()) in initialize(); adds the keys to the reconfigurableProperties set; adds an else if in reconfigurePropertyImpl → new reconfigureDecommissionAdaptiveMonitorParameters(...) handler that parses/validates each key and dispatches to DatanodeAdminManager.
• DatanodeAdminManager — a requireHubSpotMonitor(key) guard (instanceof + cast) and refresh*/get* per knob; adds ensurePositiveLong / ensureNonNegativeLong helpers (int knobs reuse ensurePositiveInt; the two override knobs use the existing ensureDisabledOrPositive).
• hdfs-default.xml — one <property> per new key (required by TestHdfsConfigFields).
• Tests — unit tests drive the controller step, the EWMA, config validation, and run() wiring; TestDecommissionWithAdaptiveBackoffMonitor runs the full TestDecommission suite with the monitor enabled; TestNameNodeReconfigure covers every knob + rejection when a non-adaptive monitor is active; TestDFSAdmin re-sorts the reconfigurable-property assertion (count 28 → 30).

How the controller works (per tick, only when enabled):

1. Sample signal = FSNamesystem.getAvgRpcQueueTimeMs(). If < 0 (RPC metrics not available yet), fail open — return the current limit without advancing controller state.
2. Seed the integral state from the current pendingRepLimit on the first tick (smooth enable, no step change).
3. smoothed = smoothSignal(prevEma, signal) — optional cross-tick EWMA, alpha = tick / (window + tick).
4. nextControllerLimit:
   • smoothed <= healthy → ramp up by ramp.up.step (capped at max);
   • smoothed >= busy → ramp down by ramp.down.step (floored at min);
   • in between → hold (the deadband is the hysteresis).
   • Two optional hard overrides (busy.rpc.processing.time.ms, max.low.redundancy.blocks) force the floor immediately.

Config keys added (all under dfs.namenode.decommission.backoff.monitor., all reconfigurable at runtime):

key	default	meaning
adaptive.enabled	false	master switch; off ⇒ identical to stock backoff monitor
min.pending.limit	100	floor (busy); >0 so decommission never fully stalls
max.pending.limit	inherits pending.limit (itself 10000)	ceiling (healthy); when unset, defaults to the configured pending.limit, not a constant
healthy.rpc.queue.time.ms	1	avg RPC queue time (ms) at/below which the controller ramps the limit UP
busy.rpc.queue.time.ms	50	avg RPC queue time (ms) at/above which the controller ramps the limit DOWN; must be > healthy
ramp.up.step	500	blocks added to the limit per tick while healthy (slow up)
ramp.down.step	2000	blocks removed per tick while busy (fast down, AIMD-style)
signal.ema.window.ms	0	optional cross-tick EWMA window; 0 uses the RPC-metrics windowed mean directly
busy.rpc.processing.time.ms	-1	optional hard override → floor; -1 disables
max.low.redundancy.blocks	-1	optional hard override → floor; -1 disables

Development notes:

• Same backoff engine, only the dial moves. The pendingRep queue, scheduling into neededReconstruction, blocksPerLock lock cadence, and node tracking/completion are 100% the parent DatanodeAdminBackoffMonitor's code, untouched. The only behavioral change is that pendingRepLimit is recomputed each tick. Reviewers should not expect changes to the block-scheduling loop.
• Feedback (integral) controller, not absolute-target mapping. Modeled on HBase's FeedbackAdaptiveRateLimiter (and Janert's Feedback Control for Computer Systems): the limit is a control variable nudged each tick, not recomputed from a curve. The deadband between healthy and busy is the hysteresis and the small per-tick steps are the output-side smoothing, so adjacent ticks can never snap between floor and ceiling (which the earlier threshold-interpolation design did at both cliff edges). Ramp-down > ramp-up by default (fast to yield, slow to re-expand; AIMD).
• Primary signal is average RPC queue time, deliberately not a block-queue count. getLowRedundancyBlocksCount() rises because the monitor schedules its own decommission work, so it would create a self-reinforcing loop. Average RPC queue time (how long calls wait before a handler picks them up) measures genuine foreground contention, is not inflated by our own scheduling, normalizes for service rate, and — critically — is a windowed mean already maintained by the RPC metrics system, so it survives the coarse (30s default) monitor tick where an instantaneous Server.getCallQueueLen() sample would just alias.
• Known limits of the smoothing (call out for reviewers). getQueueMean() is the mean over the RPC metrics collection interval (configurable, commonly ~10s), which is shorter than the tick — so we read a rolling mean every tick rather than integrating the full inter-tick period; residual aliasing is greatly reduced, not zero. The in-monitor cross-tick EWMA (signal.ema.window.ms) exists to close that gap but is off by default (the metric is already a mean). DecayRpcScheduler's decayed averages were considered as an alternative smoothed source but not used, since they only exist when FairCallQueue is enabled; queue-time mean is always available.
• Hard overrides vs. the control loop. The processing-time and low-redundancy knobs are orthogonal safety gates that slam the limit to the floor immediately; they are not part of the proportional pacing and are disabled by default. The low-redundancy cap subtracts pending-reconstruction (a proxy for our own in-flight work) before comparing, and is an approximate safety ceiling, not a pacing signal.
• Fail-open and self-disable. A -1 signal leaves the limit untouched; run() wraps the computation in a broad catch (Exception) (matching the parent run()) so adaptation can never break the decommission loop. validateAndFixup() clamps bad config at startup (min ≥ 1, max ≥ min, positive steps) and disables adaptation if busy <= healthy.
• Ceiling inherits pending.limit. max.pending.limit defaults to the parent's resolved pending.limit (not a constant), and the reconfig reset path mirrors this via getConf(), so enabling adaptation never silently caps peak throughput below what the stock monitor already did.
• Reconfig caveat. Cross-field invariants (busy > healthy, max >= min) are enforced only at startup by validateAndFixup(); -reconfig sets individual knobs and does not re-run it. Per-knob positivity is validated. A degenerate reconfigured pair is not dangerous (the controller mostly holds/ramps one way) but is worth tightening in a follow-up.
• Plumbing choice. Block-count signals were already reachable (the monitor holds blockManager); the only new wiring is the RPC-server reference into FSNamesystem, which avoids adding anything to DatanodeAdminMonitorInterface. The (FSNamesystem) namesystem cast is safe because DatanodeAdminManager is always constructed with the concrete FSNamesystem.

[rmdmattingly]

Went through the PR with AI to get a high level, and some of the comments stuck out as good ideas to me. Overall I think this is the right direction

I have some experience building similar systems to this, like HBase's throttling system, and my biggest advice is that getting the throttling algorithm tuned is going to be pretty hard, so this is probably particularly impactful feedback. Likewise, visibility will be quite important as you try to tune the algorithm (metrics feedback).

For some inspiration, you can check out this work that I did on HBase to improve its ability to estimate the cost of a scan, and inform its scan throttling system: apache/hbase#5713. Happy to answer any questions about how we landed on the approach there.

[anthonyjdam]

I've added some changes as per the suggestions. Let me know if you disagree or agree with anything, thanks

[anthonyjdam]

After discussion, namenode stability is good right now and shipping this new monitor would incur a lot of effort and maintenance during future upgrades of Hadoop. Putting this on the back burner for now and marking as draft.