[server] add support of KvPreWriteBuffer backpressure

[zuston]

Purpose

Linked issue: close #3532

Without a backpressure mechanism for KvPreWriteBuffer , the tablet server is at risk of OOM, particularly when ingestion throughput is high but replication is slow. A similar issue can occur during recovery: if the follower is not yet ready while the leader continues accepting client writes, buffered data may accumulate rapidly, causing the leader tablet server to run out of memory and exit.

Brief change log

1. Introduce a global KvPreWriteBuffer memory pool for each tablet server. When memory usage reaches the configured high watermark, incoming client writes are temporarily rejected and retried until usage falls below the low watermark.

In the first phase, I will implement the recovery logic using this unified mechanism. However, recovery may fail when buffer utilization is high due to high-throughput client ingestion. This issue is unlikely to occur under normal conditions.

Tests

API and Format

Documentation

[zuston]

ptal @fresh-borzoni

[beryllw]

@platinumhamburg Could you take a look when you get a chance?

[luoyuxia]

I fell like #3463 can solve this backpressure problem

[wuchong]

Thanks for working on this. We also have #3463, which implements cooperative KV backpressure based on RocksDB L0 metrics. Could you try it and see whether it also addresses the problem described here?

In general, backpressure needs client-side cooperation—for example, retry with exponential backoff—rather than only returning a retriable error from the server. #3463 introduces the server-client throttling/backoff mechanism, so it may be better to let #3463 land first and then build this change on top of it.

My understanding is that the main difference between the two PRs is the source of the pressure signal: #3463 derives pressure from the RocksDB L0 file count, while this PR derives it from KvPreWriteBuffer memory usage to prevent OOM. If so, could this PR reuse the mechanism introduced by #3463 and add pre-write-buffer memory pressure as another backpressure source?