Repository navigation
fix(mongodb): retry an unconditional PutItem that conflicts on a GSI table - #487
LeeroyHannigan wants to merge 24 commits into
Conversation
Amazon DynamoDB applies concurrent PutItem calls on one key one after another. A call without a condition succeeds, the last writer's item is stored, and with ReturnValues ALL_OLD each call returns the item it replaced. Measured on 2026-10-21 against the service in us-east-1 with client retries off: 32 writers x 63 rounds of racing unconditional puts on a new key per round answered 2,016 of 2,016 with 200 on hash, GSI and stream tables; 32 writers on one key for 63 rounds answered 2,016 of 2,016 on GSI and stream tables; one key with half the writers sending PutItem and half DeleteItem answered 22,400 puts and 22,400 deletes with 200 and no error (seven runs of 64 writers x 100 operations on a hash table, three runs of 32 x 200 on a GSI table); 32 writers racing PutItem against DeleteItem, UpdateItem and BatchWriteItem on one key answered every call with 200; and 64 writers with one putter per three deleters on one key answered 960 puts and 2,880 deletes with 200. In every cell the ALL_OLD images, the final item, the index and the stream records formed one serial history. tests/test_put_item_create_race.py runs against the service as well as against ExtendDB and checks the race on 8 clients. On MongoDB, two puts that create the same item could each find no document and each insert it. The second insert fails with a duplicate-key error (E11000), which findAndModify reports as a command error. The data collection has two unique indexes: _id_, and one on the key fields (pk, plus the sort key field on a table with a sort key). MongoDB retries such an upsert itself only when the filter is an equality on the fields of the index that raised the error. The put filters on pk and, on a table with a sort key, the sort key field, and every measured duplicate-key error names _id_, so the server does not retry it. The sessionless path (no condition, no index, no stream) returned the error as an internal error (HTTP 500), and the transaction path (a table with an index or a stream) classified it as fatal, which also returned an internal error. On a 16-core host the race test failed 2 or 3 of its 9 cases per run, each failure a racing put answering InternalServerError; 32 writers creating a new key per round got 3 InternalServerError responses in 2,016 puts; and one-key put and delete churn from 8 and 32 threads got 66 and 45 in 2,016 puts. Both paths now retry the put under the transient retry budget of 50 attempts with the same backoff the write-conflict retry uses. The next attempt finds the item the other put created, replaces it and returns it as the old image, so the ALL_OLD images form one chain; if a delete removed that item first, the next attempt inserts. On the transaction path the index update and the stream record take that item as the old image too. A put whose 50 attempts all lost the create race returns the internal error the PostgreSQL backend returns when its own retries of this race run out, with the attempt count in the message. is_duplicate_key now recognizes the command form as well as the write form that insert_one returns. The MongoDB xfail marker on the plain-table race tests is removed. devtools/run-mongodb-tests exports EXTENDDB_TEST_MONGODB_URI, and two storage crate tests use it, on the sessionless path and on the transaction path each: one forces every attempt to lose the race with a document that holds the item's _id under another pk and checks the give-up error after 50 attempts and that the document is unchanged; the other replaces or deletes that document 100 ms after the put starts and checks that the put succeeds, returns the other put's item or no old image, and stores its own item. Without the URI the tests print a SKIP line; with EXTENDDB_TEST_MONGODB_CONTAINER set and no URI they fail. The scratch databases are dropped when a test fails. Verified against MongoDB 7.0 as a single-node replica set. The race test passes 9 of 9 cases in each of 16 runs. The same cells as the service measurement answered every unconditional PutItem, DeleteItem and BatchWriteItem with 200 and no invariant violation; one-key put and delete churn answered 44,800 puts and 44,800 deletes with 200 (ten runs of 16 x 400 and four of 64 x 100) while 4,663 puts lost a create race and succeeded on a later attempt. With the backoff removed from the sessionless retry the lost-race-then-success test fails after 50 attempts; with a cap of 5 lost races put back on either path, with the transaction give-up error disabled, or with the transaction arm giving up at the first lost race, the storage crate tests fail; with the sessionless retry disabled the race test fails its 3 plain-table cases. One-key contention on GSI and stream tables still answers TransactionConflictException for some puts (issue 426), unchanged here. Closes #356
…table Amazon DynamoDB applies concurrent PutItem calls on one key one after another and answers each with 200; with a global secondary index the index holds the final item's entry only. Measured on 2026-10-23 against the service in us-east-1 with client retries off: 32 writers putting one key of a table with a GSI, 2,016 puts, 2,016 answered 200, and the index held the entry of the final item and no other. On MongoDB a put on a table with an index writes the item and its index entries in one transaction. MongoDB aborts the loser of a same-document conflict at once instead of queueing it, so of N writers on one key one commits per round and the rest start over, and a writer needs on the order of N attempts, more under the connection pool's queueing. The retry budget of 50 attempts ran out for a share of the puts, and the put answered the 400 TransactionConflictException main reserves for a single-item write that could not serialize against concurrent activity. The budget is now 1,000 attempts for every write path in data_engine.rs. The backoff is random, so a writer can lose many rounds in a row, and the chance of losing every attempt falls off exponentially with the budget: 400 attempts left one put in 2,016 without a commit on a host at a load average of 30 and 20 of 1,984 with 64 writers. backoff_sleep is capped at 12.8 ms, so 1,000 attempts bound a write at about 6.4 s of backoff plus the attempts' own round trips before it gives up; measured against a transaction another session holds open, a put returns after 10.5 s. A PutItem that runs out of attempts returns StorageError::Transient, which the engine answers as HTTP 503 ServiceUnavailable. The service answers a PutItem with TransactionConflictException only while a TransactWriteItems holds the item, which this backend cannot tell apart from a race against other puts; the DynamoDB error-handling documentation lists ServiceUnavailable (HTTP 503) as "OK to retry? Yes", and the SDK retry modes retry a 503 on their own, whereas a 400 is handed to the application. Measured on MongoDB 7.0 (single-node replica set, client never retries), 32 writers putting one key of a GSI table, 2,016 puts: main (9d13202) answered 303 of 2,016 with 400 TransactionConflictException; c3f8bc8 (#431) answered 322 of 2,016 with 400 TransactionConflictException; this branch answered 2,016 with 200 in each of 3 runs of the wire test, and the index held the final item's entry only. tests/test_put_item_gsi_contention.py runs this shape and fails on main and on c3f8bc8. Two storage crate tests run against the replica set named by EXTENDDB_TEST_MONGODB_URI: 32 writers putting one key on the transaction path all commit, and a put whose every attempt conflicts with a transaction another session holds open gives up with the transient error after 1,000 attempts, in 10.5 s, leaving the held item unchanged. Fixes #426
…gets TRANSIENT_RETRY_ATTEMPTS sized three different loops, and raising it to 1,000 for hot-key contention raised all three. Two of them are not contention loops and no measurement in this branch says anything about running them 1,000 times: commit_write_transaction retries commitTransaction in place after an UnknownTransactionCommitResult. The label means the commit's outcome is unknown, the commit is idempotent and the body must not be re-run, so those attempts wait for one reply rather than racing other writers. COMMIT_RETRY_ATTEMPTS keeps it at the 50 every write path used before. An unconditional PutItem that keeps losing the create race gives up with an internal error. #431 measured that path at 50 attempts with this backoff: a put blocked by a document another writer removes 100 ms after the put starts commits well inside it, and the storage tests fail with the cap put back to 5. CREATE_RACE_ATTEMPTS keeps it at 50, on the sessionless path and on the transaction path. The transaction path now gives up as soon as that budget is spent instead of carrying lost races through the contention budget. TRANSIENT_RETRY_ATTEMPTS stays at 1,000 and now sizes the contention loops only: put, delete, update, transact and the GSI backfill. The constant's doc comment said 1,000 attempts "bound the wait at about 6.4 s of backoff". backoff_sleep draws each sleep uniformly from 0 to the current base, capped at 12.8 ms, so 6.4 s is the mean of the sleeps and 12.8 s is the bound, before the attempts' own round trips. Measured on MongoDB 7.0 as a single-node replica set, a put that loses all 1,000 attempts to a transaction another session holds open answered in 9.46 s, 9.85 s and 10.00 s over three runs, where each attempt also waits out MongoDB's lock timeout. The comment now states the mean, the bound and the measurement, and the test prints the elapsed time it already measured. The RFC's write-conflict paragraph said is_transient_write_conflict returns true for the UnknownTransactionCommitResult label. It does not, by design: under that label the commit may already have applied, so the body must not be re-run, and classify_commit_error handles it alone. The paragraph now also names the commit and create-race budgets and carries the corrected mean and bound. Issue #426
main added four fields to MongoEngine in #384. The with_scratch_engine test helper builds the engine with a struct literal, so against a main that carries #384 it failed to compile with E0063. It now sets the four fields the way backup_live_tests.rs does. The comments on the retry budget give the measured wall time of a put that loses every attempt as a range from six runs on a shared host.
The 1,000-attempt budget bounds attempts, not the time a caller waits. At 256 writers putting one key of a GSI table, 89 of 5,120 calls were still waiting when the client read timeout fired at 120 s, past the SDKs' default read timeouts. The contention loops of PutItem, DeleteItem, UpdateItem and TransactWriteItems now read CONTENTION_DEADLINE (10 s) before each attempt after the first. A single-item write that reaches it answers ServiceUnavailable (HTTP 503), which the SDKs retry. A transaction that reaches it answers TransactionCanceledException with a TransactionConflict reason per item, as it does when its attempts run out and as the service answers a transaction that loses to contention. The GSI backfill loop runs in the background worker and takes no deadline.
The deadline and the 1,000-attempt budget take about the same time against a contended key, so a test could not say which bound ended a write. The engine now carries the deadline, set to CONTENTION_DEADLINE in every production construction, and a storage test sets it: 2 s makes the deadline the bound, 600 s makes the attempt budget the bound. Three paths that had no test now have one. A contended PutItem, DeleteItem and UpdateItem that reach the deadline each answer the transient error the engine answers as HTTP 503, with the operation and the deadline in the message. A TransactWriteItems that reaches the deadline answers a canceled transaction with a TransactionConflict reason per operation, which is what the service answers a transaction that loses to contention. The test named for the attempt budget reaches the attempt budget again and asserts the 1,000-attempt message rather than either message. Each test is bounded to the deadline it sets, so a loop that stopped reading the deadline fails it. The doc comment on the deadline said both bounds answer the same HTTP 503, which holds for PutItem and TransactWriteItems only. On this branch DeleteItem and UpdateItem answer 503 on the deadline and 400 TransactionConflictException when their attempt budget runs out first, measured in 2 of 4 rounds against a document a transaction held. The comment and the RFC paragraph now say that, and name the change that gives all three single-item writes one answer.
… has tests/test_put_item_gsi_contention.py asserted that all 2,016 puts of 32 writers on one key answer 200. The retry deadline this PR adds makes that false under load: a put that has been losing for ten seconds is shed with 503 ServiceUnavailable. The file failed 1 of 4 runs in the shape CI uses, two xdist workers on one server, and failed with 240 of 2,016 shed when another hot-key load shared the server. The test now asserts the contract the retry loop gives. Every call answers 200 or 503 ServiceUnavailable, never 400 TransactionConflictException and never 500; the stored item is one writer's item, whole, carrying the gk that writer sent with that w; and the index holds exactly one entry for it. The number shed depends on the load the server is under, so the test reports it instead of asserting it. The docstring keeps what the service answered to this shape, 200 to every call, and says where ExtendDB diverges. docs/differences-from-dynamodb.md gets a row for that divergence under Capacity and Throttling, with the measurements at 64 and 256 writers and what each operation answers at each bound.
robinnsc
left a comment
robinnsc
left a comment
There was a problem hiding this comment.
The single-item fix holds up. On a local MongoDB 7 with a GSI + stream table and 32, 64 and 256 writers on one key, PutItem only ever answered 200 or 503. The stored item, its single index entry and the stream (one record per commit, none for a shed put, an unbroken old-image chain) were consistent after every run. UpdateItem, DeleteItem and TransactWriteItems on the same key were consistent too, and conditional puts weren't retried into wrong answers. The longest single-item call was 11.8 s at 256 writers, against 28 s on main. Uncontended puts cost the same as before. The 9 new storage tests pass, and #431's create-race fix holds (2,880 of 2,880 racing creates answered 200).
Two things I'd look at before merging, both in the first two inline comments: BatchWriteItem, and the merge with current main not compiling in test builds.
Smaller notes:
- On a GSI + stream table, writes to different keys that conflict while writing the index or the stream counter still answer 500, because those errors map to
Internalinsync_indexes_in_sessionand the stream helpers. A mixed BatchWriteItem + PutItem run gave 68 of 240 batches and 9 of 240 puts as 500 on head (main is similar). If that's #437's scope, it might be worth saying so, since the description says no 400 or 500 in any run. - A hot key keeps mongod at about 2 cores for as long as the contention lasts (about 1,800-1,900 transactions/s at 64 and 256 writers), since the backoff cap is still 12.8 ms and losers now retry for up to 10 s. That seems like a fair trade-off, just worth a sentence.
- The overshoot past 10 s (up to 12.4 s at 256 writers) comes from the attempt in flight waiting on the 50-connection pool and the commit retries. The doc comment's "deadline plus one attempt" covers it, but the attempt can be a couple of seconds on its own.
- Small doc nits: the orphaned doc comment on
fn hash_keyin the test module; the RFC decision-table row atdocs/rfcs/0000-mongodb-backend.md:238still says theUnknownTransactionCommitResultlabel is detected, which the rewritten paragraph now contradicts;contention_deadline_exhaustedprintsas_secs(), so a sub-second test deadline reads "0 seconds".
|
|
||
| let tx_options = self.write_transaction_options(); | ||
| let mut lost_create_races = 0; | ||
| let contention_started = std::time::Instant::now(); |
There was a problem hiding this comment.
The clock starts per storage call, and batch_write_item.rs:127-138 / :181-191 call put_item / delete_item once per item, so a BatchWriteItem gets 10 s per item. With items held and released 7-9 s apart, a 10-item batch answered 200 after 70 s, and with botocore's default 60 s read timeout the client timed out with 8 of 10 written. Main answered 400 in 0.4 s.
The bigger effect is the 503 itself. Earlier items in the batch have already committed when a later one gives up, and since 503 is retried by the SDK (main's 400 wasn't), the whole batch is resent. Repro with [delete A, put B], B held for 14 s, and another client putting A=v2 at 5 s, which got 200: the batch's resend deleted A again, so the final state is A absent, with stream records REMOVE v0, INSERT v2, REMOVE v2. With [put A=v1, put B] the final state was A=v1 after the other client's v2. On main the same shapes answered 400 once, and v2 survived.
DynamoDB returns items it couldn't write in UnprocessedItems, so the SDK never resends the applied ones. Could consider carrying one deadline per request into the batch, and returning the item that gave up plus the rest as UnprocessedItems rather than failing the batch. ImportTable (import_export.rs:154) has the same per-item shape, though there it's a fresh table.
There was a problem hiding this comment.
Fixed in 6c36c14. The BatchWriteItem handler now runs inside a request clock scope (the new extenddb_storage::request_clock), and the MongoDB contention loops measure CONTENTION_DEADLINE from the start that scope carries, so the requests of one batch share one 10 s deadline. A request that gives up with the transient error, or with TransactionConflict (DeleteItem's spent budget on this branch), is returned in UnprocessedItems, and the requests after it are still tried, each with at least one attempt. The batch answers the error only when none of its requests was applied, so a client that retries that error resends nothing that took effect. The capacity of an unprocessed request is still counted.
DynamoDB does the same. With TransactWriteItems calls updating one item and client retries off, 46 of 80 [put held, put free] batches and 53 of 80 [put free, put held] batches answered 200 with the held request in UnprocessedItems and the free item written, each reporting the 2 capacity units of both requests. A batch whose only request is on the held item answered 500 InternalServerError in 54 of 80 calls and never 200 with its request unprocessed. ExtendDB answers that batch with 503.
tests/test_mongodb_batch_write_contention.py runs your repro: [delete A, put B], B held past the deadline by a mongosh transaction, another client putting A=v2 at 5 s, and the SDK's default retries on the batch client. On this head the batch answers 200 in 10.01 s with only the put of B in UnprocessedItems, and A holds v2. Against the build before this commit the same test fails: the batch answered 200 after 15.02 s with nothing unprocessed, because the first send answered 503 at the deadline and the SDK resent the batch, and the resend deleted A=v2 and wrote B=v1 once the hold ended. A second test holds B1 for 6 s and B2 past the deadline: on this head the batch answers in 10.02 s with B2 unprocessed. The build before this commit answered after 14.63 s with nothing unprocessed, because B2's own deadline started 6 s in and B2 committed when its hold ended.
ImportTable is not changed. It writes a table it created at the start of the import, so its puts meet contention only when another client writes that table during the import, and it runs outside any scope, so each of its puts keeps a deadline of its own.
| /// construction uses. A test sets it short to make the deadline the bound | ||
| /// a contended write reaches, or long to make the attempt budget the | ||
| /// bound, instead of racing the two. | ||
| pub(crate) contention_deadline: std::time::Duration, |
There was a problem hiding this comment.
git merge-tree against current main is clean, but cargo check -p extenddb-storage-mongodb --tests on the merged tree fails with E0063: missing field contention_deadline at stream_records_live_tests.rs:46. That literal came in with #543 after this branch merged main. It's the same kind of break as the #384 one the description mentions, and #431 on its own also fails against main (its with_scratch_engine literal). A single test constructor for MongoEngine might save the next round of these.
There was a problem hiding this comment.
Fixed in 9212328, after merging main in ac1f3b4. MongoEngine::new now builds the engine in from_parts, and a test-only MongoEngine::for_test calls the same function. The literals in backup_live_tests.rs, stream_records_live_tests.rs, snapshot_engine.rs and the restore backfill test call for_test and set only the fields they change with struct update syntax, so a field added to MongoEngine is set in from_parts and nowhere else. The contention tests moved into data_engine_live_tests_contention.rs and build their engine through the live test module's scratch_engine, setting only contention_deadline. On the merged tree cargo clippy -p extenddb-storage-mongodb --all-targets -- -D warnings is clean and the crate's unit and live tests pass.
| # the server is under, so it is reported rather than asserted; what is | ||
| # asserted is that a shed put is the only failure a caller can get. | ||
| print(f"{len(failures)} of {total} puts were shed, {total - len(failures)} committed") | ||
| assert set(failures) <= {SHED}, ( |
There was a problem hiding this comment.
This only checks the error class, so any build that answers 503 at the first conflict passes as long as one put commits. That includes the old budget of 50 with the new give-up. The storage tests do pin the budget (the deadline tests and the 256-writer test fail with 50), so it's not unguarded, but a loose floor here, say at least half committed, would give the wire test some teeth without the 0-shed flakiness the description mentions. Similarly, unconditional_put_that_spends_its_attempt_budget_gives_up_with_a_transient_error builds its expected message from TRANSIENT_RETRY_ATTEMPTS and only bounds time from above, so it can't tell which budget ran.
There was a problem hiding this comment.
Fixed in d9e3299. The wire test now also requires at least half of the 2,016 puts to commit, so a server that sheds most of them fails it. I kept the floor at half because the shed count follows the server's load (240 of 2,016 with a second hot-key load on the same server), and the storage tests pin the budget and the deadline. On this head the test passed with 0 of 2,016 puts shed, at a host load average of about 14.
The attempt-budget test now expects "PutItem: the write lost 1000 attempts to concurrent writers; retry the request" written out, and requires the give-up to take at least 5 s. The backoff sleeps of 1,000 attempts alone come to about 6.4 s, against about 0.3 s for 50 attempts. With TRANSIENT_RETRY_ATTEMPTS set to 50 the test fails on the message, which reads "lost 50 attempts", and the whole test finished in 0.56 s.
| kwargs["endpoint_url"] = endpoint_url | ||
| if endpoint_url.startswith("https://"): | ||
| kwargs["verify"] = False | ||
| return boto3.client(**kwargs) |
There was a problem hiding this comment.
nit: this module-scoped client is shared by all 32 writer threads. A client per thread would avoid the occasional TLS error a shared botocore client gives under threads. Here that would fail the test rather than pass it, so it's a flake risk, not a false pass.
There was a problem hiding this comment.
Fixed in d9e3299. Each of the 32 writer threads now gets a client of its own from the raw_clients fixture, so no botocore client is shared between threads. The test passed on this head with 0 of 2,016 puts shed.
| /// 503 whichever bound it reaches. | ||
| /// | ||
| /// The deadline is read before an attempt starts, so a request answers within | ||
| /// the deadline plus one attempt. It governs the loops that serve a request: |
There was a problem hiding this comment.
The TTL worker reaches one of these loops too: ttl_worker.rs:474 calls DataEngine::delete_item with a condition, so on a GSI or stream table it takes the transaction loop with the deadline. A contended expired item can now hold the sweep for up to 10 s (or 1,000 attempts) where it was 50 attempts before. That's probably fine, but the comment says only request loops take the deadline.
There was a problem hiding this comment.
Fixed in efe8bc9. The comment now says that the TTL worker's conditional delete runs the delete loop and takes the deadline too, so an expired item that stays contended holds the sweep until the deadline or the 1,000 attempts end that loop, where 50 attempts ended it before. The worker then logs the failure and moves to the next item (ttl_worker.rs, the Err(e) arm after DataEngine::delete_item). The code is unchanged: the sweep runs outside any request clock scope, so each of its deletes measures its own deadline.
| // this loop does. The error means another put's item is already in | ||
| // the collection, so the next attempt finds that item, replaces it | ||
| // and returns it as the old image; if a delete removed that item | ||
| // first, the next attempt inserts. Attempts share the transient |
There was a problem hiding this comment.
nit: since 544103c the create race has its own CREATE_RACE_ATTEMPTS budget, so "Attempts share the transient retry budget" here and "uses the same retry budget and backoff as a write conflict" near line 563 are out of date.
There was a problem hiding this comment.
Fixed in efe8bc9. The sessionless path's comment now says it makes at most CREATE_RACE_ATTEMPTS attempts, and the duplicate-key arm's comment says the transaction path counts each such attempt against CREATE_RACE_ATTEMPTS and gives up with the create race error once that budget is spent.
UpdateItem's native upsert, which came in with main's #473, now counts its create race against CREATE_RACE_ATTEMPTS too (88420dd).
| | Provisioned throughput | Token bucket per table/partition | Token bucket per table/partition, matching DynamoDB's burst and refill behavior | | ||
| | On-demand capacity | Automatic scaling | Fixed initial burst capacity (4000 WCU / 12000 RCU), no auto-scaling | | ||
| | Throttling | Always on; throttles requests that exceed provisioned/burst capacity. No setting to disable | Configurable via `throttling_enabled` runtime setting (default: `true`) | | ||
| | Sustained contention on one item (MongoDB backend) | Applies concurrent single-item writes on one key one after another and answers 200 to each. Measured to every call at 32 and 64 writers putting one key of a table with a GSI. A `PutItem` is answered 400 `TransactionConflictException` only while an in-flight `TransactWriteItems` holds the item (33 of 80 calls measured), and a `TransactWriteItems` that loses to other transactions on one key is answered 400 `TransactionCanceledException` with a `TransactionConflict` reason (107 of 160 calls measured) | A write on a table with a secondary index writes the item and its index entries in one transaction, so concurrent writes on one key abort each other and are retried. One request's retry loop stops after 10 seconds (`CONTENTION_DEADLINE`) or 1,000 attempts, whichever comes first, and the request is then shed instead of waiting out the queue of writers on that key: `PutItem` answers 503 `ServiceUnavailable` on either bound, which the SDK retry modes retry and which succeeds once the contention has passed. `DeleteItem` and `UpdateItem` answer 503 on the deadline and 400 `TransactionConflictException` when their 1,000 attempts run out first. `TransactWriteItems` answers `TransactionCanceledException` with a `TransactionConflict` reason per item on either bound, which is the answer the service gives. Measured with one GSI table and all writers on one key: at 64 writers all 1,280 puts committed and none were shed, with the longest call 9.7 s; at 256 writers 1,262 of 5,120 committed and 3,858 were shed, with the longest call 11.7 s. Without the deadline the same 256-writer shape left calls waiting 99.5 s, so a client sees a shed request where it used to see a request outlive its own read timeout. The other three backends keep writes on one key waiting rather than shedding them, and the deadline does not apply to them | |
There was a problem hiding this comment.
"at 64 writers all 1,280 puts committed and none were shed" didn't hold on a moderately loaded host here: 106 and 149 of 1,280 were shed at load averages 6.8 and 10.5. The description already says the shed count depends on load. Might be worth saying the same in this user-facing row, or keeping only the 256-writer bound.
There was a problem hiding this comment.
Fixed in efe8bc9. The row now says that how many requests are shed depends on the load on the server's host, keeps only the 256-writer measurement, and drops the 64-writer claim. It also says how a BatchWriteItem answers a request that reaches the deadline.
Main added a live test module for the data engine in #473: its tests are #[ignore]d, read EXTENDDB_TEST_MONGODB_URI through one uri() helper, fail rather than skip when a run passes --ignored without it, and run in the mongodb-rust job through devtools/run-mongodb-tests --rust-live. Its scratch_engine helper builds the engine with MongoEngine::new and struct update syntax, so a field added to MongoEngine needs no change there. The two create-race storage tests move into that module as data_engine_live_tests_put_race.rs. They used their own URI gate, which printed SKIP without the variable, and a with_scratch_engine helper that built MongoEngine from a struct literal. Against main that literal missed the four fields #384 added, and the test build failed with E0063. Both helpers are gone, and the tests check what they checked before. devtools/run-mongodb-tests exported EXTENDDB_TEST_MONGODB_URI a second time for these tests. Main exports it since #473, so the line is removed.
The comment on the duplicate-key arm of the transaction path said the arm handles a put that lost the create race to another put. It does not. On MongoDB 7.0, two snapshot transactions that upsert the same _id fail the second upsert with WriteConflict (code 112, "WiredTigerIdIndex::_insert: index: _id_") and the TransientTransactionError label, both while the first transaction is open and after it committed past the second one's snapshot. TxErr::from already retried that error before this branch. The comment now says the arm is a defence: it is reached only when a document in the transaction's snapshot holds the item's _id without matching the key filter, which is the document the storage tests insert.
test_put_item_create_race.py shared one boto3 client across its 8 writer threads. A botocore client used from several threads at once can fail a request with a TLS error, and with retries off and the MongoDB xfail marker gone, that error fails the test. Each writer now has its own client.
Merged with main, the storage-mongodb test build failed with E0063: the MongoEngine literals in stream_records_live_tests.rs (from #543) and in snapshot_engine.rs (from #395) do not set contention_deadline. Each test that needed an engine listed every field, so each field added to MongoEngine broke every such literal. MongoEngine::new now builds the engine in from_parts, and a test-only MongoEngine::for_test calls the same function with the driver's default pool of 10 connections. The test literals in backup_live_tests.rs, stream_records_live_tests.rs, snapshot_engine.rs and the restore backfill test now call for_test and set only the fields they change with struct update syntax. A new field is set in from_parts and nowhere else. The four contention storage tests move from the data_engine test module into data_engine_live_tests_contention.rs, next to the create-race tests #431 moved. They build the engine through the live test module's scratch_engine, are #[ignore]d like the other live tests and run under devtools/run-mongodb-tests --rust-live, and set the deadline with struct update syntax. The with_scratch_engine helpers and the second URI gate go with them. What each test checks is unchanged.
…deadline BatchWriteItem calls put_item or delete_item once per request, and each call started its own CONTENTION_DEADLINE, so a batch of contended requests could keep the client waiting 10 s per request. When a later request gave up with the transient error, the handler failed the whole batch with HTTP 503 after the earlier requests had committed. The SDKs retry a 503, so the retry sent those requests again, and a put or delete that another client had overwritten in between was applied a second time. The new extenddb_storage::request_clock module carries the start of an API request in a task-local value. The BatchWriteItem handler runs inside request_clock::scope, and the MongoDB contention loops measure the deadline from that start (contention_clock_start), so the requests of one batch share one deadline. The first attempt of every request still runs, so a request that comes after the deadline is tried once. Outside a scope a loop measures from the start of its own call, as before. A request whose write fails with StorageError::Transient or StorageError::TransactionConflict is returned in UnprocessedItems, and the requests after it are still tried. On MongoDB the first is the contention give-up and the second a spent attempt budget of DeleteItem; neither applied the write. The batch answers the error only when none of its requests was applied, so a client that retries it resends nothing that took effect. Every other error still fails the batch. The capacity of a request returned unprocessed is still counted. Amazon DynamoDB answers a batch this way. Measured in us-east-1 with client retries off while TransactWriteItems calls kept updating one item: 46 of 80 batches [put held, put free] and 53 of 80 batches [put free, put held] answered 200 with the held item's request in UnprocessedItems and the free item written, each reporting 2 capacity units, the units of both requests. 54 of 80 batches with only the held request answered 500 InternalServerError, and none answered 200 with its request unprocessed. The handler change applies to every backend: a batch put or delete that returns one of these two errors on another backend is now returned in UnprocessedItems where it failed the batch before. Tests: - request_clock: a call sees the start of the scope it runs in, and an inner scope replaces it until it ends. - the_contention_clock_starts_with_the_request_scope: the MongoDB loops read the scope's start, and their own start outside a scope. - live_a_write_in_a_request_scope_measures_the_deadline_from_the_scope_start: in a scope that started 5 s ago with a 2 s deadline, a put of a held item gives up after one attempt and a put of a free item commits. - tests/test_mongodb_batch_write_contention.py: the shape from the review, [delete A, put B] with B held past the deadline and another client putting A at 5 s, now answers 200 with only the put of B unprocessed and the other client's A kept; and in a batch whose first request is held for 6 s and whose second is held past the deadline, the second request gives up when the batch's deadline passes and is returned unprocessed.
…empt budget tests/test_put_item_gsi_contention.py checked only the error class of the calls that failed, so a server that gives up at the first conflict passed as long as one put committed. The test now also requires at least half of the 2,016 puts to commit. The floor is loose on purpose: the shed count depends on the server's load (240 of 2,016 with a second hot-key load on the same server), and the storage tests pin the budget and the deadline. Each writer thread now has a client of its own. A botocore client shared by threads can fail a request with a TLS error, which with client retries off fails the test. live_unconditional_put_that_spends_its_attempt_budget_gives_up_with_a_transient_error built its expected message from TRANSIENT_RETRY_ATTEMPTS and bounded the time only from above, so it also passed with a budget of 50. It now expects the message for 1,000 attempts written out, and requires the give-up to take at least 5 s: backoff_sleep sleeps a uniform draw from 0 to 12.8 ms before all but the first eight attempts, so the sleeps of 1,000 attempts alone come to about 6.4 s, against about 0.3 s for 50 attempts.
- The sessionless PutItem path and the duplicate-key arm of the transaction path said their attempts share the transient retry budget. Both count against CREATE_RACE_ATTEMPTS since the create race got a budget of its own. - CONTENTION_DEADLINE's comment said only the loops that serve a request take the deadline. The TTL worker's conditional delete runs the delete loop too, so an expired item that stays contended holds the sweep until the deadline or the attempt budget ends that loop; the worker then logs the failure and moves on. The comment now says so, says that the requests of one BatchWriteItem share one deadline, and says that the attempt already running when the deadline passes has no bound of its own: it waits for a connection from the pool and for the commit's own retries, and the longest call at 256 writers on one key measured 11.7 s. It also states the cost of retrying for up to 10 s, CPU on the MongoDB server for as long as a key stays hot. - contention_deadline_exhausted printed the deadline with as_secs(), so a deadline a test sets below one second read as "0 seconds". It now prints as_secs_f64(): "10 seconds", "2 seconds", "0.25 seconds". - The RFC's decision table still listed the UnknownTransactionCommitResult label as a conflict the body retry detects, which the write conflict section contradicts. The row now says that label retries commitTransaction in place and never re-runs the body. - The RFC section and the code now name #475, the fix for #437, as the change that gives DeleteItem and UpdateItem the transient give-up. - docs/differences-from-dynamodb.md said all 1,280 puts committed at 64 writers. On a loaded host 106 and 149 of 1,280 were shed. The row now says the shed count depends on the host's load, keeps the 256-writer measurement, and says how a BatchWriteItem answers a request that reaches the deadline.
0d3f76d rewrote the comment on the transaction path's duplicate-key arm: a put that loses the create race inside a snapshot transaction gets WriteConflict 112, so the arm only sees a document that holds the item's _id without matching the key filter. Two comments in the same function and the doc of create_race_exhausted still described the arm as the create race, and now say what it is. The create race test's client fixture no longer explains why each writer has its own client; the docstring states what the fixture gives.
…tem-gsi-conflict Brings #431's b7c9258, which describes the transaction path's duplicate-key arm as a document that holds the item's _id rather than as the create race. The one conflict is the PutItem transaction path's give-up after its loop. This branch keeps its own version: the loop gives up with the create race error once CREATE_RACE_ATTEMPTS attempts have met such a document, and a spent contention budget answers the transient error. The comment on that check and the doc of create_race_exhausted now use #431's wording, and the doc says the transaction path gives up after CREATE_RACE_ATTEMPTS attempts rather than after every attempt.
The comment on CONTENTION_DEADLINE gave the CPU cost of retrying a hot key for up to 10 s without a measurement from this tree. Measured on this head with 64 and then 256 writers putting one key of a GSI table for 20 s, client retries off, on a 16-core host at a load average of 18 to 21: - 64 writers: 457 puts committed and 12 were shed; mongod used a median of 132 percent CPU and ran 35,063 transactions, 34,606 of them aborted (1,666 a second). - 256 writers: 150 committed and 444 were shed; mongod used a median of 128 percent CPU and ran 39,335 transactions, 39,185 of them aborted (1,811 a second).
…EATE_RACE_ATTEMPTS Main's #473 added upsert_native_update, whose create race loop is sized by TRANSIENT_RETRY_ATTEMPTS. That is 50 on main and 1,000 on this branch, and the merge ac1f3b4 gave the loop this branch's value with no conflict, so a lost create race on _id retried up to 1,000 times with backoff, about 6.4 s of backoff before it answered TransactionConflictException, where main stops after 50. The loop now counts against CREATE_RACE_ATTEMPTS, the 50-attempt budget a lost create race has on the PutItem paths. Also from the review of this branch before the push: the BatchWriteItem doc comment says which writes answer TransactionConflict; the differences row says which error a batch answers when none of its requests was applied, and what the service answered; and a wire test checks that a batch whose only request is held past the deadline answers 503 and applies nothing.
|
The branch now merges main (31f7c6e, main at 03baa32; the first merge was ac1f3b4) and #431's current head (63df25e, #431 at b7c9258). Both merges of #431 conflicted in Both points you would hold the PR for are fixed:
The smaller notes:
Tests run on the final head, 31f7c6e:
88420dd came out of a check before this push: main's #473 loop in UpdateItem's native upsert had taken this branch's 1,000-attempt budget in the merge, and now counts against |
test_unconditional_puts_on_one_key_of_a_gsi_table_commit_or_are_shed holds the backend to the MongoDB retry loop's contract: every contended put answers 200 or a shed 503. The Cassandra backend answers some of these puts with 400 TransactionConflictException (17 of 2,016 in this branch's cassandra-pytest job on 2026-10-24, 31 of 2,016 on 2026-11-03), where the service answers 200 to all of them. That is a Cassandra write path problem outside this change, filed as #802, so the test is skipped on Cassandra until it is fixed. SQLite, PostgreSQL and MongoDB still run it.
Fixes #426. Stacked on #431 (
fix/mongodb-upsert-duplicate-key): this branch merges #431's branch and should merge after it.On MongoDB a put on a table with an index writes the item and its index entries
in one transaction. MongoDB aborts the loser of a same-document conflict at
once instead of queueing it, so of N writers on one key one commits per round
and the rest start over, and a writer needs on the order of N attempts, more
under the connection pool's queueing. A budget of 50 attempts ran out for a
share of the puts, and the put answered the 400 TransactionConflictException
that main reserves for a single-item write that could not serialize.
Two bounds now end a contended write, and both answer an error the SDKs retry.
The contention budget is 1,000 attempts, and a PutItem that spends it returns
StorageError::Transient, answered as HTTP 503 ServiceUnavailable. A request'sretry loop also stops after
CONTENTION_DEADLINE, 10 seconds, because theattempt budget bounds attempts and not the time a caller waits: at 256 writers
on one key, 89 of 5,120 calls were still waiting when the client's 120 s read
timeout fired. With the deadline the longest of 6,400 calls on that shape was
11.7 s.
The requests of one BatchWriteItem share one deadline, and a request that
reaches it is returned in
UnprocessedItemswhile the other requests areapplied. See "BatchWriteItem" below.
Merged with main
This branch merges origin/main at 03baa32. After #543 added a
MongoEnginestruct literal instream_records_live_tests.rs, the merged tree was conflict free and the test build failed with E0063 (missing fieldcontention_deadline), assnapshot_engine.rsdid after #395.MongoEngine::newnow builds the engine infrom_parts, and a test-onlyMongoEngine::for_testcalls the same function, so the test literals set only the fields they change and a new field is set in one place. The contention tests moved intodata_engine_live_tests_contention.rsand build their engine through the live test module'sscratch_engine. They are#[ignore]d like the other live tests and run underdevtools/run-mongodb-tests --rust-live, which CI runs.What the service does
Measured against DynamoDB in us-east-1 with client retries off
(
retries={"max_attempts": 0}), on existingbb-tables with a string hashkey, a GSI and a stream, on 2026-10-23 and 2026-10-24.
ADD #c :one SET #g = :gwith ALL_NEW on one keySET l = list_append(l, :m)on one keyThe service sheds none of the single-item load at any writer count measured,
including 64 concurrent writers on one key. It answers a single-item write with
TransactionConflictExceptiononly while aTransactWriteItemsholds the item,which this backend cannot tell apart from a race against other puts. It does
shed concurrent transactions on one key, with the same
TransactionCanceledExceptionandTransactionConflictreason this backendanswers when a transaction stops retrying.
What each bound answers
Measured on this branch with another session holding the contended item inside
a MongoDB transaction it never commits, so every attempt of the retry loop
conflicts. Four writes ran at once on that key, client retries off, four
rounds, host load 11 to 13:
PutItem answers 503 on either bound, and
TransactWriteItemsanswersTransactionCanceledExceptionwith aTransactionConflictreason per item oneither bound. DeleteItem and UpdateItem answer 503 on the deadline and 400
TransactionConflictExceptionwhen their 1,000 attempts run out first, whichthe delete did in 2 of the 4 rounds above: the two bounds take about the same
time on this shape, so which one the loop reaches decides which error class the
caller is told. Those two 400s are what main answers for every spent budget on
those paths, and #475, the fix for #437, routes them through the same transient
give-up. After #475 merges, every single-item write answers 503 on either
bound. The deadline moves the common case on those two paths to 503, so the
400 is reached less often than on main.
A client on an SDK retry mode retries the 503 and the retry commits once the
contention has passed. The message the give-up carries ("PutItem: the write
lost to concurrent writers for 10 seconds; retry the request") reaches the
server log; the client gets the fixed 503 body every transient error on this
backend answers with.
BatchWriteItem
BatchWriteItem calls the storage layer once per request. Before this change
each call started its own deadline, so a batch of contended requests could
keep a client waiting 10 s per request, and when a later request gave up the
handler failed the whole batch with 503 after the earlier requests had
committed. The SDKs retry a 503, so the retry applied those requests again,
after any write another client had made to the same items.
The handler now runs inside a request clock scope
(
extenddb_storage::request_clock), and the MongoDB contention loops measurethe deadline from the start that scope carries, so the requests of one batch
share one deadline. The first attempt of every request still runs. A request
whose write fails with
TransientorTransactionConflictis returned inUnprocessedItems, and the requests after it are still tried. The batch answersthe error only when none of its requests was applied, so a client that retries
it resends nothing that took effect. ExtendDB answers 503 there, where the
service answered the single-request batch in the table below with 500
InternalServerError. The capacity of a request returned
unprocessed is still counted. On the other backends a batch put or delete that
returns one of those two errors is now returned unprocessed too, where it
failed the batch before.
Measured against DynamoDB in us-east-1 on 2026-11-03, client retries off, while
TransactWriteItems calls kept updating one item H on an on-demand table with a
string hash key:
The shape from the review, against a mongosh session that holds B in a
transaction for 15 s:
[delete A, put B]sent by a client on the SDK's defaultretry mode, while another client puts A=v2 5 s in. On this head the batch
answers 200 in 10.01 s with only the put of B in
UnprocessedItems, and A holdsv2. On the build before this change it answered 200 after 15.02 s with nothing
unprocessed: the first send answered 503 at the deadline, the SDK resent the
batch, and the resend deleted A=v2 and put B once the hold ended.
ImportTable is not changed. It writes a table it created at the start of the import, so its
puts meet contention only when another client writes that table during the import,
and it runs outside any scope, so each of its puts keeps a deadline of its own.
#475 also changes this handler to return such requests in
UnprocessedItems,for a wider set of errors, so the one of the two PRs that merges second carries
one version of it.
What each status means here
UnprocessedItems(BatchWriteItem): the requests listed reachedthe deadline or their attempt budget and were not applied; the others were.
10 s deadline. The error-handling documentation lists ServiceUnavailable
(HTTP 503) as "OK to retry? Yes", and the SDK retry modes retry it, whereas a
400
TransactionConflictExceptionis handed to the application.operation: a
TransactWriteItemsreached either bound. This is what theservice answers a transaction that loses to contention on one key, measured
in 107 of 160 calls.
an item another writer had created and removed again. That is fix(mongodb): retry an unconditional PutItem that loses the create race #431's answer,
the same one the PostgreSQL backend gives when its own retries of this race
run out.
One constant sized three loops, and which loops take the deadline
TRANSIENT_RETRY_ATTEMPTSsized three different loops, so raising it to 1,000raised all three. Two of them are not contention loops:
TRANSIENT_RETRY_ATTEMPTS = 1000now sizes the contention loops only: put,delete, update, transact, the GSI backfill, and the transaction of the
startup key repair pass that main gained in fix(mongodb): store the key when UpdateItem creates an item #473. Set by the measurements in
its doc comment: 50 attempts left 15 to 25 percent of the puts of 32 clients
on one key without a commit, 400 left one put in 2,016 on a loaded host.
COMMIT_RETRY_ATTEMPTS = 50sizescommit_write_transaction's in-placeretry after an
UnknownTransactionCommitResult. That label means thecommit's outcome is unknown, the commit is idempotent and the body must not
be re-run, so those attempts wait for one reply rather than racing other
writers. No measurement here sizes a longer wait, and a longer one only
delays the HTTP 500 the path answers when the outcome stays unknown.
CREATE_RACE_ATTEMPTS = 50sizes the create-race give-up, on PutItem'ssessionless path and transaction path and in UpdateItem's native upsert,
which main gained in fix(mongodb): store the key when UpdateItem creates an item #473. fix(mongodb): retry an unconditional PutItem that loses the create race #431 measured that budget with
this backoff: a put blocked by a document another writer removes 100 ms
after the put starts commits well inside it, and fix(mongodb): retry an unconditional PutItem that loses the create race #431's storage tests fail
with the cap put back to 5. The transaction path now gives up as soon as
that budget is spent, rather than carrying lost races through the contention
budget.
CONTENTION_DEADLINE = 10 sbounds the four loops that serve a request: put,delete, update and transact. It is read before an attempt starts and never
before the first one, so a request answers within the deadline plus the attempt
already running, and no request answers 503 without having tried the write
once. The attempt already running has no bound of its own: under many writers
on one key it waits for a connection from the pool (50 by default) and for the
commit's own retries, and the longest call at 256 writers on one key was
11.7 s. A loop measures the deadline from the start of its request, which for
BatchWriteItem is the start of the batch. The TTL worker's conditional delete
runs the delete loop too, outside any request, so an expired item that stays
contended holds the sweep until the deadline or the attempt budget ends that
loop, where 50 attempts ended it before; the worker then logs the failure and
moves to the next item. The GSI backfill loop keeps the attempt budget and
takes no deadline: it runs in the background worker, where no caller is
waiting for a reply and giving up would leave the index CREATING. The
sessionless PutItem fast path, UpdateItem's native upsert and
commit_write_transactiontake no deadline either, because none of them is acontention loop. The key repair pass runs in a background task at startup and
takes no deadline, as the GSI backfill does.
The engine carries the deadline as a field set to
CONTENTION_DEADLINEinevery production construction, which is what lets a test decide which bound it
exercises instead of racing 1,000 attempts against a wall clock.
How long each bound takes
backoff_sleepdraws each sleep uniformly from 0 to the current base, cappedat 12.8 ms, so 1,000 attempts sleep about 6.4 s in total on average and 12.8 s
at worst, before the attempts' own round trips.
Measured on MongoDB 7.0 as a single-node replica set, against a transaction
another session holds open, where each attempt also waits out MongoDB's lock
timeout:
What the bounds buy and what they cost
MongoDB 7.0 single-node replica set, client retries off, a fresh GSI table per
run, all writers on one key, 20 puts each. The server was the only extenddb
process on the host; the host itself was shared. Host load average 5.6 to 22.3
for the budget-only columns and 5.0 to 15.8 for this head, so the two sets are
not from an idle host and are not from the same load. How many calls are shed
depends on that load: the 64-writer row below shed none, and the same shape
shed 106 and 149 of 1,280 on another host at load averages 6.8 and 10.5.
At 64 writers on one key the raised budget turns 943 shed calls into commits,
and on that run the deadline was never reached, so nothing was shed and the
longest call fell from 28.0 s to 9.7 s. At 256 writers the deadline turns
3,471 calls that used to commit after a long wait into 503s the SDK retry
modes retry, and the commit rate is 6.9 against 5.4 on a different host load.
In both runs the stored item was the last writer's and the index held one
entry for its indexed value.
The same 256-writer shape against this head's parent commit, which has the
1,000-attempt budget and no deadline, on the same host and container, 5 rounds
each: without the deadline p50 20,983 ms, p99 98,856 ms and the longest call
99,534 ms; with it p50 10,079 ms, p99 10,662 ms and the longest call 11,658 ms.
Those two runs sent 5 rounds and 20 rounds, so their commit counts measure
different amounts of work; the per-call latencies are measured on the same
shape and the same host. A client on botocore's default 60 s read timeout gets
an answer it can retry on this shape instead of a socket timeout.
The cost is CPU on the MongoDB server. The losers of a hot key retry with a
backoff capped at 12.8 ms for up to 10 s, so mongod stays busy committing and
aborting transactions on that key for as long as the contention lasts. With
64 and then 256 writers putting one key of a GSI table for 20 s, client retries
off, on a 16-core host at a load average of 18 to 21, mongod used a median of
132 and 128 percent CPU and ran 1,666 and 1,811 transactions a second, of which
98.7 and 99.6 percent aborted, while the puts committed 21.7 and 6.9 times a
second. On another host the same shapes measured about 2 cores and 1,800 to
1,900 transactions a second.
The GSI backfill keeps its attempt budget and no deadline
A table with 600 items, 48 writers putting one of those keys continuously with
client retries off, then
UpdateTableadding a GSI ong: the index reachedACTIVE 22.6 s after the
UpdateTable, which is past the 10 s deadline, andheld 600 entries for the 600 base items. The 48 concurrent writers answered
3,038 times with 200 and twice with 503. So the request path's deadline does
not stop the background backfill, which is why that loop reads no clock.
What this PR does not change
The hot-key commit rate on this backend, measured above on a shared host, is
2.4 to 6.5 commits per second at budget 50, 5.4 to 25.9 at budget 1,000, and
6.9 to 38.9 at budget 1,000 with the deadline. The service answered 1,920 calls
from 64 writers on one key in 2.5 s, about 770 per second. That comes from
aborting the loser of a same-document conflict instead of queueing it, which
neither this PR nor #475 touches. Each changes which error a losing write
receives, not the rate at which writes commit.
On a table with a GSI and a stream, writes to different keys can conflict while
they write an index entry or draw a stream sequence number, and those errors
map to
Internalinsync_indexes_in_sessionand the stream helpers, so theyanswer 500 here as on main. That is #437. #475 classifies those conflicts as
transient so the retry loop sees them. The results in this description that
show no 400 and no 500 are for writers on one key of a GSI table without a
stream.
Tests
tests/test_put_item_gsi_contention.py(skipped on Cassandra, where some ofthese puts answer TransactionConflictException, Cassandra: unconditional PutItem on one key of a GSI table answers TransactionConflictException under contention #802): 32 writers putting
one key of a table with a GSI, 2,016 puts, client retries off, each writer with a client
of its own. Every call must answer 200 or 503 ServiceUnavailable, never 400
TransactionConflictException and never 500, and at least half of the puts
must commit; the stored item must be one writer's item, whole, carrying the
gkthat writer sent with thatw; and the index must hold exactly oneentry for it. How many calls are shed depends on the load the server is
under, so the test reports that count and bounds it only loosely: it failed
1 of 4 runs in the shape CI uses when an earlier revision asserted that every
call answers 200, and 240 of 2,016 were shed when another hot-key load shared
the server. The file also runs on the PostgreSQL and SQLite integration
jobs, which shed nothing on this shape.
tests/test_mongodb_batch_write_contention.py, MongoDB only, holding itemsthrough mongosh in the runner's container: the review's
[delete A, put B]shape answers 200 with only the put of B unprocessed and keeps the other
client's A; and a batch whose first request is held for 6 s and whose second
is held past the deadline answers within the batch's deadline with the second
request unprocessed. Both fail against the build before the change: each
batch answered 200 after about 15 s with nothing unprocessed, and in the
first the other client's A was deleted. A third test checks that a batch
whose only request is held past the deadline answers 503 and applies nothing.
docs/differences-from-dynamodb.mdcarries a row for the divergence underCapacity and Throttling: what each operation answers at each bound, that the
shed count depends on the host's load, the 256-writer measurement, and how a
BatchWriteItem answers a request that reaches the deadline.
EXTENDDB_TEST_MONGODB_URI. 32 writers putting one key on the transactionpath all commit. A contended PutItem, DeleteItem and UpdateItem that reach
the deadline each answer the transient give-up naming the operation and the
deadline. A contended
TransactWriteItemsthat reaches the deadline answersa canceled transaction with a
TransactionConflictreason per operation, andneither operation is applied. A put that spends its 1,000 attempts answers
the give-up with the budget written out in the test, with the held item
unchanged, and takes at least 5 s, which the sleeps of 50 attempts cannot
reach. A put of a held item inside a request clock scope that started past
the deadline gives up after one attempt, and a put of a free item in the same
scope commits. Each of those tests sets the engine's deadline, 2 s to make the
deadline the bound and 600 s to make the attempt budget the bound, so none of
them races the two.
runs, no attempt starts once the deadline has passed, a deadline a test sets
decides the answer, the give-up carries the deadline that passed, with the
fraction of a deadline below one second); the request clock (a call sees the
start of its scope, an inner scope replaces it, and the MongoDB loops read
the scope's start).
named separately, and still report the give-up after 50 attempts on both
paths.
What fails without this branch
On MongoDB 7.0, 32 writers putting one key of a GSI table, 2,016 puts per run,
client retries off:
Each storage test of the two bounds was run against a broken version of the
path it covers, and each failed: with the loop reading the production constant
instead of the engine's deadline, all three deadline tests and the
attempt-budget test fail; with the deadline check removed from the transact
loop, the transaction test fails on its elapsed bound; with either give-up
returning
TransactionConflictinstead ofTransient, the test of thatgive-up fails on the error class. The tests added for the review were checked
the same way: with
TRANSIENT_RETRY_ATTEMPTSset to 50, the attempt-budgettest fails on its message; with the loops measuring the deadline from their own
start instead of the request's, the request scope test fails because the held
put waits out a 2 s deadline of its own.
Gates
cargo +1.97.0 fmt --all -- --check: clean.cargo +1.97.0 clippy -p extenddb-storage -p extenddb-storage-mongodb -p extenddb-engine --all-targets -- -D warnings, and the same for extenddb-storage-mongodb with itstest-hooksfeature enabled: no warnings.cargo +1.88.0 test --locked -p extenddb-storage -p extenddb-engine -p extenddb-storage-mongodb, withEXTENDDB_TEST_MONGODB_URIset so the URI-gated tests ran: 675 passed, 0 failed, 53 ignored over 13 targets.cargo +1.88.0 test --locked -p extenddb-storage-mongodb --lib -- --ignored live, against a MongoDB 7 single-node replica set: 48 passed, 0 failed.mongodb-test-hooksfeature enabled, as CI builds it:test_mongodb_batch_write_contention.py2 passed, andtest_put_item_gsi_contention.pypassed with 0 of 2,016 puts shed at a host load average of about 14. The PostgreSQL and SQLite wire jobs were not run locally for this revision.Cargo.lockunchanged, so the license notice checks do not apply.