Skip to content

fix: answer TTL and tag calls during a table's DELETING window - #599

Open
LeeroyHannigan wants to merge 1 commit into
fix/505-deleting-window-answersfrom
fix/551-552-deleting-window-ttl-tags
Open

LeeroyHannigan wants to merge 1 commit into
fix/505-deleting-window-answersfrom
fix/551-552-deleting-window-ttl-tags

Conversation

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

Fixes #552 and #551. The branch is based on #550 (fix/505-deleting-window-answers), which adds the Table is being deleted sentence; it targets main once #550 is merged.

While a table is DELETING, Amazon DynamoDB answers these calls as follows, on the table and on the Amazon Resource Name (ARN) of its stream, measured 2026-10-25 and 2026-10-26:

Call On the table On the table's stream ARN
DescribeTimeToLive 400 ValidationException "Cannot describe time to live while table is in DELETING state: Current table state is DELETING"
UpdateTimeToLive 200, with the rules it applies to an ACTIVE table
TagResource, UntagResource 400 ResourceInUseException "Attempt to change a resource which is still in use: Table is being deleted: " 200
ListTagsOfResource 200 with the tags 200 with the stream's tags

DescribeTimeToLive on a CREATING table answers the same ValidationException with CREATING in the sentence, and is served on an UPDATING table.

ExtendDB now answers these calls this way on PostgreSQL, SQLite, MongoDB and Cassandra. Before, UpdateTimeToLive answered "Table is not in ACTIVE state", DescribeTimeToLive answered 200, and the tag calls answered ResourceNotFoundException on MongoDB and Cassandra while PostgreSQL and SQLite accepted both writes.

How it works:

  • The engine reads the table's status with describe_table. TableKeyInfo has no status, and the MongoDB and Cassandra key-info reads report a DELETING row as not found, so the tag path reads the status before the key-info read.
  • update_ttl admits DELETING beside ACTIVE on every backend.
  • For a table it read as DELETING, UpdateTimeToLive sends no data definition language (DDL) statements for the index, because the delete removes the data table or collection with its index. On PostgreSQL a disable's DROP INDEX waited with no lock timeout behind any holder of the data table, and the table's reads and writes queued behind it. On MongoDB an enable's create_index could create the dropped collection again. MongoDB create_ttl_index also reads an ACTIVE catalog document only.
  • The tag write refusal applies to the table's ARN. On the service a stream's tags stay with the stream, which it keeps describable after the table is gone (A disabled stream of a deleted table stays describable on the service #518).

Catalog cost: the status read is four catalog queries on PostgreSQL and SQLite and two on MongoDB and Cassandra, once per DescribeTimeToLive, UpdateTimeToLive, TagResource, UntagResource and ListTagsOfResource call. The tag path keeps the key-info read after it for every status but DELETING, so a tag call on an ACTIVE table sends seven queries before its tag statements on PostgreSQL and SQLite (three before this change) and four on MongoDB and Cassandra (two before). These are control-plane calls; the item and batch paths are unchanged.

Differences from the service that remain:

  • ListTagsOfResource returns the tags for the whole window on PostgreSQL and SQLite. On MongoDB it returns no tags once the delete call has removed them, and with storage-cassandra: two concurrent DeleteTable calls on one table both complete #553 Cassandra removes them in the last statements of the delete, after the drops. The service returns them for the whole window.
  • The refusal removes tag writes admitted during the window. A write whose status read came before the delete can still be written after the delete removed the tag rows, as on main.
  • For several seconds after a tag write the service refuses another tag write on the resource with LimitExceededException "Table tags are being updated: " (seven rows); a DeleteTable in that window was refused once with ResourceInUseException and the same sentence. ExtendDB has no such state.
  • After the table is gone, ListTagsOfResource on its stream ARN answers ResourceNotFoundException here, where the service answers 200 with the stream's tags. Pre-existing, filed separately.
  • ExtendDB does not validate tags. The service checks the tag key ("The Tag Key provided is invalid, Key: ") before the DELETING refusal.
  • UpdateTimeToLive has no rate limit ("Time to live has been modified multiple times within a fixed interval"), a documented difference, and no attribute check ("TimeToLive is active on a different AttributeName: current AttributeName is "), which the service applies to the disable form too.
  • On MongoDB an enable admitted while the table is ACTIVE can still create the data collection again, when its catalog read comes before DeleteTable marks the document and its create_index runs after the drop: 1 of 20 delete races through the API in one run and 0 of 80 in another, on a single-node replica set with this change; 4 of 20 on the base.

Tests:

  • Engine unit tests for the DescribeTimeToLive gate with both sentences and for the tag write gate on table and stream ARNs.
  • Storage tests on all four backends for update_ttl during the window. On PostgreSQL a disable through the handler leaves the Time to Live (TTL) index in place. On MongoDB an index build on a DELETING document creates no collection, and a test drives the five handlers over a MongoEngine with the document DELETING.
  • Dual-target wire tests on a held PostgreSQL window for every row of the table above.
  • devtools/run-mongodb-tests runs the ignored MongoDB window tests in continuous integration (CI).
  • No test drives the handlers over SQLite or Cassandra inside the window; the skip of the index DDL is pinned by the PostgreSQL storage test.

Merge notes

With #553 (fix/553-cassandra-concurrent-delete-table): one conflict in tests/python/test_deleting_window_control_plane.py, where both branches append test classes at the same point; keep both blocks, this branch's two classes first, then TestConcurrentDeleteTable. crates/storage-cassandra/tests/direct/deleting_window.rs merges on its own.

With #543 (fix/438-get-records-round-trips): devtools/run-mongodb-tests, where both add a live-test step at the same place; the resolution is one step with --ignored records_live deleting_window.

… service does

DescribeTimeToLive on a DELETING table answers 400 ValidationException
"Cannot describe time to live while table is in DELETING state: Current table
state is DELETING", and on a CREATING table the same sentence naming
CREATING; it is served on ACTIVE and UPDATING tables. UpdateTimeToLive on a
DELETING table runs the same rules as on an ACTIVE one. Amazon DynamoDB
answers these calls this way (measured 2026-10-25 and 2026-10-26). On
ExtendDB the UpdateTimeToLive write goes to the catalog row, which the
delete removes.

ExtendDB answered UpdateTimeToLive on a DELETING table with 400
ResourceInUseException "Table <name> is not in ACTIVE state", because every
backend's update_ttl admitted ACTIVE rows only, and DescribeTimeToLive with
200 on CREATING and DELETING tables, because nothing read the status.

The engine reads the status with describe_table, since TableKeyInfo carries
no status and MongoDB and Cassandra report a DELETING row as not found from
the key-info read. The read is four catalog queries on PostgreSQL and SQLite
and two on MongoDB and Cassandra, once per call. update_ttl admits DELETING
beside ACTIVE in the conditional write on PostgreSQL, SQLite and MongoDB, and
in the status check and both lightweight transactions on Cassandra. CREATING
and UPDATING keep TableNotActive, and a missing table keeps TableNotFound.

UpdateTimeToLive sends no index DDL for a table it read as DELETING, since the
delete removes the data table or collection with its index. On PostgreSQL a
disable's DROP INDEX waited with no lock timeout behind whatever held the data
table and queued the table's data-plane requests behind it. On MongoDB an
enable's create_index could create the collection again after the delete
dropped it, and the collection outlived the table. MongoDB create_ttl_index
also reads an ACTIVE catalog document only. A delete that commits between the
status read and the write meets the index work, which the base refused at the
write; on MongoDB the ACTIVE filter covers that order.

TagResource and UntagResource on the ARN of a DELETING table answer 400
ResourceInUseException "Attempt to change a resource which is still in use:
Table is being deleted: <name>", and ListTagsOfResource is served with the
tags. On the table's stream ARN both writes and the read are served. Amazon
DynamoDB answers this way (measured 2026-10-25 and 2026-10-26).

ExtendDB answered ResourceNotFoundException for all three on MongoDB and
Cassandra and accepted both writes on PostgreSQL and SQLite. An accepted
TagResource on the table ARN writes an unconditioned tag row; on PostgreSQL
the delete removes the ARN's tag rows before the catalog row, so a tag written
between the two statements could outlive the table and be inherited by the
next table of that name.

The tag path reads the table's status before the key-info read, so all four
backends reach the window's answers, and refuses the two writes when the ARN
names the table. The key-info read still follows for every status but
DELETING, so a tag call on an ACTIVE table sends seven catalog queries before
its tag statements on PostgreSQL and SQLite, where it sent three, and four on
MongoDB and Cassandra, where it sent two. The refusal removes the writes
admitted during the window. A write whose status read came before the delete
can still be written after the delete removed the tag rows, as before.

ListTagsOfResource returns the tags for the whole window on PostgreSQL and
SQLite. On MongoDB it returns no tags once the delete call has removed them;
the service returns them for the whole window.

A MongoDB storage test drives the tag and TTL handlers on a DELETING
document, and devtools/run-mongodb-tests runs the crate's ignored
deleting_window tests in CI.

Closes #552
Closes #551

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants