Skip to content

fix(memory): serialize batched flushes so retries and late sends can't overtake newer state - #701

Open
tejaskash wants to merge 1 commit into
fix/read-agent-latest-batched-snapshotfrom
fix/serialize-batched-flushes
Open

tejaskash wants to merge 1 commit into
fix/read-agent-latest-batched-snapshotfrom
fix/serialize-batched-flushes

Conversation

@tejaskash

Copy link
Copy Markdown
Contributor

What

Serializes batched flushes in AgentCoreMemorySessionManager with one lock, and fixes the retry paths that put buffered data back.

  • _flush_messages_only, _flush_agent_states_only and _flush_messages run under _flush_lock.
  • _flush_messages takes the agent states before the messages, so a saved state never counts a message that wasn't saved.
  • A failed message flush prepends its messages, as fix(memory): restore latest agent snapshot from batched events #700 does for agent states.
  • A partial agent-state failure puts back only the agents that weren't sent.

Why

With flush_interval_seconds set, the timer thread and the agent thread can flush at the same time. On main and on #700 that causes four problems:

  1. A slow or failed flush of an older snapshot is written after a newer one, so read_agent restores stale state.
  2. An agent state is saved while the messages it counts are still buffered. After a crash, the restored history is wrong.
  3. A failed message flush puts the failed messages after newer ones.
  4. When one agent's send fails, agents that were already sent are written again.

Stacked on #700. GitHub retargets this PR to main when #700 merges.

How tested

  • Six new unit tests in TestConcurrentFlushes. Five fail without the fix. The threaded one passed 50 of 50 repeat runs.
  • pytest tests/bedrock_agentcore/memory/: 790 passed. ruff check and ruff format --check are clean.
  • End to end against a real AgentCore Memory resource in us-west-2, with injected delays and failures in front of the real create_event:
    • A real Strands agent with batch_size=3, the interval timer and Claude Haiku 4.5 restores the latest state and all 6 messages.
    • Five scenarios, one per bug, all pass. Without the fix, all five fail: stale v=1 restored twice, state v=2 with 1 stored message, stored order ['m2', 'm1'], and a duplicate agent-a event.

…t overtake newer state

With flush_interval_seconds set, the interval timer and the agent thread
can flush at the same time. Without a lock, a slow or failed flush could
write an older agent snapshot after a newer one, an agent state could be
saved ahead of the messages it counts, a failed message flush reordered
messages, and a partial agent-state failure re-sent agents that were
already written.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant