Skip to content

Firestore emulator v1.22.0: concurrent batchWrite to the same doc fails with ABORTED "Transaction lock timeout." after a NOT_FOUND update (regression from v1.21.0) #11160

Description

@merryharry

[REQUIRED] Environment info

firebase-tools: 15.28.1 (also 15.30.2; the regression arrives with 15.25.0, which updated the Firestore emulator to v1.22.0)

Firestore emulator: cloud-firestore-emulator-v1.22.0 (fails); v1.21.0 (works)

Platform: macOS 27.2 arm64, both jars run with java -jar on the same machine (OpenJDK 21), client Node 24. v1.22.0 reproduces the same way in a Linux Docker container, and the downstream hang shows up on Linux x64 GitHub Actions runners.

[REQUIRED] Test case

Two concurrent, non-transactional batchWrite calls to the same document, which does not exist yet:

  • an update with currentDocument.exists: true, which fails with NOT_FOUND as expected;
  • a plain set, sent right after it.

On v1.22.0 the update returns NOT_FOUND within a few ms. The set then often blocks for 2 s and fails with ABORTED: Transaction lock timeout., although no other request is touching the document by then. It looks as if the lock held by the failed update is released without waking the waiting write. v1.21.0 never does this.

No SDK is involved; the script only uses fetch (Node >= 18):

// FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 node repro.mjs 50
const H = process.env.FIRESTORE_EMULATOR_HOST ?? '127.0.0.1:8080';
const P = `repro-${Date.now()}`;
const url = `http://${H}/v1/projects/${P}/databases/(default)/documents:batchWrite`;
const post = body => fetch(url, {
  method: 'POST',
  headers: { authorization: 'Bearer owner', 'content-type': 'application/json' },
  body: JSON.stringify(body),
}).then(r => r.json());

let aborted = 0;
const N = Number(process.argv[2] ?? 20);
for (let i = 0; i < N; i++) {
  const name = `projects/${P}/databases/(default)/documents/c/d${i}`;
  const t0 = Date.now();
  const upd = post({ writes: [{
    update: { name, fields: { b: { integerValue: '2' } } },
    updateMask: { fieldPaths: ['b'] },
    currentDocument: { exists: true },
  }] }).then(r => ({ r, ms: Date.now() - t0 }));
  const set = post({ writes: [{
    update: { name, fields: { a: { integerValue: '1' } } },
  }] }).then(r => ({ r, ms: Date.now() - t0 }));
  const [s, u] = await Promise.all([set, upd]);
  const ss = s.r.status?.[0] ?? {}, us = u.r.status?.[0] ?? {};
  if (ss.code === 10) aborted++;
  console.log(`#${i} set: code=${ss.code ?? 0} ${ss.message ?? ''} (${s.ms}ms) | update: code=${us.code ?? 0} (${u.ms}ms)`);
}
console.log(`set ABORTED ${aborted}/${N}`);

[REQUIRED] Steps to reproduce

  1. Start the Firestore emulator from firebase-tools >= 15.25.0 (emulator v1.22.0), e.g. firebase emulators:start --only firestore, or run the jar directly.
  2. FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 node repro.mjs 50
  3. Repeat against v1.21.0 (firebase-tools 15.24.x, or java -jar cloud-firestore-emulator-v1.21.0.jar --port 8181).

[REQUIRED] Expected behavior

As with v1.21.0: the update fails with NOT_FOUND, and the set succeeds right away.

set ABORTED 0/50

[REQUIRED] Actual behavior

v1.22.0: about a third of the sets wait roughly 2 s and fail with ABORTED, while the update already finished in 3 ms.

#3 set: code=10 Transaction lock timeout. (2008ms) | update: code=5 (3ms)
...
set ABORTED 19/50

Sequential calls (update, then set) do not reproduce it, so the lock is not leaked permanently. It only affects a write that is already waiting when the failing write gives the lock up.

Impact: BulkWriter in the Node SDK sends two writes to the same document as separate, concurrent batchWrite calls, so the application sees exactly this pattern. In our case it combined with a BulkWriter retry bug (filed separately: googleapis/google-cloud-node#9444) to make close() hang forever, which froze our e2e suite on 1.22.0. Possibly related to #8105 and #6758, but those involve transactions, while here both writes are non-transactional and the behaviour is new in v1.22.0.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions