Repository navigation
fix: stop msnodesqlv8 streams losing rows under back pressure - #1919
Merged
dhensby merged 2 commits intoOct 14, 2026
Merged
Conversation
dhensby
force-pushed
the
fix/msnodesqlv8-stream-back-pressure
branch
from
October 13, 2026 17:47
1293db6 to
9de5010
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Cancellation can emit queued rows after a listener resumes reentrantly, and the integration query has nondeterministic ordering.
3 open findings
What changed in this PR
Fixes row loss in msnodesqlv8 streams under back pressure and adjusts paused-stream cancellation behavior.
Changes:
- Queues stream events while paused and pauses the driver at safe batch boundaries.
- Defers cancellation until the driver can safely stop.
- Adds unit, integration, and API documentation coverage.
| File | Description |
|---|---|
lib/msnodesqlv8/request.js |
Implements buffered events and safe pause/cancel handling. |
README.md |
Documents paused-stream and cancellation behavior. |
test/common/unit.js |
Adds driver-version event simulations. |
test/common/tests.js |
Adds partial-batch back-pressure coverage. |
test/msnodesqlv8/msnodesqlv8.js |
Registers the integration test. |
test/tedious/tedious.js |
Registers cross-driver regression coverage. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
dhensby
force-pushed
the
fix/msnodesqlv8-stream-back-pressure
branch
from
October 13, 2026 19:28
618555e to
471163f
Compare
msnodesqlv8 fetches a result set's rows 50 at a time. Paused part way through a result set's last, short batch, it moves on without the rest of that batch. On the last result set it ends the query, and the rows still to come are never emitted. On an earlier one it starts the next result set, and on resume emits the rest of the batch under the next result set's columns; from 5.0 it then skips the next result set as well. 5.6 waits for the rest of a batch, but when paused on the only row of a result set's last batch it still skips the next result set. Back pressure from pipe() and toReadableStream() pauses a stream on almost every row, so a piped result set whose row count is not a multiple of 50 silently lost up to 49 rows. The driver is now paused only once it has handed over a whole batch, or before it has read anything. Rows that arrive in the meantime are held, along with anything else the stream emits after them, and emitted in order once the stream is resumed, and the stream completes once it has emitted them. Nothing is emitted while a stream is paused; before, a row that arrived while it was paused was dropped. msnodesqlv8 cancels a query cleanly only once it has stopped reading: paused or cancelled part way through a fetch, or as it moves to the next result set or ends the query, it can free the statement twice and break the queue of the connection. So a paused stream whose driver is still reading is cancelled when the driver next stops, at the end of a batch or the start of a result set, and a query that ends first, or times out, completes as cancelled. A paused stream that is cancelled drops the rows still waiting and reports ECANCEL, also when the query had already ended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dhensby
force-pushed
the
fix/msnodesqlv8-stream-back-pressure
branch
from
October 13, 2026 20:22
bc33026 to
5e7322f
Compare
|
🎉 This PR is included in version 12.7.7 🎉 The release is available on: Your semantic-release bot 📦🚀 |
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What this does:
Piping a msnodesqlv8 stream (
request.pipe()/toReadableStream()) under back pressure silently loses rows. msnodesqlv8 fetches a result set's rows 50 at a time. Paused part way through a result set's last, short batch, it moves on without the rest of that batch. Back pressure pauses a stream on almost every row, so any result set whose row count isn't a multiple of 50 is affected. The existingquery with pipe and back pressuretest usestop 2000, so it never hit this.Confirmed on Windows CI (SQL Server 2022, a writer taking one row every 10 ms) in run 38378595926:
top 2010top 75; top 75top 51; top 51None of these raise an error, and the connection is usable afterwards. tedious delivers every row in every case.
The fix is in the msnodesqlv8 adapter. v2–v4 and 5.2.1 (the last v5 release with a Node 20 prebuild) won't get upstream fixes, and 5.6.0 is still affected.
DRIVER_ROW_BATCH_SIZEexplains why.Cancel had to change with it. The driver now keeps reading for a while after a stream is paused, so it can be in the middle of a fetch, moving to the next result set, or ending the query when the stream is cancelled. msnodesqlv8 doesn't cope with being paused or cancelled at those points. On 5.6 it frees the statement twice and drops an extra queue operation, which can be the next request's query. On 5.0–5.5 the stream gets a spurious
HY010error. So:freeand nodone.cancel()behaves as it does on master.ECANCEL, also when the query had already ended.Behaviour change to note:
pause(), msnodesqlv8 keeps reading to the end of a full batch, and those rows are held untilresume().Known limit (driver-side, same on master): on 5.6, a stream that is paused before its query starts and then cancelled while the statement is still executing leaves msnodesqlv8's reader paused. Its work queue stays blocked unless something resumes it. Master behaves the same unless the caller explicitly calls
resume()after cancelling.Testing:
msnodesqlv8 query events›stream back pressureplay a query the way each reader generation does (2.x–4.x, 5.0–5.5, 5.6). That includes refused and dropped cancels, the cancel token msnodesqlv8 holds before a statement exists, stuck fetches, timeouts, and queries that end on an error. Each fails against the adapter before its fix.query with pipe and back pressure across partial row batches, is wired for both drivers. It pipes result sets of 75, 51 and 120 rows into a writer that takes one row every 10 ms. The writer has to be timer-paced: asetImmediateone drains faster than the driver fetches, and misses the 5.6.0 case.user defined typesand the two bulk-load tests for spatial and hierarchyid columns from fix: bulk load created tables and Xml columns on tedious #1914. Those three need CLR, which SQL Edge doesn't have, and fail the same way on master.lib/*.js(2.7.0, 4.5.0, 5.2.1, 5.5.1, 5.6.0) over a fake native layer, and comparing against master's adapter. The last round found nothing new.Related issues:
ECANCELfrom'done',HY008dropped, nothing after'done') this keeps.Pre/Post merge checklist:
🤖 Generated with Claude Code