Classic scroll and ES 6 search_after turn a page parse failure into a silent end-of-stream

[fupelaqu]

Summary

On the classic scroll path (every client) and on the ES 6 search_after path, a page that Elasticsearch returned fine but that the client fails to convert into rows is turned into an empty page, and an empty page is the paging loop's end-of-stream signal. The stream then completes successfully with fewer rows than match — a silent truncation, the same defect class as #205 / #207 / #209 / #224 (silent wrong answer, no error to the caller).

The PIT + search_after path was hardened in #238 (extractHitsOnly now throws IllegalStateException("Failed to parse PIT page …"), which fails the page and therefore the stream — non-retriable). The two other paths still swallow.

Where (at feature/perf-238 HEAD)

Client	Helper	Behaviour on parseSingleSearchResponse failure	Loop consequence
es8 JavaClientApi.extractAllResults	classic scroll	logger.error(...); Seq.empty	scrollClassic: if (results.isEmpty …) None ⇒ stream completes
es9 JavaClientApi.extractAllResults	classic scroll	same	same
es7 RestHighLevelClientApi.extractAllResults	classic scroll	same	same
es6 RestHighLevelClientApi.extractAllResults	classic scroll	same	same
es6 RestHighLevelClientApi.extractHitsOnly	search_after (no PIT on ES 6)	logger.error(...); Seq.empty	unfoldAsync returns None ⇒ stream completes
es6 JestScrollApi.extractAllResults / extractHitsOnly	classic scroll / search_after	same	same

An ERROR line is logged, but the caller (JDBC, Arrow, REPL, scrollAs) receives a successful, short result.

When it bites

Any row-conversion failure mid-stream: an unexpected _source shape in one document (mapping drift, a field that became an object), a ClassCastException in a typed scrollAs[T], a script field of an unexpected type, a JSON tree the converter does not handle. Classic scroll is still the path for aggregations, for preferSearchAfter = false, and for ES < 7.12; ES 6 has no PIT at all.

Proposal

Mirror the #238 fix: a conversion failure fails the page (IllegalStateException — non-retriable, the response was valid; a retry would fail identically), so the stream fails loudly and the scroll/PIT context is still cleaned up by the existing termination hooks (clearScroll / closePit).

• es6/es7/es8/es9 extractAllResults: replace Seq.empty with a throw (the logger.error can stay, or become the exception message).
• es6 rest + jest extractHitsOnly: same.
• Keep the classic-scroll loop's "empty page = end" rule for a genuinely empty hits.hits — the end-of-stream decision should be taken on the RAW hit count (as Row extraction pages Elasticsearch sequentially and does not use shard parallelism #238 did for PIT slices: rawHits.isEmpty), never on the number of converted rows.

Acceptance

• A unit test per client path (no Docker: feed a hit whose _source the converter rejects) asserting the stream FAILS with a message naming the page/scroll rather than completing with a shorter Seq.
• Existing ScrollCompletenessSpec (classic scroll case preferSearchAfter = false) still green on real ES 6.8 / 7.17 / 8.18 / 9.0.
• A genuinely empty last page still ends the stream cleanly (no regression on the normal end-of-scroll).

Found while validating #238 (review pass #2, deferred as pre-existing and out of the PIT scope).