Repository navigation
feat(io): reject_duplicate_keys option for JSON and NDJSON reads - #428
Merged
Merged
Conversation
A repeated object key folds into the parsed Map keeping only its last value - the serde / pandas / Polars reading, and the default stays exactly that. The new option rejects the read instead: a small lexer over the raw text (ahead of @json.parse, only when the option is on, so the default read pays nothing) finds the first repeat, comparing keys decoded so "a" and "\u0061" count as the same key exactly as they do after parsing; nested objects are scanned too, and malformed input is left to the parser's own diagnostic. NDJSON scans per line so the error names the physical line; the lazy scan source rides the same option through its driver.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The one deferred item flagged "awaiting a decision" in the v0.6 plan. Polars, serde, and pandas all keep the last value for a repeated JSON object key, so that stays the default - unchanged, and pinned by a test. The new
reject_duplicate_keysoption offers the strict reading: a small lexer over the raw text finds the first repeat ahead of@json.parse(only when the option is on), comparing keys decoded so"a"and"\u0061"are the same key, scanning nested objects, and leaving malformed payloads to the parser's own diagnostic. NDJSON reports the physical line; the lazyscan_ndjsonsource rides the same option through its driver. Purely additive (a defaulted constructor field). 1446 tests green, coverage fully covered, all guards pass.