Keep cells holding two values as text instead of truncating to the first - #91
Open
nick-gorman wants to merge 3 commits into
Open
Keep cells holding two values as text instead of truncating to the first#91nick-gorman wants to merge 3 commits into
nick-gorman wants to merge 3 commits into
Conversation
_remove_series_notes_after_values reduced a cell to the value before its first parenthesis. For cells holding two values, each with its own note, that silently discarded every value but the first and then cast the result to a numeric type, so consumers saw a plausible-looking number with no signal that half the cell was gone. For example '930 (NSW works) 964 (QLD works)' became 930, and 'Storage properties' lost the 325 MW pump capacity from '250 (generation) 325 (pump)'. Add _where_multiple_values_with_notes to detect these cells and guard the substitution with it, leaving them as text. The column then fails to cast to a numeric type, which is how such cells behaved before the truncation was widened. A second value is only recognised where nothing but non-alphanumeric characters separates it from the first note's closing parenthesis, which distinguishes it from a footnote reference such as '400 (with VNI SIPS) - Note 8'. Across all five packaged workbooks this preserves 13 cells and still truncates the other 187. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015QBSw6PJqTrzzZHU6xg7cb
Matches the style of the other sanitisers, which pass their patterns directly to the pandas string method. The pattern is unchanged, so example_output is unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015QBSw6PJqTrzzZHU6xg7cb
The three rows were grouped under one comment implying none of them needed the predicate to fire. That is true of the two whose shape substitution 1 never matches, but 930 - NSW works 964 - QLD works is still cut down to 930 by substitution 2, dropping a value. Split it out and label it as the known gap it is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015QBSw6PJqTrzzZHU6xg7cb
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a check for multiple values in a cell before stripping out notes.