Allow low frequency waveform data into parquet - #83
Merged
Conversation
carefully, and to have a HF and LF version. (tests with LF currently fail because it's not implemented yet)
Define the data that needs to be passed for output.
otherwise parquet conversion is going to be too complex
filtering/searching
Collaborator
Author
|
CI failure is because PIXL has moved on. This is being fixed in #81 |
jeremyestein
marked this pull request as ready for review
August 18, 2026 10:00
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #83 +/- ##
==========================================
- Coverage 23.07% 20.39% -2.69%
==========================================
Files 12 13 +1
Lines 429 510 +81
==========================================
+ Hits 99 104 +5
- Misses 330 406 +76 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
thompson318
reviewed
Aug 19, 2026
thompson318
left a comment
thompson318
left a comment
Collaborator
There was a problem hiding this comment.
I like the new structure for json to to message classes. Logical and easy to read. Tests look good and fairly comprehensive.
One comment regarding what happens on a value error from write frame to think about.
tests, not a real drop in coverage anyway)
Closed
2 tasks
thompson318
approved these changes
Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an extra column to the CSV + parquet files:
valuesbecomesnumeric_valuesandstring_values.HF -> always
numeric_values.LF -> can be either (but not both): categoricals are always mapped to their string representation; numericals are left as such.
As things stand, a variable with a mix of units (eg. 1408) will create one file per unit.
For LF, always store data as an array of length 1 per row. We have the flexibility to group together values in arrays in future if we decide that. It's an array because parquet is strongly typed so data columns have to be arrays (didn't want to add more cols for scalar values).
I kept the column names/types the same between LF and HF to simplify the parquet conversion and pseudonymisation code. Although those that don't apply (channel ID, sampling rate) can be null.