feat(poplar): raw-data quality and coverage report

[ceyhunolcan]

Adds poplar/raw/quality.py: a cross-stream raw-data quality and coverage report for a study folder sitting beside poplar.raw.doc (which describes streams) and poplar.raw.readers (which reads them), as the validation piece.

Per participant × stream it reports presence vs. expected-for-OS, file count and size, row count, first/last observation, days spanned and coverage, largest gap, duplicate and out-of-order rows, unreadable files, and a schema check against the documented header. run() writes one long format data_quality.csv, one row per (backend_id, stream).

Design

• Stream knowledge isn't hard-coded: reads STREAMS (per-OS availability) and HEADERS (expected columns) from poplar.raw.doc, so it tracks whatever Beiwe adds. Schema check is three-state to match HEADERS: validate a column list, undocumented for "To do", na for null.
• Device OS is read from the identifiers stream (headers differ by OS, e.g. power_state).
• run(study_folder, output_folder, tz_str, frequency, time_start, time_end, users) mirrors the other trees; frequency sets the coverage-bin resolution (HOURLY_AND_DAILY treated as daily).
• numpy/pandas only; deliberately does not reuse clean_dataframe (dedups in place, which would erase the duplicates I need to count) or read_data (GPS/accel-specific, returns filenames for accel).

Notes / open questions

• Coverage-via-frequency, or would you prefer fixed hour + day columns?
• Placement poplar/raw/quality.py reasonable, or elsewhere?

Tests: tests/poplar/test_quality.py defect-injection (clean stream; a stream with schema mismatch + gap + duplicate + out-of-order + unreadable file; an expected-but-absent stream) + end-to-end. ruff/flake8/mypy clean; full suite green.
Docs: docs/source/quality.md (+ toctree).

[biblicabeebli]

One thing that we have now is compressed data files (.csv.zst as the file type). pandas.read_csv works transparently with such files when it is handed them provided the zstandard library support is present. Can you confirm that that doesn't throw anything off here?

[biblicabeebli]

this issue probably interacts, but should not be considered a blocker:

• Unify file (csv) read-in #332

(I am making new issues for cleanup related tasks I would like to get to)

[ceyhunolcan]

Good catch you're right that it interacts. _device_os and the per-stream file listing filtered discovery to .endswith(".csv"), so a .csv.zst stream would be skipped entirely (reported absent / zero files), even though pandas.read_csv would decompress it fine once handed the path.

Fixed to include both extensions (endswith((".csv", ".csv.zst"))), matching the sycamore #320 approach. Added a regression test that writes a real .csv.zst accelerometer stream and asserts the report discovers and reads it (24 rows, present) it fails without the fix. zstandard is already a dependency, so no requirement change. Full poplar suite green.

Also noted #332 agreed it's not a blocker here; happy to align this with a unified reader when that lands.

[biblicabeebli]

(the target branch has been merged into develop, retargeting this pull to develop)