Repository navigation
refactor: modernize ImageDT for Python 3 and open source development - #6
Merged
Merged
Conversation
Preserve legacy public entry points through lazy exports, modernize optional backends and packaging, fix numerical and file/data integrity issues, and add behavioral and integration regressions.
Add Ruff conventions, portable contributor commands, CI coverage across Python versions and operating systems, and contribution, testing, security and issue guidance.
Record compatibility changes, backend limits, historical technical review and verified release candidate checks. Co-Authored-By: OpenAI Codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Modernize the Python 2 era utility package for Python 3.10+ while retaining historical public names and arguments through explicit lazy exports. Replace obsolete framework integrations with maintained backends and clear errors for unsupported conversions. The release candidate is 0.6.0rc1; no PyPI upload has occurred.
Fix image/XML pair renaming, classification split leakage and stale shards on regeneration, zero-based class labels, metric and batching behavior, numerical Caffe conversion and wrapper resource cleanup. Add behavioral, compatibility and real-backend tests.
Establish Ruff lint/format conventions, contributor/testing/security guidance, issue and PR templates, portable developer commands, and CI for Python 3.10/3.12/3.14 on Linux/macOS/Windows plus a real-backend integration lane. Four commits separate runtime/tests, development standards, migration/release documentation, and Windows test portability. The main base was created from the existing master commit as explicitly requested.
Test Coverage
225 tests pass with real optional backends in the final Python 3.12 environment. Coverage is 93.35% statements, 83.43% branches and 91.31% combined; the required combined gate is 88%. Tests cover legacy export/signature compatibility, file and image behavior, metrics, text generation, datasets, serialized Caffe inference, TensorFlow graph loading and LiteRT conversion.
The initial path audit rated 25 of 30 major paths meaningfully covered, two weakly covered and three gaps. Subsequent regressions strengthened the identified gaps; no exhaustive per-path rerating or complete backend-variant coverage is claimed. Serialized BatchNorm/Scale is checked against a NumPy numerical oracle; split regeneration checks empty train/validation cases and preservation of unrelated files.
Pre-Landing Review
Two independent native reviews found concrete issues, which were repaired and rechecked. Final reviews report no remaining actionable findings in the supported scope. External Claude review is unavailable: automatic approval rejected sending unpublished source to that third party without explicit authorization; it was not executed.
Exploratory QA
Bounded command-line probes passed paired-file rollback/collision checks, Caffe numerical conversion and explicit padding rejection, and repeated classification split generation. The exploratory ledger is inconclusive solely because an earlier artifact check was marked superseded by the final rebuild; final artifact verification passed separately. No browser surfaces exist. Multi-file shard publication is explicitly not atomic across crashes or concurrent writers.
Design Review
No frontend files changed; design review is not applicable.
Eval Results
No prompt-related files changed; model evaluation is not applicable.
Plan Completion
The requested migration, tests, optimization and open source development conventions are implemented. Historical entry points remain where feasible; removed framework capabilities fail explicitly and compatibility changes are documented. Publication to PyPI remains a separate release action. Hosted Windows checks identified two test portability defects, repaired with explicit UTF-8 source reads and native-path comparison. Final PR and push workflows both pass all ten jobs: nine Python/OS combinations and TensorFlow integration.
Verification Results
Full backend tests, lint, format, build, distribution metadata/source checks and installed-wheel tests passed locally. Python 3.10 and 3.14 core lanes previously passed 181 tests with 26 optional-backend skips; the installed wheel also passed 181 with 26 skips outside the checkout. Runtime backend repairs were verified in the full backend lane; the final Windows changes affect test portability only. Hosted CI is the cross-platform verification source. Final PR workflow and push workflow both completed successfully on the final e5ec409 commit.
Documentation
Status: updated. Final factual documentation corrections for supported BatchNorm/Scale fusion, classification TFRecord regeneration and fresh coverage evidence; no blockers.
Audited scope: second and last ship-owned audit of the 112 selected release files against baseline
78c9e671526422f28bd564cad9879ef95f12b454. Retained the prior full authored-document/template/source review and checked the final repaired implementation and numerical/ratio-rerun regression evidence. Actual preamble:SESSION_KIND: spawned.Documentation health:
Documentation coverage and debt
No changed public capability has zero coverage. Nonblocking common debt: task examples/tutorials for TFRecord/conversion/LiteRT, metrics/batching/synthetic text, and paired renaming/PDF.
/document-generatecan fill those quadrants in a follow-up; consider adocs-debtlabel. No content generation was performed.Diagram drift: none identified; historical diagram labels remain valid. Relative authored-doc links resolve. PyPI remains unpublished and remote CI is not represented as executed. Git/index, workflows, protected manifests, CHANGELOG/TODOS and historical findings were not modified.
Documentation freshness exception: after the second audit, hosted Windows CI required two test-only changes (explicit UTF-8 reads in legacy API tests and an operating-system-neutral UUID path assertion). Runtime code, package configuration, public interfaces and authored documentation are unchanged. The user explicitly accepted the limited risk that the documentation audit predates these two test repairs. A third documentation audit was not performed; retain the audit scope and evidence limitations above.
Test plan
Generated with OpenAI Codex.