Skip to content

refactor: modernize ImageDT for Python 3 and open source development - #6

Merged
hanranCode merged 4 commits into
mainfrom
codex/python3-open-source-refactor
Oct 12, 2026
Merged

hanranCode merged 4 commits into
mainfrom
codex/python3-open-source-refactor

Conversation

@hanranCode

@hanranCode hanranCode commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

Modernize the Python 2 era utility package for Python 3.10+ while retaining historical public names and arguments through explicit lazy exports. Replace obsolete framework integrations with maintained backends and clear errors for unsupported conversions. The release candidate is 0.6.0rc1; no PyPI upload has occurred.

Fix image/XML pair renaming, classification split leakage and stale shards on regeneration, zero-based class labels, metric and batching behavior, numerical Caffe conversion and wrapper resource cleanup. Add behavioral, compatibility and real-backend tests.

Establish Ruff lint/format conventions, contributor/testing/security guidance, issue and PR templates, portable developer commands, and CI for Python 3.10/3.12/3.14 on Linux/macOS/Windows plus a real-backend integration lane. Four commits separate runtime/tests, development standards, migration/release documentation, and Windows test portability. The main base was created from the existing master commit as explicitly requested.

Test Coverage

225 tests pass with real optional backends in the final Python 3.12 environment. Coverage is 93.35% statements, 83.43% branches and 91.31% combined; the required combined gate is 88%. Tests cover legacy export/signature compatibility, file and image behavior, metrics, text generation, datasets, serialized Caffe inference, TensorFlow graph loading and LiteRT conversion.

The initial path audit rated 25 of 30 major paths meaningfully covered, two weakly covered and three gaps. Subsequent regressions strengthened the identified gaps; no exhaustive per-path rerating or complete backend-variant coverage is claimed. Serialized BatchNorm/Scale is checked against a NumPy numerical oracle; split regeneration checks empty train/validation cases and preservation of unrelated files.

Pre-Landing Review

Two independent native reviews found concrete issues, which were repaired and rechecked. Final reviews report no remaining actionable findings in the supported scope. External Claude review is unavailable: automatic approval rejected sending unpublished source to that third party without explicit authorization; it was not executed.

Exploratory QA

Bounded command-line probes passed paired-file rollback/collision checks, Caffe numerical conversion and explicit padding rejection, and repeated classification split generation. The exploratory ledger is inconclusive solely because an earlier artifact check was marked superseded by the final rebuild; final artifact verification passed separately. No browser surfaces exist. Multi-file shard publication is explicitly not atomic across crashes or concurrent writers.

Design Review

No frontend files changed; design review is not applicable.

Eval Results

No prompt-related files changed; model evaluation is not applicable.

Plan Completion

The requested migration, tests, optimization and open source development conventions are implemented. Historical entry points remain where feasible; removed framework capabilities fail explicitly and compatibility changes are documented. Publication to PyPI remains a separate release action. Hosted Windows checks identified two test portability defects, repaired with explicit UTF-8 source reads and native-path comparison. Final PR and push workflows both pass all ten jobs: nine Python/OS combinations and TensorFlow integration.

Verification Results

Full backend tests, lint, format, build, distribution metadata/source checks and installed-wheel tests passed locally. Python 3.10 and 3.14 core lanes previously passed 181 tests with 26 optional-backend skips; the installed wheel also passed 181 with 26 skips outside the checkout. Runtime backend repairs were verified in the full backend lane; the final Windows changes affect test portability only. Hosted CI is the cross-platform verification source. Final PR workflow and push workflow both completed successfully on the final e5ec409 commit.

Documentation

Status: updated. Final factual documentation corrections for supported BatchNorm/Scale fusion, classification TFRecord regeneration and fresh coverage evidence; no blockers.

Audited scope: second and last ship-owned audit of the 112 selected release files against baseline 78c9e671526422f28bd564cad9879ef95f12b454. Retained the prior full authored-document/template/source review and checked the final repaired implementation and numerical/ratio-rerun regression evidence. Actual preamble: SESSION_KIND: spawned.

Documentation health:

  • README.md: Current (installation, examples, lazy exports, supported backends and historical-review discoverability).
  • CONTRIBUTING.md: Current (Python conventions, setup, branch/release policy and CI support matrix).
  • TESTING.md: Current (checks, backend skips, coverage gate and artifact validation).
  • SECURITY.md: Current (intentional trusted-model boundary preserved).
  • CHANGELOG.md: Current, reviewed read-only; version remains 0.6.0rc1.
  • .github/ISSUE_TEMPLATE/bug_report.md: Current.
  • .github/ISSUE_TEMPLATE/feature_request.md: Current.
  • .github/pull_request_template.md: Current.
  • docs/migration.md: Updated (protobuf Scale gamma preservation in supported fusion; staging and replacement of owned six-digit classification shards, obsolete-shard cleanup, preservation of unrelated filenames and explicit non-atomic multi-file publication).
  • docs/release-readiness.md: Updated (93.4% statement coverage, 83.4% branch coverage, 91.3% combined; final 225-test backend rerun distinguished from earlier Python 3.10/3.14 core and installed-wheel verification).
  • docs/technical-reviews/2026-10-12-78c9e67/architecture-model.md: Current, preserved historical baseline.
  • docs/technical-reviews/2026-10-12-78c9e67/technical-review.md: Current, preserved historical review.
  • docs/technical-reviews/2026-10-12-78c9e67/gate-review.md: Current, preserved historical gate.
  • docs/technical-reviews/2026-10-12-78c9e67/review-manifest.json: Current, historical metadata.
  • docs/technical-reviews/2026-10-12-78c9e67/findings.json: Current, historical findings.

Documentation coverage and debt

Changed surface Reference How-to Tutorial Explanation
Python 3 installation, extras, contribution checks README/CONTRIBUTING/TESTING setup/check commands development setup migration/historical review
Lazy imports and graph lifecycle README/migration wrapper example limited migration
TFRecord regeneration and supported BN/Scale conversion migration/source limited missing migration
Metrics, batching and synthetic text migration/source missing missing migration
Paired renaming/PDF contracts README/migration/source limited missing migration
Candidate release workflow README/TESTING/release report build/upload commands limited contributor policy

No changed public capability has zero coverage. Nonblocking common debt: task examples/tutorials for TFRecord/conversion/LiteRT, metrics/batching/synthetic text, and paired renaming/PDF. /document-generate can fill those quadrants in a follow-up; consider a docs-debt label. No content generation was performed.

Diagram drift: none identified; historical diagram labels remain valid. Relative authored-doc links resolve. PyPI remains unpublished and remote CI is not represented as executed. Git/index, workflows, protected manifests, CHANGELOG/TODOS and historical findings were not modified.

Documentation freshness exception: after the second audit, hosted Windows CI required two test-only changes (explicit UTF-8 reads in legacy API tests and an operating-system-neutral UUID path assertion). Runtime code, package configuration, public interfaces and authored documentation are unchanged. The user explicitly accepted the limited risk that the documentation audit predates these two test repairs. A third documentation audit was not performed; retain the audit scope and evidence limitations above.

Test plan

  • Full suite with coverage: 225 passed; combined coverage 91.31%.
  • Ruff check and format check: passed across package, tests, scripts and examples.
  • Build wheel and source distribution; Twine check and release checker: passed.
  • Installed-wheel tests outside the checkout: 181 passed, 26 optional-backend skips.
  • Python 3.10 and 3.14 core tests: 181 passed, 26 optional-backend skips each.
  • Independent native reviews and final documentation audit completed.
  • Hosted CI: Python 3.10/3.12/3.14 across Linux/macOS/Windows and TensorFlow integration all passed on the final commit.

Generated with OpenAI Codex.

ranh-ms and others added 4 commits October 12, 2026 15:13
Preserve legacy public entry points through lazy exports, modernize optional backends and packaging, fix numerical and file/data integrity issues, and add behavioral and integration regressions.
Add Ruff conventions, portable contributor commands, CI coverage across Python versions and operating systems, and contribution, testing, security and issue guidance.
Record compatibility changes, backend limits, historical technical review and verified release candidate checks.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
@hanranCode
hanranCode merged commit c7be971 into main Oct 12, 2026
20 checks passed
@hanranCode
hanranCode deleted the codex/python3-open-source-refactor branch October 12, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant