refactor: modernize ImageDT for Python 3 and open source development

.editorconfig

@@ -0,0 +1,18 @@
+root = true
+
+[*]
+charset = utf-8
+end_of_line = lf
+insert_final_newline = true
+trim_trailing_whitespace = true
+
+[*.py]
+indent_style = space
+indent_size = 4
+
+[*.{yml,yaml,json,toml}]
+indent_style = space
+indent_size = 2
+
+[Makefile]
+indent_style = tab

.gitattributes

@@ -0,0 +1,10 @@
+* text=auto
+*.py text eol=lf
+*.md text eol=lf
+*.toml text eol=lf
+*.yml text eol=lf
+*.yaml text eol=lf
+*.sh text eol=lf
+*.png binary
+*.jpg binary
+*.tflite binary

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,22 @@
+---
+name: Bug report
+about: Report a reproducible ImageDT behavior problem
+labels: bug
+---
+
+## Expected and actual behavior
+
+## Minimal reproduction
+
+Use synthetic input and temporary files. Do not include credentials or private data.
+
+## Environment
+
+- ImageDT version:
+- Python version:
+- Operating system:
+- Installed optional extras/backends and versions:
+
+## Error output
+
+Paste the relevant exception and traceback after removing sensitive paths/data.

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1 @@
+blank_issues_enabled: true

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,13 @@
+---
+name: Feature request
+about: Propose a concrete utility or compatibility improvement
+labels: enhancement
+---
+
+## Problem or use case
+
+## Proposed behavior
+
+## Existing workaround and compatibility constraints
+
+Include the desired public interface, input/output example and affected backend if relevant.

.github/pull_request_template.md

@@ -0,0 +1,17 @@
+## Problem and change
+
+What failed or was missing, and what happens after this change?
+
+## Compatibility
+
+Mention changed public names, argument names, output shapes/numerical semantics, dependencies or checkpoint behavior. Write “unchanged” when applicable.
+
+## Validation
+
+List commands run and observed pass/skip counts. Name backends/platforms that were not tested.
+
+- [ ] Ruff lint and format checks pass
+- [ ] Relevant behavior and failure/boundary tests pass
+- [ ] Legacy API fixture remains valid
+- [ ] User-facing docs and changelog updated where needed
+- [ ] No credentials, private data or built distributions committed

.github/workflows/test.yml

@@ -0,0 +1,41 @@
+name: Python package checks
+on:
+  push:
+    branches: [main, master, 'codex/**']
+  pull_request:
+    branches: [main]
+  workflow_dispatch:
+permissions:
+  contents: read
+concurrency:
+  group: tests-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+jobs:
+  core:
+    strategy:
+      fail-fast: false
+      matrix:
+        python: ['3.10', '3.12', '3.14']
+        os: [ubuntu-latest, macos-latest, windows-latest]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - run: python -m pip install ".[dev,plot,excel,pdf,caffe]"
+      - run: python -m ruff check imagedt tests scripts test
+      - run: python -m ruff format --check imagedt tests scripts test setup.py
+      - run: python -m pytest -q
+      - run: python -m build
+      - run: python -m twine check dist/*
+      - run: python scripts/check_release.py
+  tensorflow:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - run: python -m pip install ".[dev,plot,excel,pdf,tensorflow]"
+      - run: python -m pytest --cov=imagedt --cov-report=term-missing --cov-fail-under=88

.gitignore

@@ -108,4 +108,16 @@ venv.bak/
 *.tfrecords
 *.ipynb
 *.xml
-test/sources/
+test/sources/
+# Python 3 refactor build and test artifacts
+.venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+coverage.xml
+build/
+dist/
+*.egg-info/

CHANGELOG.md

@@ -0,0 +1,18 @@
+# Changelog
+
+## 0.6.0rc1 — 2026-10-12
+
+Python 3 migration candidate. Requires Python 3.10+ and retains historical public names, module paths and argument names.
+
+- Add modern package metadata, optional dependency groups, portable PDF rendering, CI and isolated release builds.
+- Make imports lazy and remove global GPU/eager-mode changes.
+- Fix image cleanup, padding, hashing, XML/CSV I/O, filename handling and paired rename failures.
+- Prevent annotation overwrites during MD5 pair renaming; preserve split-specific corrupt-image reports and use zero-based converter class IDs.
+- Correct BatchNorm protobuf normalization and terminal Caffe output aliases; reject numerically non-equivalent padding.
+- Add contributor/testing/security guidance, editor conventions, strict Ruff/format CI checks and PR/issue templates.
+- Correct detection/classification F1 and train/validation record leakage; accelerate ROC and Caffe DAG ordering.
+- Migrate TensorFlow graph execution, Caffe protobuf, LeNet and Lite inference to supported Python 3 backends.
+- Modernize synthetic text, manage pools/sessions/figures, and add self-contained regression, API compatibility and backend integration tests.
+- Retain removed CoreML/handwriting/legacy TOCO entrances with explicit unsupported errors.
+
+Read `docs/migration.md` for deliberate behavioral changes and checkpoint compatibility limits.

CONTRIBUTING.md

@@ -0,0 +1,47 @@
+# Contributing to ImageDT
+
+ImageDT is an Apache-2.0 Python utility package. Bug reports, regression fixtures and focused pull requests are welcome. Contributions use the existing project license; preserve copyright and license notices when modifying vendored code.
+
+## Set up a development environment
+
+Use Python 3.10 or newer. Create an isolated environment and install the project with the capabilities you will change:
+
+```bash
+python3 -m venv .venv
+# Linux/macOS: source .venv/bin/activate
+# Windows PowerShell: .venv\Scripts\Activate.ps1
+python -m pip install -e '.[dev,plot,excel,pdf,caffe]'
+# For model conversion / TensorFlow integration:
+python -m pip install -e '.[dev,plot,excel,pdf,tensorflow]'
+```
+
+Work on a feature branch based on `main`. Keep each commit focused and pair behavior changes with tests. Use commit subjects such as `fix: handle empty annotations` or `docs: clarify optional backends`. A PR should explain the problem, changed behavior, checks executed, and any compatibility limits. Do not commit generated distributions, virtual environments, credentials, personal datasets or machine-specific paths.
+
+## Python code conventions
+
+- Follow [PEP 8](https://peps.python.org/pep-0008/) and [PEP 257](https://peps.python.org/pep-0257/), with Ruff formatting at 88 columns and four-space indentation. The formatter's line length is a target; do not contort URLs, documentation or generated schema bytes to fit it.
+- Run Ruff's error, import, common-bug and Python modernization rules (`E4/E7/E9`, `F`, `I`, `B`, `UP`). Import groups follow standard library, third-party and project-local order. Explicit re-exports use `as` or `__all__`; avoid wildcard imports.
+- Keep historical public names, import paths and parameter names, including misspellings. New APIs use `snake_case` functions, `CapWords` classes and descriptive argument names. Add useful type annotations to new interfaces; do not require a heavyweight backend to resolve annotations for core tools.
+- Document public behavior, units/shape conventions, return values and meaningful errors. Prefer a short docstring over comments that merely restate a line. Do not silently change behavior to satisfy a style rule.
+- Prefer explicit validation and scoped exceptions. Preserve exception causes with `raise ... from error`. Do not use bare `except`, swallow failures, or rely on `assert` for public input validation.
+- Keep backend imports optional and expose a helpful missing-extra error. Importing `imagedt` must not change eager execution, GPU settings, plotting settings or launch workers.
+- Close sessions, figures, files and pools using context managers or explicit `close()`. File cleanup/rename tests must operate only in temporary fixtures. Avoid overwriting an existing destination without a documented contract.
+- Generated `caffe_pb2.py` is excluded from formatting/lint because it embeds schema bytes. TensorFlow test import ordering has one documented `E402` exception so the optional backend can skip before dependent imports. Add no broad suppressions merely to make CI pass.
+
+## Before submitting
+
+Run the commands in [TESTING.md](TESTING.md). Formatting and lint checks are enforced by CI; optional TensorFlow tests run in a separate integration job. A pass with skipped backends must be reported with its skip count rather than as full backend validation. Explain any changed metric values or checkpoint naming in [docs/migration.md](docs/migration.md). Update README/CHANGELOG when installation or user-visible behavior changes.
+
+On systems with GNU/BSD Make, `make PYTHON=python lint`, `make PYTHON=python test` and `make PYTHON=python build` wrap the same commands. Native Python commands remain the portable interface, including on Windows.
+
+## Compatibility and release policy
+
+The legacy API fixture is a baseline, not an authorization to remove methods. Tests must check behavior as well as name/argument preservation. Numerical bug fixes should state the old and new semantics. Keep optional capability and unsupported-operation boundaries explicit.
+
+Support is Python 3.10+. Core CI covers 3.10, 3.12 and 3.14 across Linux/macOS/Windows; TensorFlow integration runs on 3.12. Intermediate Python versions are within the support range but are not separate matrix lanes. If a new dependency requires a higher Python version, discuss that compatibility change before merging.
+
+Do not upload PyPI as part of an ordinary PR. The maintainer reviews the candidate, runs release validation, then explicitly publishes. Existing PyPI versions cannot be overwritten. `bash upload_pip.sh` builds/checks; `--upload` is the explicit publication step. Never include API tokens in a PR or committed config.
+
+## Community conduct
+
+Be respectful, specific and evidence-based in issues and reviews. Critique code and behavior rather than people. Harassment, discriminatory remarks and disclosure of personal information are not acceptable. Maintainers may edit or remove abusive content and limit participation. Report sensitive security issues through [SECURITY.md](SECURITY.md), not a public issue with exploit details.

MANIFEST.in

@@ -0,0 +1,6 @@
+include LICENSE README.md CHANGELOG.md CONTRIBUTING.md TESTING.md SECURITY.md Makefile .editorconfig .gitattributes pyproject.toml requirements.txt upload_pip.sh
+recursive-include tests *.py *.json
+recursive-include test *.py
+recursive-include scripts *.py
+recursive-include docs *.md *.json
+include .github/workflows/test.yml

Makefile

@@ -0,0 +1,24 @@
+PYTHON ?= python3
+
+.PHONY: format lint test coverage build check
+
+format:
+	$(PYTHON) -m ruff format imagedt tests scripts test setup.py
+	$(PYTHON) -m ruff check imagedt tests scripts test --fix
+
+lint:
+	$(PYTHON) -m ruff check imagedt tests scripts test
+	$(PYTHON) -m ruff format --check imagedt tests scripts test setup.py
+
+test:
+	$(PYTHON) -m pytest -q
+
+coverage:
+	$(PYTHON) -m pytest --cov=imagedt --cov-report=term-missing --cov-fail-under=88
+
+build:
+	$(PYTHON) -m build
+	$(PYTHON) -m twine check dist/*
+	$(PYTHON) scripts/check_release.py
+
+check: lint test

README.md

@@ -1,56 +1,89 @@
 # ImageDT
-<font size=5 > a python lib for neural networks, file and image processing etc. </font>
-
-Installation
-------------
-<font size=5 >1.  pip install
-    pip install imagedt <br>
-2. source install </font>
-
-    #1  pip install imagedt
-    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
-    #2  git clone https://github.com/hanranCode/ImageDT.git
-        cd ImageDT 
-        python setup.py install
-    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
-    #3  vi ~/.bashrc
-        export PYTHONPATH="/path_to_ImageDT:$PYTHONPATH"
-        import imagedt in your project
-
-
-Dependencies
-------------
-<font size=4 >1. tensorflow <br>
-2. coremltools <br>
-*Coremltools* has the following dependencies:
-
-- numpy (1.10.0+)
-- protobuf (3.1.0+)
-
-In addition, it has the following soft dependencies that are only needed when
-you are converting models of these formats:
-
-- Keras (1.2.2, 2.0.4+) with corresponding Tensorflow version
-- Xgboost (0.7+)
-- scikit-learn (0.17+)
-- libSVM  <br>
-</font><br>
-
-
-TODO
-------------
-- <font size=6 > <b>tools </b></font> <br>
-    - [x] Converte VOC data to tfrecords <br>
-    - [ ] Data interface  <br>
-- <font size=6 ><b> networks </b></font><br>
-
-    - <font size=5 >tensorflow</font>
-        + [x] lenet test
-        + [ ] ssd_mobilenet <br>
-
-    - <font size=5 >caffe</font>
-        - [ ] <font size=4 >? </font> <br>
-
-    - <font size=5 >pyroch</font>
-        - [ ] ?? <br>
-
+
+Python 3 utilities for image processing, files, annotations, metrics, synthetic text, TFRecords and model conversion. The original module paths, function names and public argument names are retained, including historical spellings such as `noise_padd`, `fillter_file_type`, `wirte_xml_file` and `gernerate_infos_dict`.
+
+## Install
+
+Requires Python 3.10 or newer. The candidate release is **0.6.0rc1**.
+
+```bash
+python -m pip install imagedt
+# Optional capabilities:
+python -m pip install 'imagedt[plot,excel,pdf]'
+python -m pip install 'imagedt[tensorflow]'  # TensorFlow 2 + LiteRT
+python -m pip install 'imagedt[lite]'        # LiteRT inference without TensorFlow
+python -m pip install 'imagedt[caffe]'       # Caffe protobuf/graph tooling
+```
+
+For this checkout, use `python -m pip install -e '.[dev,plot,excel,pdf,tensorflow]'`. The candidate is only available on PyPI after publication. OpenCV's headless package supports processing without requiring a desktop display; `cv_show` needs a GUI-enabled OpenCV installation. Install exactly one OpenCV distribution in an environment.
+
+Importing `imagedt` does not import TensorFlow, matplotlib, OpenCV or spreadsheet backends. Capabilities load when their exported names are accessed. An unavailable optional backend reports the extra to install.
+
+## Familiar interfaces
+
+```python
+import imagedt
+
+paths = imagedt.loop('./images', ['.jpg', '.png'])
+imagedt.write_csv([['name', 'count'], ['图片', 3]], 'counts.csv')
+boxes = imagedt.Anno_OP.parse_lab_xml('annotations/example.xml')
+```
+
+`FilesOp`, `Anno_OP`, `RecordWriter` and `plot_tools` remain singleton objects at their historical import paths. Traversal is deterministic. Text files use UTF-8; CSV readers also accept a UTF-8 BOM.
+
+```python
+from imagedt.tensorflow.tools.tfmodel_wrapper import TFmodel_Wrapper
+
+with TFmodel_Wrapper('model.pb', input_nodename='data',
+                     output_nodename='softmax,features') as model:
+    classifications = model.predict(images)
+    features = model.extract(images)
+```
+
+Graph wrappers own isolated sessions and support `close()` and context managers. Importing a backend does not disable eager execution or change GPU environment variables. TF1 graph APIs use `tf.compat.v1`; queue-based `Data_Provider` should be created within `tf.Graph().as_default()` and run with a Session and queue-runner lifecycle.
+
+## Behavioral corrections and migration limits
+
+See [the migration guide](docs/migration.md) for precise changes. Important corrections include class-aware detection matching, filename-aligned classification evaluation, no train/validation leakage, bounded dataset batching, binary XML writing, complete image cleanup, and collision-aware paired renaming.
+
+Maintained conversion support covers the bundled Caffe layer subset to frozen GraphDef, frozen GraphDef to float TFLite, and float32 single-image NHWC LiteRT classification. Arbitrary Caffe variants or model checkpoints are not guaranteed. LeNet uses Keras layers while retaining endpoint keys; **old TF-Slim variable/checkpoint names are not preserved**.
+
+These old capabilities retain callable entrances and raise explicit errors:
+
+- Caffe-to-CoreML: removed from modern coremltools; first migrate the model to a supported format.
+- Handwriting generation: the external TF1 pretrained model was never bundled. Computer text generation remains supported.
+- Legacy TOCO `GRAPHVIZ_DOT` and uint8 quantization without representative calibration data.
+
+PDF rendering uses portable PDFium wheels rather than ImageMagick/Ghostscript. Single-page output keeps the requested filename; multiple pages use `<stem>-0`, `<stem>-1`, etc.
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for Python conventions, branches and contribution workflow; [TESTING.md](TESTING.md) for checks and test lanes; [SECURITY.md](SECURITY.md) for private vulnerability reporting.
+
+## Develop and verify
+
+```bash
+python -m pip install -e '.[dev,plot,excel,pdf,tensorflow]'
+python -m ruff check imagedt tests scripts test
+python -m ruff format --check imagedt tests scripts test setup.py
+python -m pytest --cov=imagedt --cov-report=term-missing
+python -m build
+python -m twine check dist/*
+python scripts/benchmark_utils.py
+```
+
+Tests generate their own images, XML, fonts and tiny models in temporary directories. No personal `/data` paths, production datasets or GPU are needed. TensorFlow tests skip explicitly when the optional backend is absent. `tests/fixtures/legacy_api.json` captures public names and argument names from the original revision and guards compatibility. It does not claim numerical equivalence for intentionally corrected bugs or unchanged trained-checkpoint compatibility.
+
+CI defines Python 3.10/3.12/3.14 core jobs on Linux, macOS and Windows, plus a Python 3.12 TensorFlow/LiteRT integration job. Remote CI execution is separate from local verification.
+
+## Release
+
+`bash upload_pip.sh` builds into a fresh temporary directory and checks metadata. It does not upload. Publish only after reviewing the artifacts and configuring PyPI credentials:
+
+```bash
+PYTHON=python3 bash upload_pip.sh --upload
+```
+
+See [release validation](docs/release-readiness.md) and [changes](CHANGELOG.md). License: Apache-2.0.
+
+The [historical technical review](docs/technical-reviews/2026-10-12-78c9e67/technical-review.md) records the original revision baseline, architecture and migration decisions.