diff --git a/.env.example b/.env.example index 377ae3d..89be357 100644 --- a/.env.example +++ b/.env.example @@ -45,7 +45,8 @@ PREVIEW_ROW_LIMIT=25 FLATTEN_MAX_DEPTH=10 # XLSX-only export budget, in cells (rows x columns). Derived from the -# measurement in docs/export-budget-v1.2.md. CSV/TSV are streamed and uncapped. +# measurement in docs/plan-status-v1.2.md, Appendix A. CSV/TSV are streamed and +# uncapped. # 0 disables the guard. MAX_EXPORT_CELLS=250000 diff --git a/CHANGELOG.md b/CHANGELOG.md index 64ec424..693b9d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,9 +7,10 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ## [1.2.0] - 2026-08-21 — "Hardening & Performance" -Implements `docs/roadmap-v1.2.md`, closing every finding in -`docs/security-review-v1.2.md` (F1–F17) and `docs/performance-review-v1.2.md` -(P1–P13). +Implements the v1.2 roadmap, closing every security finding (F1–F17) and +performance finding (P1–P13). Those four planning documents were later +consolidated into `docs/plan-status-v1.2.md`; their full text remains in git +history at `065883f`. No response key changed name, type or meaning; `/process` only gained keys. @@ -68,7 +69,7 @@ No response key changed name, type or meaning; `/process` only gained keys. - **Diskless, memory-bounded exports (P3).** CSV/TSV are generator-streamed and uncapped. XLSX keeps a normal-mode workbook (no OS temp files) plus `MAX_EXPORT_CELLS`, measured rather than guessed — see - `docs/export-budget-v1.2.md`. + `docs/plan-status-v1.2.md`, Appendix A. - **Lazy tree picker (P4)** and **client render caps (P5)**: children build on first toggle; nested objects stop at 20 keys, primitive arrays at 20 items, strings at 500 characters. diff --git a/CLAUDE.md b/CLAUDE.md index 450ade2..8b3a583 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,11 +36,9 @@ json-table-tool/ │ ├── test_render_caps.mjs │ └── test_features.mjs ├── docs/ -│ ├── code-health-final.md -│ ├── security-review-v1.2.md -│ ├── performance-review-v1.2.md -│ ├── roadmap-v1.2.md -│ └── export-budget-v1.2.md # How MAX_EXPORT_CELLS was measured +│ └── plan-status-v1.2.md # The planning doc: what shipped, what is pending, +│ # the export-budget measurement (App. A) and the +│ # perf budget + RSS protocol (App. B) ├── .github/workflows/ci.yml # lint, format, tests, JS assertions, pip-audit ├── pyproject.toml # ruff + pytest configuration ├── requirements.txt # Runtime dependencies (exact-pinned) @@ -317,7 +315,7 @@ convention is a dedicated bump commit, not a bump ridden along with a feature. ### Re-deriving the Excel export budget `MAX_EXPORT_CELLS` is a measured number, not a chosen one. Follow the method in -`docs/export-budget-v1.2.md` (fresh process per measured request, `ru_maxrss` -converted for the platform, delta across two runs), re-fit against the narrowest -aspect ratio you care about, and re-run a confirming point at the value you -intend to ship. Record the new data in that file. +`docs/plan-status-v1.2.md` Appendix A (fresh process per measured request, +`ru_maxrss` converted for the platform, delta across two runs), re-fit against +the narrowest aspect ratio you care about, and re-run a confirming point at the +value you intend to ship. Record the new data in that appendix. diff --git a/MEMORY.md b/MEMORY.md index 8e1154c..83c6607 100644 --- a/MEMORY.md +++ b/MEMORY.md @@ -154,7 +154,7 @@ Keep entries short — if it grows past ~10 lines, it probably belongs in `READM ### 2026-08-21 — The XLSX export budget is measured, in cells, and on by default (area: performance) **What:** `MAX_EXPORT_CELLS` (default 250,000) caps Excel exports only. CSV/TSV are generator-streamed and stay uncapped. -**Why:** openpyxl memory tracks `rows × columns`, not rows — at equal cell counts a narrow, tall sheet costs *more* (84.3 MiB at 50k×3 vs 73.6 at 15k×10), so a row limit says almost nothing about the footprint. The default comes from the measurement in `docs/export-budget-v1.2.md`, not from feel. An unlimited default would leave a High finding unmitigated; uncapped CSV/TSV is what keeps the export contract as wide as the input contract. +**Why:** openpyxl memory tracks `rows × columns`, not rows — at equal cell counts a narrow, tall sheet costs *more* (84.3 MiB at 50k×3 vs 73.6 at 15k×10), so a row limit says almost nothing about the footprint. The default comes from the measurement in `docs/plan-status-v1.2.md` (Appendix A), not from feel. An unlimited default would leave a High finding unmitigated; uncapped CSV/TSV is what keeps the export contract as wide as the input contract. **How to apply:** Re-derive the number whenever the measurement is re-run — do not round it to something tidy. Never truncate an oversized export: `/process` advertises `total_cells`/`max_export_cells` so the client greys Excel out beforehand, and `/export-xlsx` returns 400 for direct callers. And keep exports diskless: openpyxl's `write_only` mode writes worksheet parts to OS temp files, and `SpooledTemporaryFile` is either pointless (its default `max_size=0` never rolls over) or disk-backed. **Owner / source:** performance review P3, decision D6. diff --git a/README.md b/README.md index eccfb76..6a9ca1d 100644 --- a/README.md +++ b/README.md @@ -559,7 +559,7 @@ need no server round trip and nothing is persisted — responses over `GZIP_MIN_SIZE` are gzipped to keep that affordable. The preview rows are a truncated *copy*, so exports keep full fidelity. Excel exports are bounded by `MAX_EXPORT_CELLS`, measured rather than guessed -(`docs/export-budget-v1.2.md`); CSV and TSV stream and stay uncapped. The JSON +(`docs/plan-status-v1.2.md`, Appendix A); CSV and TSV stream and stay uncapped. The JSON tree picker builds children only when a node is opened. --- diff --git a/config.py b/config.py index 32d3562..30002d7 100644 --- a/config.py +++ b/config.py @@ -5,8 +5,8 @@ # Publicly known, and therefore only ever acceptable outside production. DEV_SECRET_KEY = 'dev-secret-key-change-in-production' -# See MAX_EXPORT_CELLS below and docs/export-budget-v1.2.md for how this number -# was measured. +# See MAX_EXPORT_CELLS below and docs/plan-status-v1.2.md (Appendix A) for how +# this number was measured. DEFAULT_MAX_EXPORT_CELLS = 250_000 @@ -164,9 +164,9 @@ class Config: # 500 columns have wildly different footprints at the same row count. # # Enabled by default: an unlimited default would leave P3 (High) unmitigated. - # The value is derived from the Performance Review section 4 measurement (see - # docs/export-budget-v1.2.md), not chosen by feel -- re-derive it whenever that - # measurement is re-run. 0 disables the guard for operators who knowingly opt + # The value is derived from the measurement in docs/plan-status-v1.2.md + # (Appendix A), not chosen by feel -- re-derive it whenever that measurement + # is re-run. 0 disables the guard for operators who knowingly opt # out. CSV/TSV stay uncapped and streamed, so every dataset /process accepts # remains exportable by some route. MAX_EXPORT_CELLS = env_int('MAX_EXPORT_CELLS', DEFAULT_MAX_EXPORT_CELLS) diff --git a/docs/code-health-final.md b/docs/code-health-final.md deleted file mode 100644 index 6c854d0..0000000 --- a/docs/code-health-final.md +++ /dev/null @@ -1,695 +0,0 @@ -# Code-Health Review — JSON Table Converter - -**Date:** 2026-05-02 -**Reviewer:** Automated senior code-health review -**Repository:** https://github.com/Badry-Kudu/json-table-tool -**Revision reviewed:** `main` branch (HEAD at time of review) - ---- - -## 1. Initial Score Estimate - -| Category | Score | Weight | -|---|---|---| -| Maintainability | 7/10 | 10% | -| Test coverage & quality | 8/10 | 15% | -| Complexity | 7/10 | 10% | -| Duplication | 8/10 | 5% | -| Architecture consistency | 8/10 | 10% | -| Dependency / security risk | 3/10 | 15% | -| Documentation | 7/10 | 10% | -| CI/CD reliability | 0/10 | 15% | -| Input validation & error handling | 9/10 | 10% | -| Usability & developer experience | 7/10 | 10% | - -**Overall weighted score: ~68 / 100** - -**Target score: 85+ / 100** - ---- - -## 2. Project Overview (Verified) - -**Stack:** Python 3.11+, Flask 3.0.0 (app-factory pattern), Flask-WTF (CSRF), Flask-Limiter, gunicorn, requests, openpyxl. Frontend: Vanilla HTML / CSS / JS (no framework, no build step). Tests: pytest. - -**Purpose:** Lightweight stateless web tool that converts JSON and JSONL data into viewable HTML tables, with CSV, TSV, and Excel export. Supports file upload, paste, and external API fetch with auth. - -**Directory layout:** -``` -app.py Flask app factory -config.py All settings via env vars -extensions.py Shared CSRF + limiter instances -security.py SSRF validation + security headers -helpers.py Data parsing, flattening, path selection -routes.py Blueprint with /, /health, /process, /export-csv, /export-xlsx -static/ css/style.css js/app.js -templates/ index.html (structure only, no inline scripts/styles) -tests/ conftest.py, test_helpers.py, test_routes.py, test_security.py -render.yaml Render.com deployment blueprint -requirements.txt -``` - ---- - -## 3. Commands Run & Results - -### Install -```bash -pip install -r requirements.txt -# → Success. All 7 direct dependencies installed. -``` - -### Tests -```bash -python -m pytest tests/ -v -# → 82 passed, 2 warnings (openpyxl DeprecationWarning from library internals), 0 failures -``` - -### Coverage -```bash -coverage run -m pytest tests/ && coverage report -m -``` - -| File | Stmts | Miss | Cover | -|---|---|---|---| -| app.py | 16 | 1 | 94% | -| config.py | 14 | 0 | 100% | -| extensions.py | 5 | 0 | 100% | -| helpers.py | 78 | 1 | 99% | -| routes.py | 181 | 33 | 82% | -| security.py | 33 | 3 | 91% | -| tests/ | 446 | 0 | 100% | -| **TOTAL** | **773** | **38** | **95%** | - -### Dependency Audit -```bash -pip-audit -r requirements.txt -``` -**Found 7 known vulnerabilities in 4 packages:** - -| Package | Current | CVE | Fix Version | Severity | -|---|---|---|---|---| -| Flask | 3.0.0 | CVE-2026-27205 | 3.1.3 | — | -| requests | 2.31.0 | CVE-2024-35195 | 2.32.0 | Medium | -| requests | 2.31.0 | CVE-2024-47081 | 2.32.4 | Medium | -| requests | 2.31.0 | CVE-2026-25645 | 2.33.0 | — | -| gunicorn | 21.2.0 | CVE-2024-1135 | 22.0.0 | High | -| gunicorn | 21.2.0 | CVE-2024-6827 | 22.0.0 | — | -| pytest | 7.4.4 | CVE-2025-71176 | 9.0.3 | — | - -### Linting / Formatting -``` -No linting or formatting tools are configured for this project. -``` -Run (locally or in CI): -```bash -pip install ruff -ruff check . -ruff format --check . -``` - -### Type checking -``` -No type annotations present. No mypy or pyright configured. -``` - -### CI/CD Workflows -``` -No .github/workflows/ directory found. Zero CI/CD configuration. -``` - ---- - -## 4. Category-by-Category Findings - -### 4.1 Maintainability — 7/10 - -**Strengths:** -- Clean app-factory pattern (`create_app()`) with proper extension initialization. -- Well-separated modules: `config.py`, `extensions.py`, `security.py`, `helpers.py`, `routes.py`. -- Consistent docstrings on all public functions. -- Naming conventions are clear and consistent throughout. - -**Issues:** -- `process_json()` in `routes.py` is ~150 lines with three nested branches (file/paste/api), five auth sub-branches, and a post-parse block — cyclomatic complexity ≈ 15+. -- No type annotations anywhere. With Flask's dynamic nature, type hints would catch many errors at development time. -- No linter or formatter configured; code style is consistent now but will drift as contributors add code. -- `openpyxl` is imported lazily inside `export_xlsx()` — inconsistent with all other imports at module top, and hides ImportError until runtime. - -### 4.2 Test Coverage & Quality — 8/10 - -**Strengths:** -- **95% overall coverage** — exceptional for a Flask project. -- 82 tests across three well-organized files (`test_helpers`, `test_routes`, `test_security`). -- All auth paths tested via mock. SSRF blocking well-covered (16 URL validation tests). -- Edge cases covered: UTF-8 decoding failure, max-size exceeded, JSONL invalid line, path selection modal flow. - -**Issues:** -- `routes.py` is 82% covered; uncovered lines include auth header/key/bearer/query-param branches (lines 92–109) and the `ValueError` path in `parse_jsonl` when called from API fetch. -- No edge-case input tests for helpers: `null` values in JSON objects, Unicode / emoji column names, boolean-only arrays, numbers as top-level JSON, empty-object arrays `[{}, {}]`, extremely large number of columns. -- No test confirming the `PREVIEW_ROW_LIMIT` server config is reflected in the JS badge (hardcoded "25" in `app.js` line 219 and 220 is inconsistent with server config). -- No test for file upload with no `json_file` field (line 50 branch is untested). - -### 4.3 Complexity — 7/10 - -**Strengths:** -- `helpers.py` functions are small, single-purpose, and easy to reason about. -- `security.py` is clear and minimal. - -**Issues:** -- `process_json()` has high cyclomatic complexity and should be refactored: extract `_parse_input(request, data_format)` → returns `(json_data, error_response)` and `_apply_path_selection(json_data, json_path)` → returns `(table_data, error_response)`. -- `extract_table_data()` falls through multiple isinstance checks with no early return at each branch; a match/case or helper would improve clarity (Python 3.10+). -- `renderTable` / `renderTableDOM` split in `app.js` is unnecessary — `renderTable` just stores refs then calls `renderTableDOM`, creating indirection without benefit. - -### 4.4 Duplication — 8/10 - -**Strengths:** -- `escapeHtml()` is defined once and reused across all JS rendering functions — no XSS risk from duplication. -- `get_all_columns()` is reused for both preview and CSV columns. - -**Issues:** -- `export_csv` and `export_xlsx` both implement identical `dict/list → JSON string` serialization for cell values: - ```python - if isinstance(v, (dict, list)): - clean_row[k] = json.dumps(v) - ``` - This logic should live in a shared helper (e.g., `serialize_cell_value(v)`). -- `downloadDelimited` in `app.js` and `export_csv` route in `routes.py` both implement CSV serialization independently. This is intentional (client-side fallback + server fallback) but should be documented as such. - -### 4.5 Architecture Consistency — 8/10 - -**Strengths:** -- Blueprint pattern used correctly. Extensions initialized with `init_app()` pattern. -- Security headers applied globally via `after_request`. No inline styles/scripts (CSP-safe). -- Config exclusively from environment variables with documented defaults. - -**Issues:** -- `app = create_app()` at module level in `app.py` (line 28) is fine for `python app.py` dev use, but means importing `app` in tests or tools always creates an app instance with production config. Tests avoid this by calling `create_app()` directly, but it's a subtle trap. -- `RATE_LIMIT_PROCESS` / `RATE_LIMIT_EXPORT` are stored as custom config keys rather than using Flask-Limiter's standard `RATELIMIT_*` namespace, requiring the lambda wrapper on every route decorator. -- No `pyproject.toml` or `setup.cfg` — project metadata (name, version, Python constraint) exists only in `config.py` and `render.yaml`, not in a standard packaging file. - -### 4.6 Dependency / Security Risk — 3/10 - -**Strengths:** -- All direct dependencies are pinned to exact versions — no floating ranges. -- Minimal dependency footprint: 7 direct deps for a full web app is lean. -- No unused dependencies found. - -**Issues:** -- **7 CVEs across 4 packages** — all have published fixes: - - `gunicorn 21.2.0` → CVE-2024-1135 (HTTP Request Smuggling, **High**) + CVE-2024-6827. Fix: `gunicorn==22.0.0`. - - `requests 2.31.0` → 3 CVEs. Fix: `requests==2.33.0` (supersedes all three). - - `Flask 3.0.0` → CVE-2026-27205. Fix: `Flask==3.1.3`. - - `pytest 7.4.4` → CVE-2025-71176 (test-only, lower risk). Fix: `pytest==9.0.3`. -- No `requirements-dev.txt` / `requirements-test.txt` separation — `pytest` (test-only) is mixed with production dependencies, meaning it gets installed in production. -- No transitive dependency pinning (`pip-compile` or lock file). Transitive deps are resolved fresh on every `pip install`, so builds are not fully reproducible. -- `openpyxl==3.1.2` triggers `DeprecationWarning: datetime.datetime.utcnow()` (library-internal issue, tracked upstream). Fix: upgrade to `openpyxl==3.1.5`. - -### 4.7 Documentation — 7/10 - -**Strengths:** -- `README.md` is thorough: quick start, full deployment section (Render, gunicorn+systemd+nginx, Docker, Railway, Fly.io), usage guide, security notes, troubleshooting, environment variable table. -- `CLAUDE.md` provides excellent internal contributor guidance. -- All Python functions have docstrings. - -**Issues:** -- No `CONTRIBUTING.md` or `DEVELOPMENT.md` — new contributors must infer linting/testing conventions from `CLAUDE.md`. -- No `CHANGELOG.md` — no history of changes or versioning rationale. -- No `docs/` directory or architecture decision records. -- README does not mention linting, type checking, or coverage — the "Development" section only shows `python app.py` and `pytest`. -- README badge for "License" points to no `LICENSE` file (the file doesn't exist in the repo). -- Missing: JSON format edge-case examples (null values, booleans, Unicode, deeply nested, large files). -- `APP_VERSION = '1.1.0'` in `config.py` is not linked to any tag or release — version can drift silently. - -### 4.8 CI/CD Reliability — 0/10 - -**Critical gap.** There is no `.github/workflows/` directory. No CI pipeline exists. - -This means: -- No automated test runs on pull requests or pushes to `main`. -- Dependency vulnerabilities are never automatically flagged. -- Linting and formatting checks never run automatically. -- Regressions can be merged without detection. -- The `render.yaml` auto-deploys on every push — with no CI gate, broken code deploys directly to production. - -**Required:** A GitHub Actions workflow running on `push` and `pull_request` to `main` that: -1. Installs dependencies -2. Runs `pytest` with coverage -3. Runs `ruff check` and `ruff format --check` -4. Runs `pip-audit` - -### 4.9 Input Validation & Error Handling — 9/10 - -**Strengths:** -- CSRF protection on all POST routes via Flask-WTF. -- SSRF protection with DNS validation blocking private/loopback/link-local/multicast IPs. -- Rate limiting on all routes (configurable). -- `MAX_CONTENT_LENGTH` enforced (10MB default). -- API response size capped + streamed (no full-load-then-check). -- `allow_redirects=False` on API fetch. -- UTF-8 decoding error caught and returns 400. -- `UnicodeDecodeError`, `JSONDecodeError`, `ValueError`, `Timeout`, `RequestException` all handled with appropriate HTTP status codes. -- All `RequestException` detail is swallowed and a generic message returned (no internal host leakage). -- `escapeHtml()` used consistently in all JS rendering paths (no XSS risk from JSON data). - -**Issues:** -- No JSON parsing depth limit — a specially crafted JSON with 10,000 levels of nesting could exhaust the Python call stack before `flatten_for_csv`'s `max_depth` guard is reached (Python's default recursion limit is ~1000; `json.loads` itself is C-level and handles it, but `extract_table_data` is recursive Python with no depth guard). -- No file extension validation on upload (any file extension accepted; only content parsing catches non-JSON). -- No explicit `Content-Type` validation on file upload (`accept=".json,.jsonl"` is client-side only). -- The JS `formatValue()` function does not guard against extremely large arrays/objects in a cell — rendering a cell with 50,000-item nested array would freeze the browser tab. - -### 4.10 Usability & Developer Experience — 7/10 - -**Strengths:** -- Drag-and-drop file upload, dark/light theme with `localStorage` persistence, format toggle (JSON/JSONL), multi-format export dropdown. -- Keyboard-accessible tabs (they are `