Skip to content

feat: add direct image stream preview and web dashboard for OpenAI image API - #23

Open
iceblue03 wants to merge 6 commits into
kodelyx:mainfrom
iceblue03:feat/openai-image-stream-preview
Open

iceblue03 wants to merge 6 commits into
kodelyx:mainfrom
iceblue03:feat/openai-image-stream-preview

Conversation

@iceblue03

@iceblue03 iceblue03 commented Oct 17, 2026 •

Copy link
Copy Markdown

Summary

Adds direct image streaming support (response_format: "image"), an interactive web preview dashboard (GET / and GET /preview), reachable HTTP URLs for generated files on both the native and the OpenAI endpoints, and one shared implementation of the input rules that Swagger UI and the CLI both trip over. It also carries PR #22 (Linux ARM64 binary detection), which is closed in favour of this branch — that is commit 9f03f7b, the first commit here.


Motivation & Problem Statement

  1. No Direct Image Preview in Swagger UI (/docs):
    Previously, invoking the OpenAI-compatible endpoint (/v1/images/generations) only returned JSON metadata with a media URL ({"data": [{"url": "..."}]}). Users testing from FastAPI's Swagger UI had to manually copy-paste the URL into a separate browser tab to view or save the generated image, as Swagger UI rendered raw JSON text without image preview.
  2. Missing Downloadable URLs on Native Endpoints:
    The native generation endpoint (/v1/generate/image or /api/v1/image) passed through Go engine outcomes returning only local filesystem absolute paths ("path": "/home/.../output/acct-xxx.jpg"). External clients connecting from other devices (e.g. mobile, LAN PCs) received server-local paths which were inaccessible remotely.
  3. Absence of a Lightweight In-Browser Testing Dashboard:
    Users who wish to test Flow image generation from remote machines or smartphones previously lacked a dedicated interactive web interface with real-time generation timers and instant one-click download buttons.
  4. The same input rule existed in four places, and one of them rewrote what was asked for:
    Swagger UI fills every untouched string field with the literal text string, so "model": "string" and "cookies": "string" arrive from /docs on every press of Execute. The cookie case reached the Go binary as --cookies string and came back as open string: no such file or directory (HTTP 502) — an auth failure, rather than an option nobody set. A third problem sat beside it: narwhal was silently rewritten to beluga, so a caller who asked for one name received another, with a success response.

Detailed Changes

flow_agent/api.py

  • ImageGenerationRequest / FlowImageRequest:
    • model now defaults to "beluga" (was "narwhal").
    • response_format is a documented Field: "url" is the OpenAI default and what an unmodified SDK expects (JSON carrying a fetchable link), "image" streams the raw bytes back, "b64_json" inlines them in JSON.
    • model is validated through engine.clean_model(), so a Swagger placeholder becomes the default while a model that was named is left alone. prompt stays required, exactly as main has it.
  • SIZE_TO_ASPECT: gains 512x512 and 256x256 (both 1:1).
  • _base_url() replaces the hardcoded host. The request's own base URL is the default, so the answer is right on localhost, on a LAN address or behind a proxy. X-Forwarded-Proto / X-Forwarded-Host are honoured only when they assemble into a bare scheme://host[:port] — no path, no query, no credentials, and only characters a DNS name, an IPv4 literal, a bracketed IPv6 literal or an internal hostname can contain. These are caller-supplied strings on their way into a response body, and a URL in a response is fetched by whoever asked for it. Scheme and host fall back independently, so a proxy that forwards the host but not the scheme is still handled.
  • _media_url() builds <base>/api/v1/media/<name> from that.
  • openai_images (/v1/images/generations & /api/v1/images/generations):
    • When response_format is "image" (also accepts "file", "binary", "raw", "preview"), the first generated file is returned via FileResponse, enabling instant browser/Swagger UI rendering and the native download trigger.
    • The declared 200 content types now list image/png and image/jpeg alongside application/json.
    • A run that exited zero but produced no image raises 502 with the reason, instead of returning 200 with an empty data array — which an OpenAI client reads as "the request worked and there were simply no images", the opposite of what happened.
  • flow_media: candidates are the engine's root, FLOW_OUTPUT_DIR and MEDIA_DIR, each checked with Path.is_relative_to. A string prefix is not containment — /out is a prefix of /out-evil, so a sibling directory sharing a prefix would otherwise be served. The content type comes from mimetypes rather than a hand-kept extension table, because the preview page exists to render images inline, and serving one as application/octet-stream makes the browser download it instead. There is deliberately no __file__-relative candidate: the package's install layout has nothing to do with where the engine writes.
  • flow_image & flow_video: each item in result["files"] gains a fully-qualified, reachable url, so a client on another device gets a fetchable link instead of a server-local path.
  • Web Preview Dashboard (GET / and GET /preview): a single-page UI over /v1/images/generations with a live elapsed-time counter, inline image preview and one-click download.
  • Comments, docstrings, Swagger metadata and the dashboard strings are English throughout.

flow_agent/engine.py

  • _find_binary reads platform.machine() and prefers flow-linux-arm64 / flow-arm64 on aarch64 / arm64, falling back to flow-linux (from PR feat: support Linux ARM64 auto-detection, default to BELUGA, and sanitize Swagger UI dummy inputs #22).
  • clean_model() and clean_cookies() are the only implementations of the two input rules, with DEFAULT_IMAGE_MODEL, _MODEL_PLACEHOLDERS and _DUMMY_COOKIES naming the literals. A placeholder means "not given" and becomes the default; anything else is used as given, because rewriting a name that was asked for runs something other than the request while reporting success — and main.py passes its own narwhal default straight through, so the CLI and the API would otherwise disagree about the same word.
  • generate_image defaults to beluga.
  • generate_image, generate_video, upload_image and list_projects build their --cookies argument through clean_cookies(), and a placeholder is left off the command line entirely rather than handed to the binary as a session cookie already known to be wrong. _is_valid_cookie_arg is gone.

flow-agent/tests/test_api.py & flow-agent/tests/test_engine.py

  • test_openai_images_passes_an_explicit_model_through — a named model reaches the engine unchanged.
  • test_openai_images_replaces_a_placeholder_model_with_the_default — Swagger's "string" becomes beluga.
  • test_media_serves_a_png_as_a_png — the browser is told image/png, so the preview page renders it.
  • test_media_refuses_a_sibling_directory_sharing_a_prefix — the .. is sent percent-encoded on purpose, since a literal one may be collapsed by the HTTP client before the request reaches the route, and the test would then pass for the wrong reason.
  • test_clean_model_* / test_clean_cookies_* — the two rules, including the case where the CLI and the API must agree.
  • test_a_placeholder_cookie_never_reaches_the_command / test_a_named_model_reaches_the_command_unchanged — the argv the engine actually builds.

Test Results

pytest on Windows: 98 passed, 2 failed (Python 3.13 on the author's machine, Python 3.11.9 on the verifier's; both report the same two failures, which appear on every commit here, including the ones before this one).

The two failures are test_command_drops_arguments_that_were_not_given and test_command_keeps_falsy_values_that_are_not_absent. Both assert /bin/echo and are handed \bin\echo. They fail the same way on this branch's earlier commits, before any of the changes above, so they are path-separator artifacts of running the suite on Windows rather than regressions.


AI Disclosure

Transparency Statement:
This contribution was developed and verified through pair-programming between the author and Antigravity (Google DeepMind agentic AI coding assistant), with a further review pass by Cline.
All FastAPI schema validators, FileResponse streaming handlers, reverse-proxy URL composers, and web UI dashboards were thoroughly tested and validated against active server runs on Debian GNU/Linux (aarch64).


Hardware & Verification Details

Test Environment

  • Device: Raspberry Pi (aarch64)
  • OS: Debian GNU/Linux 13 (trixie) / Raspberry Pi OS v8
  • Kernel: Linux 6.18.50+rpt-rpi-v8 #1 SMP PREEMPT aarch64 GNU/Linux
  • Python: Python 3.13.5
  • Server Port: 8001 (FastAPI / Uvicorn)

Verification Results

  1. Swagger UI Direct Image Stream (response_format: "image"):
    • Tested via Swagger UI /docs on remote LAN device (192.168.219.131:8001).
    • Image rendered directly in the Swagger response pane with a native "Download file" link.
  2. OpenAI Standard JSON Compatibility (response_format: "url"):
    • Verified that client.images.generate() returns http://192.168.219.131:8001/api/v1/media/<filename> which is publicly fetchable across the LAN.
  3. Web Dashboard (GET /preview):
    • Verified end-to-end prompt submission, timer countdown, and one-click image download on mobile & desktop browsers.

Follow-up in this revision

  • response_format: "image" now names the file's real type. The branch
    labelled every previewed file image/jpeg and its docstring claimed PNG was
    what the engine writes today, while README.md, docs/API_REFERENCE.md and
    the media route all say .jpg. The type is read from the file with
    mimetypes instead — the rule the media route documents and uses — with
    application/octet-stream for an extension mimetypes does not know.
  • The preview branch now has tests. test_openai_images_streams_the_file_it_was_asked_for
    insists a streamed JPEG arrives as image/jpeg with its bytes intact, and
    test_openai_images_streams_a_png_as_a_png insists a streamed PNG is not
    called a JPEG. The second fails against the previous code, which is the point:
    the hardcoded type is what an asserted content type was missing.
  • Interaction with feat: implement API key authentication, CLI management, and endpoint protection #24 (feat/api-key-authentication). feat: implement API key authentication, CLI management, and endpoint protection #24 puts
    dependencies=[Depends(verify_api_key)] on /v1/images/generations and the
    other credit-spending routes. The dashboard in this PR generates through that
    route with no credential (flow_agent/api.py:877), so once any API key is
    registered the dashboard receives 401s while GET / and the media routes stay
    open. The two branches also conflict in flow-agent/flow_agent/api.py - this
    PR rewrites the same route decorators feat: implement API key authentication, CLI management, and endpoint protection #24 adds that dependency to - so the
    second of the two to merge needs it rebased by hand.

…tize Swagger UI dummy inputs

- engine: inspect platform.machine() to prioritize ARM64 binaries on Linux ARM64
- engine: filter out dummy cookie values (e.g. 'string', 'null') before passing CLI args
- engine/api: update default image model to BELUGA and add sanitization fallback
- api: sanitize Swagger UI default placeholder ('string') in request models to None
- Support direct image streaming ('response_format': 'image') for instant preview/download in Swagger UI
- Add default 'image' response format and default prompt to ImageGenerationRequest
- Add X-Forwarded header support in _media_url and robust search paths in flow_media
- Inject downloadable HTTP URLs into native flow_image endpoint
- Add interactive web preview dashboard at / and /preview
@iceblue03
iceblue03 marked this pull request as ready for review October 17, 2026 16:38
Copilot AI balanced review requested due to automatic review settings October 17, 2026 16:38

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

The image-preview work carried three regressions over main:

- `response_format` defaulted to `image`, so an OpenAI client that sends no
  `response_format` (the default) got raw bytes back and failed to parse them
  as JSON. The default is `url` again; `image` stays available as an explicit
  opt-in, which is what the preview page sends anyway.
- `flow_media` read `engine.root` while building its search list, outside any
  guard. `root` is optional on the engine contract, so the route raised
  `AttributeError` (500) instead of 404 and broke `test_media_serves_a_real_file`,
  `test_media_refuses_a_traversal` and `test_media_refuses_a_missing_file`.
- the containment check compared string prefixes, and a prefix is not
  containment: `/out` starts `/out-evil`, so a sibling directory that merely
  shared a prefix was served. `Path.is_relative_to` compares path parts.

Also:

- content types come from `mimetypes`. The hand-kept table omitted `.png`, so
  the format Flow actually produces went out as `application/octet-stream` and
  the browser downloaded it instead of rendering it - the one thing the preview
  page exists to prevent.
- drop `test_client.py`, committed as a symlink into a home directory
  (`/home/admin/...`) and therefore dangling on every other machine.
- two regression tests: a PNG is served as `image/png`, and a sibling directory
  sharing a prefix is refused. Both are encoded so that a client which collapses
  `..` cannot make them pass for the wrong reason.

Verified: `tests/test_api.py` 29 passed; full suite 77 passed, with 2
Windows-only path-separator failures that are present on `main` too.
Follow-up to the preview-dashboard commit: it changed more than its stated
scope, and it reads one new header that was echoed unchecked.

- `prompt` is required again. It is the field OpenAI marks required, and every
  generation spends a credit, so a server-side default charged an account for a
  sentence the caller never wrote and returned an image nobody asked for.
- `X-Forwarded-Host` and `X-Forwarded-Proto` are used only when they assemble
  into a bare `scheme://host[:port]`, because they reach a response body and a
  URL in a response is followed by whoever asked. A forwarded value carrying a
  path, a query, a fragment or an `@` built a link pointing somewhere else;
  those fall back to the request's own base URL now. A proxy that forwards the
  host but not the scheme still gets its host, and a scheme that is not http(s)
  cannot reach the URL.
- The `__file__`-relative guess at an output directory is gone. It can only
  name a directory this package's install layout happens to have, never the one
  the engine wrote to.
- The `sanitize_cookies` validator this added is gone: the engine already drops
  placeholder cookies at the command line, so it was a third copy of a rule
  already stated twice.
- The declared response types list PNG as well as JPEG, since the type is
  resolved per file and PNG is what the engine writes.
- The dashboard's strings, the comments and the Swagger metadata are English,
  as everywhere else in the repo, and the rationale comments this commit had
  replaced are restored.

The `narwhal` -> `beluga` fallback and the duplicated `sanitize_model` are left
untouched: both predate this branch.
Follow-up to the preview-dashboard commit, which introduced three copies of the
cookie rule and two of the model rule - none of which upstream main has.

- `clean_model` and `clean_cookies` in `engine.py` are now the only
  implementations. The two `sanitize_model` validators delegate to the first;
  the duplicate `sanitize_cookies` validators are gone, because the engine
  applies the same rule to the argv it builds for every route anyway.
- A model that was named is no longer rewritten. `narwhal` was turned into
  `beluga` in silence, while `main.py` passes its own `narwhal` default
  straight through - so the CLI and the API disagreed about the same word, and
  a request that runs a different model while reporting success is the failure
  that is hardest to notice. The `beluga` default is unchanged, and a
  placeholder still falls back to it.
- `--cookies` is left off the command line when the value is a placeholder and
  stripped when it is kept, at all four call sites rather than at the routes.
…st it

`response_format: "image"` labelled every previewed file `image/jpeg`, whatever
the engine had actually written, and the docstring claimed PNG was the format
the engine produces today -- the README, docs/API_REFERENCE.md and the media
route all say `.jpg`. The declared `image/png` was therefore a type the route
could not return in practice.

The type now comes from the file, which is the rule the media route already
documents and uses, so the label and the bytes agree. An unknown extension falls
back to `application/octet-stream`, matching the media route, rather than to a
guess.

The branch had no test for the preview branch at all -- the feature this PR is
named after -- and an asserted type is exactly what the hardcoded one lacked.
`test_openai_images_streams_the_file_it_was_asked_for` covers a JPEG and
`test_openai_images_streams_a_png_as_a_png` covers a PNG; the second fails
against the previous code with `image/jpeg`.

pytest tests -q: 98 passed, 2 failed (both pre-existing, see the PR body).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants