feat(1.1.0): image recognition

[jaymar921]

1.1.0 — image recognition

HirayaCoder can look at pictures. Attach a screenshot, a photo, or a sketch of a screen
you want built, and what is in it becomes part of the conversation: the subject of the
answer in Ask mode, part of the job in Agent and Plan.

The release also cuts the README to a fifth of its length and moves the rest into doc/.

Three commits, reviewable in order. The middle one is a bug the first one shipped and
the third is a small feature on top; they are kept separate because the bug is worth
reading on its own.

---

1. The feature

New module: app/core/imageRecognition.js. It asks a vision model what is in an attached
image and returns plain text. That indirection exists for three reasons, and each would
be enough on its own:

Most models cannot see. llama3.2 is what a lot of people are actually running, and
it has no vision capability at all. Attaching an image to it used to be refused outright.
Now the picture goes to whichever installed model can see, the description comes back,
and the coding model works from that. Nobody switches models to use a screenshot, and the
attach button is gated on the machine having a vision model rather than on the selected
one being it.

The picture does not survive the loop. Images ride on the first message only —
re-uploading 5 MB of base64 on every turn of an eight-step run spends more time on the
upload than on the thinking. So by turn four the model was reasoning about an image it
could no longer see. A description is a hundred tokens and is carried on every turn, which
is why one is produced even for a model with perfectly good vision.

A description can be checked. A model that hallucinates an image is invisible when the
image is a blob on a message. As a paragraph in the transcript, the user reads "a red
error dialog" over their green one and knows immediately. Every attachment gets a
collapsed panel holding the exact text that was read.

Which model does the looking

Automatic: the selected model when it can see, since that costs no extra load, and
otherwise the smallest installed vision model. Smallest because Ollama holds one model
resident at a time, so a second model is paid for in load time, and describing a picture is
a read rather than a piece of reasoning.

hirayacoder.vision.describeModel overrides it; hirayacoder.vision.enabled turns the
pass off. A name in settings is honoured only if that model is installed and reports
vision — otherwise the automatic choice is used and the reason goes to the log, rather than
silently disabling images.

---

2. Two bugs fixed on the way

_rethink was arguing the model out of correct answers. It sends a mismatched reply
back for one correction, and it was sending the context without the images — so a reply
written from a photograph was corrected by a model that could not see the photograph. The
redraft turned good answers into hedges about not being able to see anything.

"I cannot see this image", stored as the description. Found by the new benchmark, once
in twenty-four runs of qwen3.5:0.8b: a model that reports the vision capability, that
was sent the image, and that had described the same photograph correctly one sample
earlier. Left alone, that paragraph becomes what the coding model is told is in the
picture. The shape is now recognised and reported as a failed read.

The detector is deliberately narrow. The recognition prompt asks the describer to say when
it cannot make something out, so "I cannot see the licence plate clearly" is the
instruction working. It requires a first-person inability plus a visual object, and
examines only the opening 400 characters, so a real description that ends with a
boilerplate disclaimer survives.

---

3. The bug the first commit shipped (d0093ff)

Worth reading, because it was not caught by any test and the benchmark could not have
caught it.

In Ask mode on minicpm-v4.6, "describe the image" with a photograph of a dog attached
returned a description of HirayaCoder.

Not a vision failure. The same model and the same photograph score 24/24 in
tools/bench-vision.js, and the image was on the message. It was an instruction-following
failure, and the instructions it followed were ours:

• Ask mode's system prompt opens with "everything below this line is what you know about
  the user's project… Answer from it."
• Under it sat 2,500 characters of project description and file listing, with
  Task: describe the image on the last line.

A small model resolves that conflict by weight, and forty file paths outweigh one line of
task.

Fix, two parts. On a turn with no tools that carries an image, the project overview and
the workspace listing are dropped — they are orientation the model cannot act on, and they
were the thing being answered from. And every prompt gains a short instruction saying an
image is attached and is the subject of the question.

After both, the user turn went from 2,510 characters to 24, and the same model answered:

a cheerful Corgi with a fluffy white and tan coat, lying comfortably in a sunlit grassy
field… wears a green collar

The listing is kept in Agent and Plan mode, where it is load-bearing: a model asked to
build the screen in a mockup needs to know which paths exist, and there the picture is a
fact about the job rather than a competitor to it.

contextBuilder already carried a comment describing this exact shape for the chat route —
a model handed a project description and the message "Hello Hiraya" answers with the project
description. Same bug, different trigger, and the connection was missed the first time.

---

4. Every reply says which model wrote it and how long it took (1b82f0c)

minicpm-v4.6:latest • 25s

On this hardware the model is the main lever anyone has — a task is one to five minutes and
the way to make it faster is to run something smaller — so choosing between them means
comparing replies that are minutes and several messages apart. The status line could not
serve that: it is per-tab and the next turn overwrites it, so it says what is happening and
never what happened.

The duration is the run rather than the wait. AgentSession.run is not entered until the
turn owns the lane, so a turn queued behind another tab does not charge that wait to its own
model. ms and modelMs were already computed for the log line and thrown away; they are
now on the result too.

When a different model read an attached image it is named separately:

llama3.2:latest • 17s     + minicpm-v4.6:latest read the image, 8s

That is where the extra eight seconds went. Without it the coding model looks like it got
slower. A model that described its own image is not named twice.

---

The benchmark

tools/bench-vision.js. Six photographs in docs/test-images/, graded on four axes:
subject, detail, text read, and confusion. Confusion can only be lost, because vague is
usable and wrong is worse than nothing — the user cannot tell.

Grading is a word list rather than a model. Grading one local model's output with another
makes the benchmark's own reliability the thing in question, and a word list cannot be
talked into a false positive. It is word-boundary matched: the first draft used includes
and scored a description of a cat on a carpet as having identified a car.

Baseline, machine B, 24 graded descriptions per model:

Model	Params	Subject	Detail	Text read	Confused	Cold
minicpm-v4.6	752M	24/24	42/48	4/8	0	11s
qwen3.5:0.8b	873M	23/24	46/48	8/8	0	13s

Naming the subject is solved at this size. Reading text is not, and it depends on the size
of the text rather than of the model: all four of minicpm-v4.6's text failures are the
same photograph, whose only text is a small badge on a car grille, and it read a large clear
airline logo on every attempt.

The timings needed a --cold flag to mean anything. Ollama caches the tokenized prompt and
the prompt here is mostly the image, so the second sample of a picture returned in one second
where the first took fifteen — and the cache survived between runs, reporting one second
for everything half an hour later. --cold unloads the model between samples. None of it
touches the accuracy scores.

---

Documentation

README: 514 lines to 259, no em-dashes, cut to what somebody deciding whether to install this
needs. Everything else moved whole into doc/:

New file	What moved into it
doc/GETTING-STARTED.md	The full first-run walkthrough
doc/USING-IT.md	Modes, watching a run, request habits
doc/LIMITATIONS.md	New. Everything it cannot do, by design and otherwise
doc/IMAGE-RECOGNITION.md	New. Images, in full
doc/TROUBLESHOOTING.md	When something goes wrong, and the command allow-list
doc/CHOOSING-A-MODEL.md	Which model to download
doc/DEVELOPING.md	The engineering, benchmarks, building, contributing

LIMITATIONS.md is the one worth a look. The limits were previously spread across five
sections and phrased apologetically where they were phrased at all, and the two kinds —
deliberate design choices, and things a small model is simply not good at — were mixed
together. They are separated now, because the first kind is the reason to choose this tool
and the second is the reason not to.

Two 1:1 ads for the release in docs/images/, sources and captions alongside. ad-6 carries
measurements on its face; ADS-1.1.0.md records where each number comes from and says to
re-run the sweep before re-rendering it.

---

Security

Full report: security/sast-report-2026-08-21-1.1.0.md. Zero production dependencies, zero
advisories, no Critical or High findings.

The release adds a genuinely new shape — an image file chosen by the user, base64-encoded,
sent to a local model, whose free-text reply is then placed in another model's prompt — and
that is where the pass went. Three fixes:

• SAST-020: a credential visible in a screenshot reached the prompt unredacted. Every
  other block contextBuilder assembles is passed through redact; the image description
  was not. This path does not merely permit a secret through, it asks for one — the task
  recognition prompt instructs the describer to copy visible text exactly, because a misread
  filename sends the agent to the wrong file. A screenshot of a terminal showing
  export OPENAI_API_KEY=sk-… yields a description containing the key by design. Now
  redacted, covered by a test. The first draft of the report argued for accepting this on the
  grounds that a false positive would blank the answer; measuring redact showed that to be
  simply wrong — it replaces the matched token and leaves the sentence standing.
• SAST-018: 3.6 MB of benchmark fixtures would have shipped in the .vsix. docs/** is
  not ignored wholesale, so a new subdirectory under it is packaged by default. Rule added;
  vsce ls goes from 103 files to 97.
• SAST-016: guard ordering. The only unbounded regex in the dictation path ran before
  the 400-character bound documented as keeping that check linear. Measured linear anyway
  (0.030 ms at 40,000 characters), so this is ordering rather than a live fix — but it means
  one is never reachable there if the pattern is later edited into an ambiguous one.

---

Testing

Unit	1595 passing (up from 1581)
Integration	16 passing, against a real VS Code
Lint	0 errors
npm audit --omit=dev	0 vulnerabilities
Semgrep	6 findings, 5 locations, all the pre-existing detect-child-process rule

Run against real models, per CONTRIBUTING. This touched the agent loop, the prompts, and
the context builder, so the mocked suite is not evidence on its own — and it was not: the
bug in section 3 passed every test in this PR and was found by installing the build and
clicking the button.

Verified end to end against live Ollama:

• Ask mode, minicpm-v4.6 describing its own image → correct description of the corgi
• Ask mode, llama3.2 (no vision at all) answering from minicpm-v4.6's description, with
  no image on its own message
• Agent mode, the file listing still present where it is load-bearing
• The reply footer in all three model arrangements: one model (2s), two models (17s + 8s),
  no image (4s)

New tests: test/unit/imageRecognition.test.js (27), test/unit/imageTurns.test.js (19),
plus footer formatting in webviewComponents.test.js.

---

Known limitations

• Small text in a screenshot is unreliable, and a misread arrives as a confident
  paragraph rather than as an apology. This is why the description is shown to the user
  rather than hidden. Documented in doc/IMAGE-RECOGNITION.md and doc/LIMITATIONS.md.
• The reply footer does not appear on messages restored from a transcript. The transcript
  stores role and text only; widening its schema is a bigger change than this warranted.
• The benchmark is small-sample — six photographs, two models, two samples each. Enough
  to find two real bugs, not enough to rank models confidently.
• qwen3.5:8b does not exist in the Ollama registry (404), so the second benchmarked model
  is qwen3.5:0.8b, which is within 121M parameters of minicpm-v4.6 and makes a fair pair.

---

Review notes

The app-code surface is 9 files, +1141/-30. The rest is docs, tests, fixtures, and
benchmark results.

Suggested order:

1. app/core/imageRecognition.js — the new module, self-contained
2. app/agent/agentSession.js — _describeImages, _visibleImages, and the timing on the
   result
3. app/core/contextBuilder.js + app/core/promptRouter.js — the fix in section 3
4. Everything else is plumbing and UI