Skip to content

fix: report raw_content_status and raw_content_error in the HTTP JSON - #144

Open
konpan wants to merge 1 commit into
sahenjp:mainfrom
konpan:fix/expose-raw-content-status
Open

konpan wants to merge 1 commit into
sahenjp:mainfrom
konpan:fix/expose-raw-content-status

Conversation

@konpan

@konpan konpan commented Nov 15, 2026 •

Copy link
Copy Markdown

Problem and resulting behavior / 解決する問題と変更後の動作

#128 added raw_content_status and raw_content_error to SearchResult and fills them in Aggregator::attach_raw_content(). The README says that with include_raw_content=1 every result reports raw_content_status. However, upstream_json() in src/web/feeds.rs builds each result object by hand with json! and only copies raw_content and raw_content_truncated, so the two fields never reach the HTTP JSON (format=json, with or without full=1).

Trigger: a running 0.31.2 binary, /search?q=Rust+tokio+tutorial&format=json&full=1&include_raw_content=1. Every result lacked both fields, including the results past raw_content_limit, which should carry not_requested.

Before: a result whose page could not be read (fetch_failed, too_large, render_failed, ...) or that was past raw_content_limit has no raw_content and no explanation, so a client cannot tell why the body is missing.

After: with include_raw_content=1, each result carries raw_content_status and, where available, raw_content_error, also when raw_content itself is absent. json_response_full inherits this because it extends upstream_json.

Follow-up to #128.

Scope and compatibility / 範囲と互換性

  • Only upstream_json() changes (two if let blocks), plus two unit tests in the same file.
  • The fields are written only when SearchResult has them, which is only when the request asked for raw content (attach_raw_content runs under if request.include_raw_content). Searches without include_raw_content produce exactly the same JSON as before; json_without_raw_content_request_has_no_status_fields pins that.
  • Absent values are omitted, matching the skip_serializing_if = "Option::is_none" on the struct fields.
  • No config, API parameter, default or dependency changes.
  • Not included: the MCP output types (SearchItem, ReadPageOutput) have no status fields either. That is a separate tool contract, so it is left out of this PR; happy to open a separate one.

Validation / 検証

  • Relevant offline tests or fixtures cover the behavior: both_json_forms_report_raw_content_status_and_error (fails before the change: raw_content_status is Null) and json_without_raw_content_request_has_no_status_fields (guards the unchanged case).
  • rustfmt --check --edition 2024 on src/web/feeds.rs (rustfmt.toml): no diff. cargo fmt --all --check itself was not run; this is the only changed file.
  • cargo test --lib (stable-x86_64-pc-windows-msvc, Windows 11, -j 2): 1061 passed, 1 failed, 1 ignored (after rebasing onto the current main, f7639ce). The failure is fetch::pdf::tests::pdf_without_text_requires_ocr: the local pdftotext is xpdf 4.06 (shipped with Git for Windows), not Poppler, and exits 99 on tests/fixtures/empty.pdf. Unrelated to this change.
  • cargo test --all-targets: not run (only --lib; the integration tests in tests/ were not executed on this machine because of memory limits). tests/integration.rs reads only raw_content and raw_content_truncated from the JSON, so the added fields do not affect it.
  • cargo clippy --all-targets -- -D warnings: not run (same memory limit). Relying on CI for this.
  • Markdown lint and links: not applicable, no documentation changed.

Limitations / 残る制約

  • Unit-level coverage only; the missing fields were observed on a running 0.31.2 build before the change, and the fix is covered by unit tests, not by an added integration test.
  • MCP output is unchanged (see above).
  • Windows-only local run; Linux/macOS are left to CI.

Contribution notes / 作成・レビューについて

I found the gap while checking the #128 features against a running 0.31.2 build. The patch, the tests and this description were drafted with AI assistance (Claude Code); I reviewed the diff and the check results listed above.

🤖 Generated with Claude Code

日本語の要約

#128 では raw_content_status と raw_content_error が追加されましたが、HTTP の JSON(format=json)にはこの2つの欄が出力されていませんでした。src/web/feeds.rs の upstream_json() が結果の JSON を1件ずつ組み立てる処理で、この2つの欄を加えていなかったためです。

この PR は、upstream_json() が2つの欄を出力するように変更し、その動作を確かめる試験を2本追加します。include_raw_content を指定しない検索の JSON は変わりません。設定・引数・依存関係も変更していません。

@konpan
konpan requested a review from sahenjp as a code owner November 15, 2026 10:44
upstream_json() built each result by hand and copied only raw_content
and raw_content_truncated, so the status fields added in sahenjp#128 never
reached format=json responses. Copy both fields when present and add
tests for the requested and not-requested cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@konpan
konpan force-pushed the fix/expose-raw-content-status branch from 1a98ddd to c14dd1b Compare November 15, 2026 10:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant