Skip to content

Make the OpenLibrary fallback reachable - #861

Open
krejko wants to merge 3 commits into
Listenarrs:canaryfrom
nexalapp:fix/openlibrary-prefixed-query
Open

Make the OpenLibrary fallback reachable#861
krejko wants to merge 3 commits into
Listenarrs:canaryfrom
nexalapp:fix/openlibrary-prefixed-query

Conversation

@krejko

@krejko krejko commented Aug 20, 2026

Copy link
Copy Markdown

Summary

SearchService merges OpenLibrary-derived candidates when no enriched metadata was found, so books with no Audible edition still produce results:

// Merge OpenLibrary-derived results … so OpenLibrary-only augmentation produces
// visible, scoreable items without calling Amazon.
if ((openLibraryDerivedResults != null && openLibraryDerivedResults.Any()) && !enrichedList.Any())

That branch can never be taken. Three independent defects upstream of it leave OpenLibraryDerivedResults empty, so the fallback silently does nothing and such books return zero results.

The three defects

1. The composed query is used as the OpenLibrary title.

var books = await _openLibraryService.SearchBooksAsync(query, null, 5);
//                                    ↑ "AUTHOR:Abelson TITLE:Structure and Interpretation…"

NormalizeForOpenLibrary tokenises on [a-z0-9\-]+, dropping the colons and leaving AUTHOR/TITLE as literal search terms. The author is also discarded as null, despite the caller having parsed it.

2. Exact title matches are discarded.

if (!string.IsNullOrEmpty(book.Title) && !string.Equals(book.Title, query, StringComparison.OrdinalIgnoreCase))

An exact title match is the strongest candidate available. Searching Radicalized by Cory Doctorow returns six documents, the first two titled exactly Radicalized — all dropped. The Garden of Rama kept only a four-book omnibus, precisely because the omnibus title differs from the query while the actual book's does not.

3. Query terms are joined with +, then URL-escaped.

var q = string.Join("+", qParts);
searchParams.Add($"q={Uri.EscapeDataString(q)}");   // "+" → %2B

OpenLibrary receives literal plus characters between words. Measured against the live API:

Query numFound
radicalized%2Bcory%2Bdoctorow 0
radicalized cory doctorow 6
the%2Bgarden%2Bof%2Brama%2Barthur%2Bc%2Bclarke 1 (omnibus only)
the garden of rama arthur c clarke 5

This is the dominant one — it corrupts every OpenLibrary query the app makes.

Changes

Fixed

  • Pass the caller's already-parsed title and author into SearchBooksAsync (SearchService logs both one line before the call); strip FIELD: prefixes for callers that only have a composed string.
  • Keep documents whose title equals the query and let existing scoring/dedupe rank them.
  • Join query terms with spaces so escaping produces separators rather than literal +.

Added

  • AsinCandidateCollector_OpenLibraryQueryTests — parsed fields preferred, prefixes stripped when absent, plain queries untouched, and no call when OpenLibrary is disabled.

Testing

34/34 pass, including the existing parser suite.

Verified end-to-end against a running instance. Log lines that never appeared before now do:

Collecting candidates from OpenLibrary (title='Radicalized', author='Cory Doctorow')
OpenLibrary suggested title: Radicalized
Merging 1 OpenLibrary-derived candidate(s) into enriched results

Real searches, before → after:

Query Before After
Radicalized / Cory Doctorow 0 results 2'Radicalized' first
The Garden of Rama / Arthur C. Clarke 1 — 'Rama Series, Collection Set of 4 Books…' 1 — 'The Garden of Rama'
Structure and Interpretation… / Abelson 0 results 1

A control case still correctly returns nothing: Chronicles of Narnia Intro is an intro track rather than a book, and OpenLibrary has no such title (numFound=0); dropping "Intro" returns 234.

Notes

Each defect masked the next, so fixing any one alone leaves the symptom unchanged — I confirmed each independently against the live OpenLibrary API before and after.

Behaviour is unchanged when EnableOpenLibrarySearch is off, and no Audible/Audnexus path is touched; this only affects queries that would otherwise have returned nothing.

🤖 Generated with Claude Code

krejko and others added 3 commits August 20, 2026 16:55
AsinCandidateCollector passed the composed query straight through as the
OpenLibrary title:

    var books = await _openLibraryService.SearchBooksAsync(query, null, 5);

The composed query still carries its field prefixes, so the value was
"AUTHOR:Abelson TITLE:Structure and Interpretation of Computer Programs".
OpenLibrary's normalizer tokenizes on [a-z0-9-]+, which drops the colons and
leaves "AUTHOR" and "TITLE" behind as literal search terms:

    q=AUTHOR:Abelson TITLE:Structure and Interpretation...  -> numFound 2
    q=AUTHOR Abelson TITLE Structure and Interpretation...  -> numFound 0
    q=Abelson Structure and Interpretation...               -> numFound 2

With no documents returned, OpenLibraryDerivedResults stayed empty and the
existing fallback that merges OpenLibrary candidates when no enriched metadata
was found could never fire. Books with no Audible edition silently returned no
results at all.

Pass the fields the caller already parsed - SearchService logs them one line
before the call - and use the author too, which was previously discarded as
null. Callers that only have a composed string get the prefixes stripped
instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The candidate loop skipped any document whose title equalled the search term:

    if (!string.IsNullOrEmpty(book.Title) && !string.Equals(book.Title, query, ...))

An exact title match is the strongest candidate available, so this discarded the
best result for every book whose catalogue title matches what was searched for.
Searching "Radicalized" by Cory Doctorow returns six OpenLibrary documents, the
first two titled exactly "Radicalized"; all of them were dropped and the search
returned nothing. "The Garden of Rama" kept only a four-book Rama omnibus,
because the omnibus title differs from the query while the actual book's does
not.

Keep every document with a title and let the existing scoring and dedupe decide.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The query was assembled by joining tokens with "+" and then escaping the whole
string. Uri.EscapeDataString encodes "+" as %2B, so OpenLibrary received literal
plus characters between the words rather than separators, and matched almost
nothing:

    q=radicalized%2Bcory%2Bdoctorow                    -> numFound 0
    q=radicalized cory doctorow                        -> numFound 6
    q=the%2Bgarden%2Bof%2Brama%2Barthur%2Bc%2Bclarke   -> numFound 1
    q=the garden of rama arthur c clarke               -> numFound 5

The single Rama hit was a four-book omnibus whose title happened to survive the
corrupted query, which is why the fallback looked like it was working
occasionally rather than failing outright.

Join with spaces and undo the plus-joining that NormalizeForOpenLibrary applies
to its own tokens.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@krejko
krejko requested a review from a team August 20, 2026 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant