Add Lichess as a second platform in 1d4

[aaylward]

ROADMAP.md Phase 8 plans this, but it was written against an architecture that has since moved. This is what the work actually looks like today.

What is already done

More than the roadmap implies. The platform dimension exists end to end:

• Schema, for the platform dimension. platform VARCHAR(50) NOT NULL on game_features, indexing_requests and indexed_periods, and every uniqueness constraint is already keyed by it — indexed_periods_unique (player, platform, year_month, exclude_bullet), and dedupe_key starts with the platform. Nothing here needs a migration. (A persisted titles roster does — see below.)
• The row already carries it. index_run.cc writes row.platform = job.platform; IndexJob carries platform off the claimed row.
• ChessQL parses it. platform IN ["lichess", "chess.com"] compiles to LOWER(platform) IN (LOWER(?), LOWER(?)) and is tested. Note: it compiles but does not match anything — see ChessQL platform filters match nothing: rows store CHESS_COM, queries say chess.com #1539.
• The fetch seam exists. ArchiveSource in one_d4_worker/archive.h is a port, with ChessComArchive as the only implementation and FakeArchive in tests.

What actually blocks it

Two places, both small:

1. IndexRequestService.canonicalPlatform rejects everything but CHESS_COM. One if. IndexRequestServiceTest.validationRejectsBadInput asserts submitting lichess throws, so that assertion inverts.
2. worker_main.cc builds one ChessComArchive at startup and hands it to MakeRun for every job. Done in Choose the archive per job's platform (#1527 slice 3) #1538 — MakeRun takes a PlatformArchives registry and chooses per job; chess.com is still the only entry.

Then the web <select> in IndexView.tsx grows a second <option> — and the games views learn to show which platform a row came from, which they do not today. See slice 6.

What is genuinely new

The client can be generated — after an opal-cpp pin bump

Lichess's export endpoint returns a stream, not a JSON document:

GET /api/games/user/{username}?since={ms}&until={ms}
Accept: application/x-chess-pgn   (default)
     or application/x-ndjson

opal-cpp HEAD (6bb66f1) generates clients for exactly this; the pin (dee8162, bazel/opal.MODULE.bazel) predates it. Three additions land between them:

• @streaming blob response payloads in generated clients (opal-cpp Update toolchains_llvm digest to 6645d91 #213 slice 2). An operation whose response @httpPayload targets a @streaming blob gains a defaulted writer:

  opal::Outcome<DownloadOutput> Download(const DownloadInput& input,
                                         const opal::http::BodyWriter& write = nullptr) const;

  Bytes go to the writer as they arrive and the member is left empty. The generated code owns the accept gate and keys it on the operation's modeled success code, so a modeled error still deserializes into the typed errors from its own body rather than landing in the caller's writer.
• HttpClient::SendStreaming / opal::http::BodySink (Update toolchains_llvm digest to 6645d91 #213 slice 1) — accept(status, headers) once per response, write(piece) in order. BeastHttpClient streams for real and never assembles an accepted body.
• Retry-After honored (Update dependency protobuf to v27 #189), both delta-seconds and HTTP-date, as a floor under the backoff, capped by RetryPolicy::retry_after_cap (default 60s). This matters here specifically — see rate limiting below.

So the model is ordinary except for the payload:

@readonly
@http(method: "GET", uri: "/api/games/user/{username}", code: 200)
operation ExportGames {
    input := {
        @required @httpLabel username: String
        @httpQuery("since") since: Long
        @httpQuery("until") until: Long
        @httpHeader("Accept") accept: String
    }
    output := { @required @httpPayload games: GameStream }
}

@streaming blob GameStream

The PGN/NDJSON parsing lives in the BodyWriter, not in generated code. Constraints worth knowing before modeling:

• Only an @httpPayload blob qualifies. A @streaming blob bound anywhere else is base64 inside a JSON document and stays buffered.
• Generation fails with a diagnostic if the streaming payload rides a modeled status the retry layer treats as transient, because the retry loop withholds such a status from the sink.
• Nothing in the generated client or runtime sets Accept, so binding it on the input should pass through — worth confirming when modeling, since it is the format selector.

Pin bump cost: one breaking change. Of the four entries added since dee8162, three are additions and one is breaking — client responses are now capped at ClientConfig::max_response_bytes, default 64 MiB. chess.com archives are nowhere near it (hikaru August 2026: 493 games, 2.0 MB), so the existing client is unaffected. It is also precisely why streaming matters for Lichess: a full archive is unbounded — the spec cites an account with 500,000 games — and the sink path is the one thing the cap cannot substitute for.

Titles: Lichess states them per game, chess.com does not

Verified against live data:

# lichess.org/game/export/IDYCWNNH
[White "Sultai"]          [WhiteTitle "CM"]
[Black "Zhigalko_Sergei"] [BlackTitle "GM"]

Against 493 games from hikaru's August 2026 chess.com archive, zero carry any *Title tag — not even Hikaru's own. The full tag set chess.com emits is Black BlackElo CurrentPosition Date ECO ECOUrl EndDate EndTime Event Link Result Round Site StartTime Termination TimeControl Timezone Tournament UTCDate UTCTime White WhiteElo.

That asymmetry is why TitleRoster exists: chess.com's per-game data has no title, so the worker reads the ten /pub/titled/{title} rosters and answers TitleOf(username) from memory. Lichess has no equivalent roster endpoint — there is no /api/player/titled in the spec — and does not need one for the row it is writing:

row.white_title = TitleFrom(parsed->headers, "WhiteTitle");

index_run already parses these headers (row.eco = EcoFrom(parsed->headers)), so for the game in hand the titles are free: no roster, no per-player lookup, no extra HTTP. IndexRun::Options::titles becomes per-platform rather than global, and TitleSource goes unimplemented for Lichess.

A persisted titles table — new migration

Done in #1531. V002__player_titles.sql plus PgTitleStore, with the ordered upsert and the never-store-empty rule below. Kept here for the reasoning.

Per-row titles cover the row being written. They do not cover anything that needs to know a title outside that moment, and TitleRoster is per-process memory that dies with the worker. Anything durable needs a table:

• Answering TitleOf(username) for a player whose games have not been indexed.
• Surviving restarts and being shared across worker instances.
• Correcting rows written before a player was titled — game_features.white_title is frozen at index time on both platforms today.
• Letting chess.com degrade better. Right now a failed roster refresh means complete = false and every player in the month writes ""; a last-known-title table lets it answer from the previous good roster instead of blanking.

Four things to get right, each of which is a wrong answer rather than an error if missed:

• Key on (platform, LOWER(username)). Usernames do not carry across platforms, so a Lichess observation must never answer a chess.com lookup. LOWER() matches the existing convention — game_features stores usernames as given and V012 indexes LOWER(white_username).
• observed_at is the game's played_at, not now(). Indexing is not chronological — a backfill of 2019 runs after 2026 is indexed. Keyed on insert time, an old month silently demotes a current GM.
• Upsert must be ordered, not last-write-wins:

  ON CONFLICT (platform, LOWER(username)) DO UPDATE
     SET title = EXCLUDED.title, observed_at = EXCLUDED.observed_at, source = EXCLUDED.source
   WHERE player_titles.observed_at < EXCLUDED.observed_at

• Never write an empty title. Absence of a WhiteTitle header is not evidence of being untitled — it is also every untitled player and every chess.com game. Storing "" lets an old game shadow a real title. Insert only when the header is present and non-empty.

Openings come free too

The same Lichess PGN carries [ECO "B00"] and [Opening "Goldsmith Defense"]. row.eco is already header-derived and platform-neutral; opening_name for Lichess comes from the Opening header rather than needing an equivalent of OpeningNameFromEcoUrl, which scrapes a name out of chess.com's ECOUrl slug.

Remaining shape mismatches feeding ArchivedGame

field	chess.com	Lichess
end_time	seconds since epoch	milliseconds (spec sets minimum: 1356998400070)
time_class	bullet/blitz/rapid/daily	perfType, with ultraBullet and classical — exclude_bullet tests time_class == "bullet" and would silently keep ultraBullet
white_result/black_result	per-side words (win, resigned…) mapped by result.h	a single status + winner, or the PGN Result header
eco_url	ECOUrl slug	not applicable — use the Opening header
url	given	derive from game id: https://lichess.org/{id}

There is no monthly archive endpoint

chess.com serves a month at a time (/pub/player/{u}/games/{year}/{month}), which is exactly FetchMonth(player, YearMonth). Lichess takes since/until instead. Mapping a YearMonth onto a half-open millisecond range is easy; the thing to preserve is the port's existing contract — a quiet month is an empty vector, NotFound means the archive is not there at all (#1360), so a range query over a nonexistent user must keep failing rather than reading as "indexed, no games".

Rate limiting is not what the roadmap says

Phase 8 says "20 req/s (more generous than chess.com)". The spec says the export stream is throttled at 20 games/second anonymous, 30 authenticated, 60 for your own games — games, not requests — plus "only make one request at a time" globally. That last one is real: concurrent requests during this investigation returned 429 {"error":"Please only run 1 request(s) at a time"}, and the cooldown outlasted a 75-second backoff. For a GM with tens of thousands of games this is a materially different budget, and it argues for an OAuth token in the worker's config.

The Retry-After support arriving with the pin bump is the right tool here if Lichess sends the header on a 429 — unverified, and worth checking with curl -i while implementing rather than assuming. If it does, retry_after_cap is the knob; if it does not, the cooldown is longer than max_backoff and the retry policy needs its own floor for this client.

Suggested slicing

1. Bump the opal-cpp pin to pick up @streaming + Retry-After. Done — bazel/opal.MODULE.bazel is at 6bb66f1.
2. player_titles table + ordered upsert, populated from the chess.com roster. Done in Keep titles in Postgres so a failed roster refresh stops untitling a month #1531.
3. Dispatch: ArchiveSource per platform in worker_main.cc/worker.cc, still chess.com-only. Done in Choose the archive per job's platform (#1527 slice 3) #1538.
4. lichess_cpp client + LichessArchive, with the normalization decisions above pinned by tests the way chess_com_archive_test.cc pins chess.com's.
5. Titles per-platform: PGN headers for Lichess feeding the same table, roster for chess.com.
6. The UI and API gate. Open canonicalPlatform, invert IndexRequestServiceTest.validationRejectsBadInput, and then everything the web needs for there to be two platforms rather than one:
   • A second <option> in IndexView.tsx. Genuinely one line — the <select>'s state already defaults to CHESS_COM and posts it.
   • Show which platform a row came from. platform reaches the client on both row types (types.ts:17,53, api.ts:104) and no component renders it — not GameResultsTable, GameTable, GameDetailPanel or GamesView. Harmless with one platform; at two it is a correctness problem in the same way player_titles being keyed on (platform, LOWER(username)) is. Usernames do not carry across sites, so two players sharing a handle produce rows that are identical on screen, and a browse page mixing both gives no way to tell which player you are looking at. The detail panel has it sharpest: it links out, and the link's host is the only thing on the page saying where the game is from.
   • Decide the display string. The column holds CHESS_COM; nobody wants to read that. A label map (CHESS_COM → chess.com, LICHESS → lichess) belongs wherever the <select>'s options live, so the form and the table agree.
   • Correct the web test fixtures. They claim platform: 'chess.com' for API rows (GameResultsTable.test.tsx:15,31, GameTable.test.tsx:16,31, QueryView.test.tsx:19, GamesView.test.tsx:36, GameDetailPanel.test.tsx:26) where production stores CHESS_COM. Invisible while nothing renders the field — and a rendering test written against them would assert the wrong string. The C++ worker had the identical problem; Choose the archive per job's platform (#1527 slice 3) #1538 corrected those.
   • ChessQL platform filters match nothing: rows store CHESS_COM, queries say chess.com #1539 wants deciding by here too: platform filters in ChessQL match nothing today, and lichess (no dot) will work by accident, which would leave chess.com as the one platform whose filter silently returns empty.

Open questions

• PGN or NDJSON? PGN gives the title and opening headers the indexer already parses; NDJSON gives structured perfType/status/clock without a PGN parse. The row needs both kinds of field, so one of them is being re-derived either way.
• Does exclude_bullet mean "bullet" or "bullet and faster"? It currently means the former by accident; Lichess makes the difference visible.
• Anonymous or tokened? Affects both throughput and whether the worker needs a new secret.
• Does learning a title later trigger a backfill of game_features.white_title/black_title, or only affect rows indexed after? The reanalysis queue (one_d4: rewrite the index worker in C++ on Disservin/chess-library #1389 phase 5) is the obvious vehicle if so.

Also worth doing while here: ROADMAP.md Phase 8 describes a Java LichessClient/GameFetcher alongside ChessClient, but fetching moved to the C++ worker in #1389. That section should be rewritten or deleted rather than left to mislead the next reader.

[aaylward]

Two things found while doing slice 3 (#1538), both filed rather than folded in here:

• ChessQL platform filters match nothing: rows store CHESS_COM, queries say chess.com #1539 — platform filters in ChessQL match nothing today. The column stores CHESS_COM (canonicalPlatform uppercases and turns dots into underscores), while platform compiles to LOWER(platform) IN (LOWER(?)). Case folding cannot close a gap that is punctuation. Confirmed against Postgres: chess.com → 0 rows, CHESS_COM → 1. The example in this issue, platform IN ["lichess", "chess.com"], is exactly the spelling that returns nothing. Invisible while there is one platform and nobody filters on it; slice 6 is what makes it the first thing a user tries. Note lichess has no dot and would work by accident, leaving chess.com as the one platform whose filter silently fails.

• 1d4 web never shows which platform a game came from #1540 — the web never shows which platform a game came from. platform reaches the client on both row types and no component renders it. This issue tracks one UI task, the second <option> in IndexView.tsx, and that is accurate — the select already defaults to CHESS_COM and posts it, so it really is one line. The display gap is the part it does not track, and it matters for the same reason player_titles is keyed on (platform, LOWER(username)): usernames do not carry across sites, so two players sharing a handle are indistinguishable in the games list.

Both want deciding around slice 6 rather than after it.

Also worth recording against the slicing above: slice 3 is done (#1538) and slice 2 landed as #1531. Slice 1's pin bump is in — bazel/opal.MODULE.bazel is at 6bb66f1, with @streaming and Retry-After. So slice 4 (lichess_cpp + LichessArchive) is next, and its four open questions are still open.

---

Generated by Claude Code