Skip to content

fix(link-preview): stop caching Reddit failures, keep good results forever - #61

Merged
superterran merged 1 commit into
mainfrom
fix/reddit-preview-cache
Aug 2, 2026
Merged

fix(link-preview): stop caching Reddit failures, keep good results forever#61
superterran merged 1 commit into
mainfrom
fix/reddit-preview-cache

Conversation

@superterran

Copy link
Copy Markdown
Contributor

What was wrong

Reddit rate-limits anonymous scraping to a small per-IP budget, then returns 403 Blocked. The Worker fetched Reddit at render time, so a single visitor loading superterran.net's homepage (20 Reddit links) spent the whole budget at once, got the logged-out interstitial, and cached it. "Welcome to Reddit" is a valid-looking title, so it passed the only guard there was (if (metadata.title)) and got pinned for 7 days.

Rendering and fetching were coupled when they shouldn't be: the sites are read constantly, new posts appear every few days.

leaning.blue looked fine only because a few of its entries were cached before the block started. It was days from looking identical.

Diagnosis notes

  • All three dougie-theme sites serve byte-identical link-previews.js and share one Worker, so this was never site-specific.
  • Live Worker fetches returned the wall 5/5. old.reddit.com returns the correct og:title from an unblocked IP for the same posts.
  • The KV warmer had been a no-op: its last run (2026-05-30) downloaded waccamaw_b3343a.zip from doughatcher/blog releases and found 0 Reddit URLs. Feeds find 52.
  • Even had it worked, its cache keys stripped the trailing slash while the Worker looked them up with one, so nothing it wrote was ever read.
  • Production was running the unmerged branch feat/reddit-fallback-image-badge. Its only delta was the v2 key bump, folded in here as v3. That branch can be closed after this merges.

Changes

Worker

  • Never fetches Reddit on a cache miss; falls back to the URL-slug title (titleFromRedditSlug, previously dead code).
  • Wall titles rejected on both read and write, so a blocked fetch can't be stored or served.
  • Good Reddit metadata cached with no expiry — post titles are immutable, so fetch once and keep.
  • One shared key-normalization rule, verified to agree byte-for-byte with the warmer.
  • CACHE_VERSIONv3, abandoning the 165 existing (largely poisoned) v2 entries rather than purging.
  • Provisional responses get a 5-minute edge TTL instead of a day.

Warmer — new scheduled workflow, replaces the job bolted onto deploy-worker.yml

  • Reads each site's feed.json.
  • Skips URLs that already have good metadata: fetched once, ever.
  • Refetches wall-title entries so poison heals.
  • Spaces requests out; uses the free Reddit OAuth API if REDDIT_CLIENT_ID / REDDIT_CLIENT_SECRET are set.
  • On failure writes nothing, leaving the slug title until a later run succeeds.

Verification

  • Key normalization: JS and Python produce identical keys across trailing-slash, old./www., and query-string cases.
  • Feed collection: 52 unique Reddit URLs (vs 0 from the old artifact scan).
  • Dry run against live KV: 4/4 fetched and validated, 0 failures, no writes.
  • Both files syntax-clean.

Follow-up

Optional but recommended: add REDDIT_CLIENT_ID / REDDIT_CLIENT_SECRET (free script app). Authenticated requests aren't subject to the anonymous block, and at this posting rate they'd use a rounding error of the quota.

…rever

Reddit rate-limits anonymous scraping to a small per-IP budget and then
returns 403 "Blocked". The Worker fetched Reddit at render time, so one
visitor loading a page with 20 Reddit links spent the entire budget at
once, got the logged-out wall, and wrote it to KV. "Welcome to Reddit"
is a valid-looking title, so it passed the only guard there was and got
pinned for 7 days. Rendering and fetching were coupled when they should
not be: the sites are read constantly, but new posts appear every few days.

Worker:
- Never fetch Reddit on a cache miss. Fall back to a title derived from
  the URL slug, which is always available and never wrong. That code was
  already written (titleFromRedditSlug) and had been dead since it landed.
- Treat the wall titles as failures, on read and on write, so a blocked
  fetch can neither be stored nor served.
- Cache good Reddit metadata with NO expiry. Post titles are immutable,
  so each URL is fetched once and kept. Other sites keep a 7-day TTL
  since their <title> can change.
- Normalize the cache key. The warmer stripped the trailing slash while
  the Worker looked it up with one, so warmed entries were never read.
  Both sides now share one rule, verified to agree byte-for-byte.
- Bump CACHE_VERSION to v3, which abandons the 165 existing v2 entries
  (many of them poisoned) instead of purging them.
- Give provisional responses a 5-minute edge TTL, not a day, so the real
  title appears promptly once the warmer fills KV.

Warmer (new scheduled workflow, replaces the job on deploy-worker):
- Read each site's feed.json. The old job scanned a release artifact from
  doughatcher/blog and its last run pulled waccamaw_b3343a.zip and found
  zero Reddit URLs, so it had been a no-op. Feeds find 52.
- Skip URLs that already have good metadata: fetched once, ever.
- Refetch entries that hold a wall title, so poison heals.
- Space requests out, and use the free Reddit OAuth API when
  REDDIT_CLIENT_ID/REDDIT_CLIENT_SECRET are set.
- Never write a wall title. On failure write nothing and let the slug
  title stand until a later run succeeds.

Note: production was running the unmerged branch feat/reddit-fallback-image-badge
(its only delta was the v2 key bump, folded in here as v3).
@superterran
superterran merged commit 84de0ba into main Aug 2, 2026
1 check passed
@superterran
superterran deleted the fix/reddit-preview-cache branch August 2, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant