fix(link-preview): stop caching Reddit failures, keep good results forever - #61
Merged
Conversation
…rever Reddit rate-limits anonymous scraping to a small per-IP budget and then returns 403 "Blocked". The Worker fetched Reddit at render time, so one visitor loading a page with 20 Reddit links spent the entire budget at once, got the logged-out wall, and wrote it to KV. "Welcome to Reddit" is a valid-looking title, so it passed the only guard there was and got pinned for 7 days. Rendering and fetching were coupled when they should not be: the sites are read constantly, but new posts appear every few days. Worker: - Never fetch Reddit on a cache miss. Fall back to a title derived from the URL slug, which is always available and never wrong. That code was already written (titleFromRedditSlug) and had been dead since it landed. - Treat the wall titles as failures, on read and on write, so a blocked fetch can neither be stored nor served. - Cache good Reddit metadata with NO expiry. Post titles are immutable, so each URL is fetched once and kept. Other sites keep a 7-day TTL since their <title> can change. - Normalize the cache key. The warmer stripped the trailing slash while the Worker looked it up with one, so warmed entries were never read. Both sides now share one rule, verified to agree byte-for-byte. - Bump CACHE_VERSION to v3, which abandons the 165 existing v2 entries (many of them poisoned) instead of purging them. - Give provisional responses a 5-minute edge TTL, not a day, so the real title appears promptly once the warmer fills KV. Warmer (new scheduled workflow, replaces the job on deploy-worker): - Read each site's feed.json. The old job scanned a release artifact from doughatcher/blog and its last run pulled waccamaw_b3343a.zip and found zero Reddit URLs, so it had been a no-op. Feeds find 52. - Skip URLs that already have good metadata: fetched once, ever. - Refetch entries that hold a wall title, so poison heals. - Space requests out, and use the free Reddit OAuth API when REDDIT_CLIENT_ID/REDDIT_CLIENT_SECRET are set. - Never write a wall title. On failure write nothing and let the slug title stand until a later run succeeds. Note: production was running the unmerged branch feat/reddit-fallback-image-badge (its only delta was the v2 key bump, folded in here as v3).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
Reddit rate-limits anonymous scraping to a small per-IP budget, then returns
403 Blocked. The Worker fetched Reddit at render time, so a single visitor loading superterran.net's homepage (20 Reddit links) spent the whole budget at once, got the logged-out interstitial, and cached it."Welcome to Reddit"is a valid-looking title, so it passed the only guard there was (if (metadata.title)) and got pinned for 7 days.Rendering and fetching were coupled when they shouldn't be: the sites are read constantly, new posts appear every few days.
leaning.blue looked fine only because a few of its entries were cached before the block started. It was days from looking identical.
Diagnosis notes
link-previews.jsand share one Worker, so this was never site-specific.old.reddit.comreturns the correctog:titlefrom an unblocked IP for the same posts.waccamaw_b3343a.zipfromdoughatcher/blogreleases and found 0 Reddit URLs. Feeds find 52.feat/reddit-fallback-image-badge. Its only delta was thev2key bump, folded in here asv3. That branch can be closed after this merges.Changes
Worker
titleFromRedditSlug, previously dead code).CACHE_VERSION→v3, abandoning the 165 existing (largely poisoned) v2 entries rather than purging.Warmer — new scheduled workflow, replaces the job bolted onto
deploy-worker.ymlfeed.json.REDDIT_CLIENT_ID/REDDIT_CLIENT_SECRETare set.Verification
old./www., and query-string cases.Follow-up
Optional but recommended: add
REDDIT_CLIENT_ID/REDDIT_CLIENT_SECRET(free script app). Authenticated requests aren't subject to the anonymous block, and at this posting rate they'd use a rounding error of the quota.