Skip to content

Bot list: new bots from crawler-user-agents and ai.robots.txt, fresh crawler IP ranges - #56

Merged
czj merged 2 commits into
mainfrom
bot-sync
Nov 21, 2026
Merged

czj merged 2 commits into
mainfrom
bot-sync

Conversation

@github-actions

@github-actions github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

The bots that crawler-user-agents and ai.robots.txt added since the last sync, from script/sync_bots.rb, and the crawler IP ranges from script/sync_ip_ranges.rb. Review the answers that change, then merge.

Answers that change on the corpus
20328 user agents, 2592 differences, all expected
Keywords left out
skipped "sonic": matches mozilla/5.0 (linux; u; android 2.3.3; de-de; huawei sonic build/huaweiu8650) applewebkit/533.1 (khtm
Coverage
Category User agents browser Agentry With a category
academic_research 35 24 35 34
advertising_and_marketing 97 78 94 94
ai_agent 37 12 23 30
ai_assistant 38 20 37 37
ai_crawler 187 151 184 184
ai_search 45 36 44 43
archiver 68 50 68 68
browser_automation 23 2 2 21
feed_fetcher 92 40 91 91
http_library 103 68 102 102
monitoring_and_analytics 261 141 259 257
page_preview 136 110 135 133
scanner 108 46 106 106
search_engine_crawler 418 384 417 406
search_engine_optimization 685 481 674 671
Total 2333 1643 2271 2277

Crawler user agents added since 2024-01-01, tags browser-automation and scanner left out: 883, browser misses 354, Agentry 11.

@czj
czj enabled auto-merge (squash) November 21, 2026 11:47
@czj
czj merged commit 9ccddf1 into main Nov 21, 2026
19 checks passed
@czj
czj deleted the bot-sync branch November 21, 2026 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant