crawler-blocking

Here is 1 public repository matching this topic...

AI crawler user agent reference + nginx/Cloudflare/Vercel block configs. Stop GPTBot, ClaudeBot, MetaAI from spiking your infrastructure costs.

nginx security cloudflare robots-txt gpt-bot claude-bot ai-crawlers crawler-blocking

To associate your repository with the crawler-blocking topic, visit your repo's landing page and select "manage topics."