# --- Search engines --- User-agent: Googlebot Disallow: User-agent: Bingbot Disallow: # --- Google Ads landing page checks --- User-agent: AdsBot-Google Disallow: User-agent: AdsBot-Google-Mobile Disallow: User-agent: AdsBot-Google-Mobile-Apps Disallow: # --- ChatGPT user agent --- User-agent: ChatGPT-User Disallow: # --- Social media link preview bots --- User-agent: facebookexternalhit Disallow: User-agent: Facebot Disallow: User-agent: Twitterbot Disallow: User-agent: LinkedInBot Disallow: # --- Block common AI/data scrapers --- User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: PerplexityBot Disallow: / User-agent: YouBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Bytespider Disallow: / User-agent: imagesiftBot Disallow: / # --- Block all other crawlers by default --- User-agent: * Disallow: /