Cloudflare Is Blocking Access to Your Site

If you're seeing this page, our crawler detected that Cloudflare is blocking bot access to your site. This affects not just our AI visibility analysis, but also AI assistants like ChatGPT, Claude, and Perplexity.

The Problem: Bot Protection Blocking AI Assistants

Cloudflare's security features are designed to protect your site from malicious bots. However, overly aggressive settings can also block legitimate bots, including:

• AI assistant crawlers (GPTBot, ClaudeBot, PerplexityBot)
• Our Spyglasses analysis bot
• Search engine crawlers
• SEO monitoring tools

When these bots try to access your site, Cloudflare returns a challenge page instead of your actual content. AI assistants cannot solve these challenges, which means they cannot access or cite your content.

Why This Matters

When AI assistants are blocked by Cloudflare:

• Your content cannot be cited in ChatGPT, Claude, or Perplexity responses
• Your brand cannot be recommended when users ask AI for solutions
• You're invisible in AI-powered search and recommendations
• Competitors who allow AI access gain a significant visibility advantage

In the age of AI-powered search, blocking AI crawlers is like blocking Google—you're removing yourself from a major discovery channel.

How to Check Your Settings

1. Verify Bot Protection Level

1. Log into your Cloudflare dashboard
2. Select your domain
3. Go to Security → Bots
4. Check your Bot Fight Mode setting

High or aggressive settings will block AI assistants.

2. Check WAF Rules

1. Go to Security → WAF
2. Review your Custom Rules and Rate Limiting Rules
3. Look for rules that might be blocking crawlers

Common problematic rules:

• Blocking all non-browser user agents
• Very low rate limits (< 10 requests per minute)
• Challenge or block on "Known Bots"

3. Review Firewall Rules

1. Go to Security → Firewall Rules
2. Look for rules that challenge or block based on:
   • User Agent patterns
   • ASN (Autonomous System Numbers)
   • Country blocking

How to Fix It

Option 1: Allow Verified AI Crawlers (Recommended)

Create a Custom Rule to allow verified AI crawlers:

1. Go to Security → WAF → Custom Rules
2. Click Create Rule
3. Set up the rule:

Rule name: Allow AI Assistants
Field: User Agent
Operator: contains
Value: GPTBot OR ClaudeBot OR PerplexityBot OR anthropic-ai OR SpyglassesBot
Action: Skip → All remaining custom rules

4. Save and deploy

This allows AI assistants while keeping your other security measures active.

Option 2: Adjust Bot Fight Mode

If you have Bot Fight Mode enabled:

1. Go to Security → Bots
2. Change from "Fight" to "Allow Verified Bots"
3. Save changes

This is less precise but easier to configure.

Option 3: Create Verified Bot Allowlist

For more control:

1. Go to Security → WAF → Custom Rules
2. Create a rule for verified bots:

Rule name: Verified Bot Allowlist
Expression: 
  cf.client.bot OR 
  cf.bot_management.verified_bot
Action: Allow

This uses Cloudflare's own bot verification system.

Recommended AI Crawler User Agents

Here are the user agents you should allow:

Full Allowlist Example

(http.user_agent contains "GPTBot") or 
(http.user_agent contains "ClaudeBot") or 
(http.user_agent contains "anthropic-ai") or 
(http.user_agent contains "PerplexityBot") or 
(http.user_agent contains "Google-Extended") or 
(http.user_agent contains "Bingbot") or
(http.user_agent contains "SpyglassesBot")

Testing Your Fix

After updating your Cloudflare settings:

1. Test with Curl

curl -A "GPTBot" https://yoursite.com

You should see your page content, not a Cloudflare challenge.

2. Request a New Report

Once you've updated your settings:

1. Return to Spyglasses AI Visibility Report
2. Request a new analysis
3. Our crawler will verify the changes

3. Check Cloudflare Analytics

Monitor your Cloudflare analytics to ensure AI crawlers can access your site:

1. Go to Analytics → Security
2. Look for requests from AI crawler user agents
3. Verify they're not being challenged or blocked

Option 4: Web Bot Auth (Cryptographic Verification)

Cloudflare supports a new standard called Web Bot Auth that lets bots prove their identity using cryptographic signatures. Instead of relying on User-Agent strings (which can be spoofed), Web Bot Auth uses HTTP Message Signatures (RFC 9421) with Ed25519 keys to verify that a request genuinely comes from a known bot.

How It Works

1. The bot operator generates an Ed25519 key pair
2. They host the public key at a well-known URL (/.well-known/http-message-signatures-directory)
3. They register with Cloudflare via the Bot Submission Form
4. Each request is signed with Signature, Signature-Input, and Signature-Agent headers
5. Cloudflare verifies the signature and allows verified bots through

Why This Matters

Web Bot Auth is becoming the standard for how legitimate bots authenticate with Cloudflare-protected sites. Major AI companies are already adopting it. If your site uses Cloudflare and you want to allow specific AI crawlers while keeping protection against malicious bots, ask your AI tool providers if they support Web Bot Auth.

Spyglasses Bot Verification

SpyglassesBot supports Web Bot Auth and signs its requests with an Ed25519 key. If SpyglassesBot is registered as a Cloudflare verified bot, it will be automatically allowed through your site's bot protection — no configuration required on your part. For details about our crawler, see About SpyglassesBot.

For Bot Operators

If you operate a bot or crawler that needs to access Cloudflare-protected sites, you can implement Web Bot Auth using Cloudflare's official libraries:

• Rust: web-bot-auth crate
• TypeScript: web-bot-auth npm package

For more details, see Cloudflare's Web Bot Auth documentation.

Best Practices

Balance Security and Visibility

• Do: Allow verified AI crawlers
• Do: Use rate limiting to prevent abuse
• Do: Monitor analytics for suspicious patterns
• Do: Consider enabling Web Bot Auth for cryptographic bot verification
• Don't: Block all bots indiscriminately
• Don't: Use overly aggressive challenge modes

Monitor AI Crawler Activity

Set up alerts for AI crawler traffic:

1. Use Cloudflare's Notifications
2. Track verified bot traffic patterns
3. Adjust rules if you see unusual activity

Rate Limiting Guidelines

If you use rate limiting:

• Allow at least 30 requests per minute for crawlers
• Use longer time windows (5-10 minutes)
• Whitelist verified bot user agents

Need More Help?

• Understanding AI Visibility
• Technical SEO for AI
• Is Your Site Visible to AI Assistants?
• Is robots.txt Blocking AI?
• Cloudflare Web Bot Auth Documentation
• Cloudflare Bot Documentation
• Contact Support