Cloudflare Workers Integration

Spyglasses integrates seamlessly with Cloudflare Workers to provide edge-based bot detection and AI referrer tracking. This integration is perfect for Webflow sites, e-commerce platforms, and any application that needs to understand and control bot traffic at the edge.

Quick Start with Template

🚀 New to Cloudflare Workers? Use our ready-to-deploy template:

📖 Complete example repository: github.com/orchestra-code/spyglasses-cloudflare-worker-example

✅ No file editing required - all configuration is done in the Cloudflare dashboard!

Manual Installation

For custom implementations, follow these steps:

1. Install the package:

   npm install @spyglasses/cloudflare-worker

   with yarn:

   yarn add @spyglasses/cloudflare-worker

   with pnpm:

   pnpm add @spyglasses/cloudflare-worker

2. Create your Worker: Create a new src/index.ts file:

   import worker from '@spyglasses/cloudflare-worker';
    
   export default worker;

3. Configure your Worker: Create a wrangler.toml file:

   name = "my-spyglasses-worker"
   main = "src/index.ts"
   compatibility_date = "2024-12-30"
   compatibility_flags = ["nodejs_compat"]
    
   [vars]
   ORIGIN_URL = "https://your-origin-server.com"
   SPYGLASSES_DEBUG = "false"

Configuration

Environment Variables

Setting Your API Key

For analytics and enhanced patterns, set your API key as a secret:

npx wrangler secret put SPYGLASSES_API_KEY
# Enter your API key when prompted

Deployment

Deploy your Worker to Cloudflare:

npx wrangler deploy

Advanced Configuration

For custom configuration, create your own worker:

import { createSpyglassesWorker, SpyglassesWorkerConfig } from '@spyglasses/cloudflare-worker';
 
const config: SpyglassesWorkerConfig = {
  debug: true,
  excludePaths: [
    '/api/',
    '/admin/',
    '/_next/',
    /\.(css|js|png|jpg|gif|svg|ico)$/i, // Regex patterns supported
  ],
  platformType: 'my-platform',
  blockingTimeout: 1000, // Max time to wait for logging blocked requests
  awaitBlockedLogging: false, // Don't block response on logging
};
 
export default {
  async fetch(request: Request, env: any, ctx: ExecutionContext): Promise<Response> {
    const spyglassesWorker = createSpyglassesWorker({
      ...config,
      apiKey: env.SPYGLASSES_API_KEY,
      debug: env.SPYGLASSES_DEBUG === 'true',
      originUrl: env.ORIGIN_URL,
    });
 
    return await spyglassesWorker.handleRequest(request, env, ctx);
  },
};

Use Cases

Webflow Sites

Perfect for Webflow sites that need bot protection:

[vars]
ORIGIN_URL = "https://your-site.webflow.io"
SPYGLASSES_DEBUG = "false"

E-commerce Protection

Protect product pages from aggressive scrapers:

const config: SpyglassesWorkerConfig = {
  excludePaths: [
    '/cart',
    '/checkout',
    '/account',
    '/admin',
  ],
  // Block AI model trainers but allow price comparison bots
};

Multiple Origins

Use different origins for different paths:

export default {
  async fetch(request: Request, env: any, ctx: ExecutionContext): Promise<Response> {
    const url = new URL(request.url);
    
    // Route to different origins based on path
    let originUrl = env.ORIGIN_URL;
    if (url.pathname.startsWith('/blog/')) {
      originUrl = env.BLOG_ORIGIN_URL;
    } else if (url.pathname.startsWith('/shop/')) {
      originUrl = env.SHOP_ORIGIN_URL;
    }
 
    const spyglassesWorker = createSpyglassesWorker({
      apiKey: env.SPYGLASSES_API_KEY,
      debug: env.SPYGLASSES_DEBUG === 'true',
      originUrl,
    });
 
    return await spyglassesWorker.handleRequest(request, env, ctx);
  },
};

Understanding Detection Results

The worker adds custom headers to help you understand what's happening:

X-Spyglasses-Processed: true
X-Spyglasses-Blocked: true (if blocked)
X-Spyglasses-Reason: bot|ai_referrer (if blocked)

Detection Types

• 🤖 Bot Detection: Identifies crawlers, scrapers, and AI trainers
• 🧠 AI Referrers: Tracks human visitors from AI platforms
• ✅ Legitimate Traffic: Passes through unmodified

Blocking Behavior

• Blocked bots receive a 403 Forbidden response
• AI referrers are logged but never blocked (they're human visitors)
• Regular visitors pass through normally

Debugging

Enable debug mode to see detailed logging:

[vars]
SPYGLASSES_DEBUG = "true"

Check your Worker logs:

npx wrangler tail

You'll see detailed information about:

• Which patterns matched
• Why requests were blocked or allowed
• Logging success/failure
• Performance metrics

Custom Routing

Set up custom routing in your Cloudflare dashboard:

1. Go to your domain in Cloudflare
2. Navigate to Workers Routes
3. Add a route pattern like *your-domain.com/*
4. Select your Spyglasses Worker

Performance

• Sub-millisecond detection using compiled regex patterns
• Minimal memory footprint with smart caching
• Background logging to avoid blocking responses
• Edge distribution for global low latency
• Bundle size: ~66KB total, ~13KB gzipped

Verifying Installation

After deploying your Worker, you can verify the installation by:

1. Check logs: Use npx wrangler tail to see real-time logs
2. Test with debug: Enable debug mode to see detailed detection info
3. Dashboard data: Check your Spyglasses dashboard for incoming data
4. Headers: Look for X-Spyglasses-Processed: true in responses

Troubleshooting

Worker Not Triggering

• Verify your route pattern in Cloudflare dashboard
• Check that the Worker is deployed to the correct zone
• Ensure your domain is using Cloudflare nameservers

Origin Connection Issues

• Verify ORIGIN_URL is set correctly
• Check that your origin server is accessible
• Ensure SSL certificates are properly configured

No Data in Dashboard

• Verify your API key is set correctly
• Check Worker logs for any errors
• Ensure your domain is properly configured in Spyglasses

Performance Issues

• Enable debug mode to identify bottlenecks
• Check your exclude paths configuration
• Verify your origin server response times

Security Best Practices

1. Use Secrets: Store your API key as a secret, not a variable

   npx wrangler secret put SPYGLASSES_API_KEY

2. Limit Scope: Use path exclusions to avoid processing sensitive routes

3. Monitor Logs: Enable debug mode in staging to verify behavior

4. Test Thoroughly: Use --dry-run to test deployments

Pricing

Cloudflare Workers usage is included in most Cloudflare plans:

• Free Plan: 100,000 requests/day
• Paid Plans: 10M+ requests/month included

Need help? Contact support@spyglasses.io