Cloudflare is the most common bot protection service we encounter. There are two ways to whitelist our crawler: by User-Agent (recommended) or by IP address.

Option 1: Whitelist by User-Agent (Recommended)

Log into your Cloudflare dashboard
Select the domain you want to whitelist
Navigate to Security → WAF → Custom Rules
Click Create rule
Set the rule name to Allow PixelHarborBot
In the expression builder, configure:
Field:User Agent
Operator:contains
Value:PixelHarborBot
Set the action to Skip and select all security features to skip
Click Deploy

The expression should look like: (http.user_agent contains "PixelHarborBot")

Option 2: Whitelist by IP Address

Go to Security → WAF → Tools
Under "IP Access Rules", click Add
Enter our IP address (contact hello@pixelharbor.io for current IPs)
Select Allow as the action
Add a note: PixelHarbor crawler
Click Add

Bot Fight Mode

If you have "Bot Fight Mode" or "Super Bot Fight Mode" enabled, it may override your WAF rules.

Go to Security → Bots
Check if "Bot Fight Mode" is enabled
If enabled, either disable it or ensure your WAF rule has higher priority
For "Super Bot Fight Mode" (paid), you can configure exceptions for specific User-Agents

Verify It's Working

After adding the whitelist rule:

Wait a few minutes for the rule to propagate
Return to Pixel Harbor and re-scan your website
If pages are still blocked, check the Cloudflare Firewall Events log for details

Cloudflare Resources

Still Having Trouble?

If you've followed these steps and are still seeing blocking errors, contact us for assistance.

Contact Support