Why some competitor pricing pages can't be scraped — and how to tell which

Ripplewatch tracks competitor pricing pages automatically, and every so often one comes back "Couldn't load." The instinctive next step is to assume it's a bot-detection problem and reach for a bigger hammer — rotate IPs, spoof a browser fingerprint, solve the CAPTCHA. We don't do that. Those techniques cross from "scraping resiliently" into evading a site's explicit access controls, which isn't something we're willing to build, regardless of how easy it would make our own product.

What we do instead is diagnose the actual block before deciding whether there's a legitimate fix. Requesting a page with our own identified bot user agent (RipplewatchBot, with a link back to us — not spoofed as a browser) and reading the response headers and body tells you which of a few very different things is actually happening.

Four outcomes, one curl request

• Cloudflare JS challenge — an interstitial "Just a moment..." page with a cf-ray header. This is solvable legitimately: a headless browser that executes JavaScript (not a fingerprint-spoofing one) clears it the same way a real visitor's browser does.
• Cloudflare WAF hard block — an explicit "Attention Required!" page. This is the site operator deciding automated traffic isn't welcome at all, full stop. We treat that as a boundary, not an obstacle.
• Akamai hard block — "Access Denied" with an x-akamai-cache-status header. Similar story to a Cloudflare WAF block: a deliberate access decision, not a technical hurdle to route around.
• False positive — the page actually returns a clean 200. In one case, a pricing page we'd flagged as blocked turned out to be a stale status from a transient failure, not a real block at all. Worth checking before assuming the worst.

The honest takeaway

Roughly half of scraping failures we've diagnosed this way are fixable with better tooling (a real headless browser instead of a plain HTTP request). The other half are a site operator's explicit choice, and no amount of engineering effort should override that. Distinguishing the two before writing code saves you from building something that either doesn't work or shouldn't exist.