Why some competitor pricing pages can't be scraped, and how to tell which

Some competitor pricing pages can't be scraped because they sit behind deliberate bot-detection systems like Cloudflare or Akamai, a boundary the site operator chose, not a technical bug to route around. Diagnosing which of four outcomes actually happened (a solvable JS challenge, a hard WAF block, an Akamai block, or a false alarm) determines whether there's a legitimate fix or whether the block should just be respected.

Ripplewatch tracks competitor pricing pages automatically, and every so often one comes back "Couldn't load." The instinctive next move is to assume it's a bot-detection problem and reach for a bigger hammer: rotate IPs, spoof a browser fingerprint, solve the CAPTCHA. We don't do that. Those techniques cross the line from scraping resiliently into evading a site's explicit access controls, and that's not something we're willing to build, no matter how much easier it would make our own product.

What we do instead is diagnose the actual block before deciding whether there's a legitimate fix. Requesting a page with our own identified bot user agent (RipplewatchBot, with a link back to us, not spoofed as a browser) and reading the response headers and body tells you which of a few very different things is actually happening.

Pricing signals feed directly into our Momentum score, so a page that silently stops updating doesn't just mean stale data, it means a blind spot in whether a competitor is actually speeding up.

What are the four possible outcomes when a pricing page won't load?

• Cloudflare JS challenge: an interstitial "Just a moment..." page with a cf-ray header. This one's solvable legitimately: a headless browser that actually executes JavaScript (not a fingerprint-spoofing one) clears it the same way a real visitor's browser does.
• Cloudflare WAF hard block: an explicit "Attention Required!" page. The site operator has decided automated traffic isn't welcome, full stop. We treat that as a boundary, not an obstacle.
• Akamai hard block: "Access Denied" with an x-akamai-cache-status header. Same story as a Cloudflare WAF block: a deliberate access decision, not a technical hurdle to route around.
• False positive: the page actually returns a clean 200. One pricing page we'd flagged as blocked turned out to be a stale status from a transient failure, not a real block at all. Worth checking before assuming the worst.

Cloudflare and Akamai publish their own reasoning for these classifications: see Cloudflare's bot management documentation if you want the vendor's side of what counts as a bot.

So which pricing-page failures are actually fixable?

Roughly half the failures we've diagnosed this way are the site operator's explicit choice, the WAF and Akamai hard blocks, and no amount of engineering effort should override that. We didn't build headless-browser stealth to get past it either; that starts crossing back into evading real access controls, the same line we drew at the top of this post. Instead we built something else: when a pricing page is genuinely blocked, fall back to the most recent Wayback Machine snapshot instead of leaving the reading blank. It's not real-time, but it's a public archive of the same page, not an evasion technique, and it turns "couldn't load" into a still-useful, if slightly stale, reading. Careers pages get a parallel fix: when a company runs hiring through Greenhouse, Lever, or Ashby, their public job-board API is usually reachable directly, without ever touching the blocked page. A solvable JS challenge, the Cloudflare interstitial kind, is still the one outcome a real headless browser fixes properly; everything else gets respected, worked around through a public archive, or left honestly blank.