Websites Can't Block You If You Think Like Them
Most scraping tutorials teach you to hide. That's backwards. The sites that blocked me 40 times taught me the real lesson: don't disguise your bot, make your bot indistinguishable from a bored human at 2am. Here's what that actually looks like in code.
The $0 Scraper That Survived 6 Months Undetected
In 2021, I was consulting for a 12-person e-commerce startup that needed competitor pricing data across 8 retailer sites. Budget: zero. Timeline: yesterday. I built a scraper in a weekend using Playwright and some intuitions from my research background. It ran daily for six months, pulled ~4 million product pages, and never once got hard-blocked. Not because I was clever. Because I finally understood what anti-bot systems are actually detecting.
They're not detecting bots. They're detecting inhuman patterns. That distinction is everything.
What Cloudflare Is Actually Measuring
Cloudflare's Bot Management, PerimeterX, and Akamai's KSD all share a core detection philosophy: behavioral biometrics plus TLS fingerprinting plus request graph analysis. Let me unpack that.
TLS fingerprinting means your HTTPS handshake has a signature. Python's requests library using OpenSSL produces a different JA3 hash than Chrome 120 on macOS. Sites log this. If your JA3 hash matches zero real browsers but millions of scrapers, you're done before you send a single GET request.
Fix: use curl-impersonate (github.com/lwthiker/curl-impersonate) or Playwright with a real Chromium binary. Don't fake headers. Use actual Chrome.
Request graph analysis is sneakier. Real users don't request /products/shoes without first hitting /, loading 14 static assets, pausing 1.8 seconds, moving a mouse, then clicking. Your scraper that GETs only the product URLs looks like a laser. Real traffic looks like a drunk walk through a mall.
import asyncio, random
async def human_delay():
# Real human think-time is log-normal distributed
# Mean ~1.5s, heavy right tail
delay = random.lognormvariate(0.4, 0.5)
await asyncio.sleep(max(0.8, delay))That one function dropped my block rate from 60% to under 3% on a notoriously aggressive retail site. Log-normal, not uniform random. Humans aren't uniform.
Session Coherence Is the Thing Nobody Talks About
Here's what most scraping tutorials skip: cookies, localStorage, and session tokens need to persist and age. A session that hits 200 pages and never accumulates any tracking cookies looks wrong. Real browsers collect garbage — A/B test cookies, analytics tokens, ad pixels. Your scraper should too.
Don't clear cookies between requests. Let them pile up. Visit a homepage first, let the JavaScript run fully (use page.wait_for_load_state('networkidle') in Playwright), then navigate. You're not scraping a page. You're simulating a session.
Residential Proxies vs. Datacenter: The Math
Datacenter IPs from AWS us-east-1 are in every blocklist on the planet. Residential proxies (Oxylabs, Bright Data, Smartproxy) route through real ISP addresses. They cost ~$8-15/GB vs. $0.09/GB for datacenter. Worth it? Only if you're getting blocked. For sites with no serious protection, datacenter is fine. For anything running Cloudflare Enterprise, residential proxies aren't optional, they're the price of admission.
One more thing people get wrong: rotating proxies on every request is a red flag. Real users have one IP per session. Rotate per session, not per request.
The Only Rule That Matters
Every anti-bot system is a classifier trained on what bots historically looked like. You win by being an outlier in the bot distribution, not by being invisible. Look like the weirdest, slowest, most distracted human on the internet. That's your moat.