How to scrape any site without getting banned
Most scrapers get blocked because they're polite. The sites you want to scrape are hostile by design, and treating them nicely just makes you easier to catch. Here's exactly how I bypass rate limits, bot detection, and IP bans without spending a fortune on proxies.
Most people get blocked on the first request. Not because their code is bad. Because they announced themselves.
The default headers from Python's requests library might as well say "hi I'm a bot please ban me." User-agent is python-requests/2.31.0. No cookies. No referrer. Straight GET to the data endpoint. You're cooked before you started.
The mindset shift that fixed everything for me
Stop thinking like a developer scraping a site. Start thinking like a browser visiting a site.
Real browsers don't just hit URLs. They load CSS. They execute JS. They wait. They move a mouse around. They have 47 tabs open and a stale cookie from 2019. Your scraper needs to smell like that.
In 2021, I was pulling pricing data for a SaaS competitor tracker I was building. Around 40k requests/day, nothing crazy. Got blocked by five different sites in a week. Spent three days debugging. The fix cost me $0 and took 20 minutes once I understood what was actually happening.
Here's exactly what I do now
First, steal a real browser fingerprint. Open Chrome DevTools, go to the Network tab, copy the full request headers from any request. Paste those into your scraper verbatim. Accept-Language, sec-ch-ua, sec-fetch-dest, all of it. Sites fingerprint you on the full header bundle, not just user-agent.
Second, fake human timing. Not time.sleep(1). That's a pattern. Do this instead:
import random, time
time.sleep(random.uniform(2.3, 8.7))Weird numbers. Humans aren't on the beat. Bots are.
Third, session management. One requests.Session() object per scraping thread. Let it accumulate cookies naturally. Hit the homepage first before you hit the data page. Sites watch for sessions that appear out of nowhere on deep URLs.
When you actually need to pay
Cloudflare's bot detection is genuinely hard to beat without Playwright or Puppeteer. If you're hitting a Cloudflare-protected site and you need JavaScript rendered anyway, just use Playwright with the playwright-stealth plugin. It patches like 15 detection vectors automatically. Don't fight it manually.
Residential proxies matter for maybe 10% of cases. Datacenter proxies get flagged by most serious sites now. If you genuinely need rotating residential IPs, Bright Data is expensive but it works. Oxylabs works too. Don't bother with the cheap ones on GitHub, they're either dead or honeypots.
That works at Google-scale anti-scraping setups. For 90% of sites you're actually trying to scrape, the header trick and random timing is enough.
The thing nobody says out loud
Most sites don't have serious bot detection. They have default bot detection from whatever WAF they're running on. Cloudflare free tier, basic nginx rate limiting, that's it. You're not fighting a team of engineers. You're fighting someone's 2019 config file.
Check if there's an official API first. Check if they have a public RSS feed. Check if a data provider already sells what you need. Scraping is the right answer less often than you think, and when it is the right answer, it's way easier than Twitter threads make it sound.
The real skill isn't bypassing detection. It's knowing when not to bother.