How to scrape data from an app without getting banned

Best Practices for Scraping

We wrote a few lines of Python, sent some requests, and our scraping pipeline seemed to work flawlessly. But once we started scraping at scale, things changed quickly. Suddenly we were getting errors, running into CAPTCHAs, or receiving empty pages instead of the data we needed. That's when we realised scraping is a lot harder than it looks. So, after loads of hardship and failures, we want to share our playbook on the best practices for scraping.

Essential fundamentals

Most bans happen because people ignore the basics. Before reaching for exotic tools, get these right.

Check robots.txt and the Terms of Service

Nearly every site publishes a robots.txt at its root (e.g. example.com/robots.txt) telling automated clients which paths are off-limits. Ignoring it won't just get you blocked faster; depending on the jurisdiction and the data, it can carry legal risk. If a site offers an official API, use it. An API is a sanctioned, structured door into the data, and it will always be more stable than scraping the HTML that surrounds it.

Slow down

The single most common reason people get flagged is hammering a server with requests far faster than any human could browse. Add delays between requests, and make them variable rather than a fixed one-second tick, since perfectly regular timing is itself a bot signal. Something in the range of a few seconds, randomized, is a reasonable starting point for polite scraping.

Handle errors gracefully

When you get a 429 Too Many Requests or a 403, do not immediately retry in a tight loop. That's the fastest way to escalate a soft throttle into a hard ban. Back off, wait, and reduce your rate. Respect the signal the server is sending you.

Rotate your identity where appropriate

This means rotating IP addresses (via proxies) and varying your User-Agent and headers so you don't look like a single machine making thousands of identical requests. Residential and mobile proxies are harder to flag than datacenter IPs, which most anti-bot systems keep on known blocklists.

These fundamentals will carry you a long way on lightly-defended sites. The trouble is that a growing share of the web is not lightly defended.

Why browser-based scrapers get blocked

Here's the part that trips up most people. You can follow every rule above, rotate perfect residential proxies, set a flawless Chrome User-Agent, and still get blocked before your scraper receives a single line of HTML. To understand why, you have to understand that modern anti-bot systems (Cloudflare, DataDome, Akamai, Kasada, PerimeterX/HUMAN) don't rely on any one signal. They combine several layers, and a scraper has to pass all of them simultaneously.

Layer 1: TLS and network fingerprinting

Before a website even sees your request, it can already learn a lot about the software making it. When your scraper connects, it goes through a security step called a TLS handshake. During this process, it sends technical details that act like a fingerprint. The problem is that common scraping tools like Python's requests or Node's axios have fingerprints that don't look like real browsers. Websites compare this fingerprint with the browser you're claiming to be. So if your request says it's coming from Chrome, but the fingerprint clearly looks like a Python script, that's a strong sign you're a bot and your request may be blocked before the page even loads.

Layer 2: Browser and device fingerprinting

Even if your request gets past the first checks, websites have another way to detect bots. They run JavaScript in your browser to collect information about your device and browser. This includes things like your screen size, graphics card, installed fonts, browser settings, and many other small details that together create a unique browser fingerprint. Headless browsers, which are commonly used for scraping, often stand out because they don't behave exactly like a real browser. For example, they may reveal that they're running in headless mode or expose properties that websites use to identify automation. While there are tools that try to hide these signs, modern anti-bot systems look at dozens of signals together. Fixing one or two isn't usually enough, because real browsers have consistent, natural-looking fingerprints that are hard to fake completely.

Layer 3: Behavioral analysis

Even if your browser looks real, it also has to behave like a real person. Websites don't just look at what browser you're using, they also watch how you interact with the page. They can track things like how you move your mouse, how you scroll, how fast you type, where you click, and how long you stay on a page. Bots often behave differently from people. They move the mouse in perfectly straight lines, click almost instantly, and type at a perfectly consistent speed. Real users are much less predictable: they pause, make mistakes, scroll at different speeds, and interact naturally. Modern anti-bot systems use machine learning to spot these patterns, so even if one signal looks normal, several suspicious signals together can reveal that it's a bot.

This is why building a reliable browser-based scraper is so difficult. You have to make your browser look real, behave like a real person, and constantly keep up with new detection techniques. That means managing browser fingerprints, proxies, and human-like behaviour all while websites keep improving their defences. It quickly becomes an ongoing battle that's both complex and expensive.

The alternative: scraping through the mobile app layer

Here's a shift in perspective that a lot of scrapers miss. Almost all the detection machinery above was built to catch browser automation on the web. But an enormous amount of the world's data now lives in mobile apps, often data that isn't even exposed on the website. And the detection story on mobile is very different.

This is where Mobilerun comes into the picture. Instead of driving a headless browser, Mobilerun runs an AI agent on real Android and iOS devices in the cloud. The agent sees the screen, reads the app's UI accessibility tree, and interacts the way a person would: tapping, swiping, scrolling, and typing. It extracts structured data directly from the app.

Why this avoids so much of the detection problem

The reason browser scrapers get caught is that they present a synthetic environment and detection systems are purpose-built to spot exactly those artifacts. Mobilerun changes the situation on several fronts:

• It's a real device, not an emulator or headless browser. Real phones don't have many of the giveaways that browser-based bots do. There's no navigator.webdriver flag, no headless browser identifier, and no browser fingerprint that doesn't match the device. Everything from the hardware and operating system to the sensors and installed apps is exactly what you'd expect from a real person's phone. That's a big advantage because anti-bot systems are looking for consistency. Real devices naturally provide it, while emulators and virtual devices often expose artificial hardware details or other subtle signs that they're not genuine. A physical phone simply behaves like a normal phone, making it much harder to distinguish from a real user.

• Real sensor and network characteristics. Physical phones produce noisy accelerometer and gyroscope data, and carrier-network and latency patterns that are genuinely hard to fake. Some platforms cross-check sensor telemetry against UI gestures specifically to catch synthetic control; on a real device driven through the OS, that telemetry is authentic.

• Human-like interaction by construction. Because the agent acts through the device's normal input layer, its behavior sits much closer to a real user's than a script firing thousands of identical HTTP requests ever could. By starting from a genuine device and a genuine interaction model, you're no longer fighting the entire browser-detection stack - you've largely stepped around it.

Essentially, the mobile approach shines anywhere the valuable data lives inside an app, or where the mobile app is simply less defended than the hardened website. Common examples:

• Pricing and availability from travel, food-delivery, ride-hailing, or grocery apps whose sites are heavily protected but whose apps are the primary interface.
• Listings data from marketplaces and real-estate apps.
• Social and review data from platforms where much of the activity is mobile-first.
• Any app with no public API at all, which is the majority of them, where the UI is the only way in.

Putting it together

If you're scraping a simple website, keep it simple: respect robots.txt, use the official API if one exists, limit how fast you send requests, vary your timing, and handle errors gracefully. That's enough for many websites.

If you're scraping a site with strong anti-bot protection, it's much harder. Modern websites check your network fingerprint, browser fingerprint, and behaviour to detect bots. Getting past all of that with headless browsers takes constant effort.

If the data is available in a mobile app, using a real phone can be a much easier approach. Platforms like Mobilerun let you automate real Android or iOS devices, so you avoid many of the browser-specific detection techniques. You're not pretending to be a real user - you're using a real device.