Your Amazon scraper is returning mixed currencies and won't tell you

If you scrape Amazon through rotating residential proxies, your price column probably contains dollars, euros and zloty at the same time. Nothing in the markup says which is which, no error is raised, and the mixture changes from run to run. I spent a week measuring what an Amazon scraper actually gets back. Below are the numbers I logged, including one measurement I have not seen published anywhere: how fast a single IP burns out. Currency roulette Amazon decides your country from the IP address of the request and returns prices in that country's currency. It does not label the number. Same URL, same product, same day, two different exits: amazon.com from a European IP -> 8.63 amazon.com from an Asian IP -> 390037 Both are correct prices for the same Logitech mouse - €8.63 and ₫390,037. If your proxy pool rotates across countries and you strip the symbol to get a clean number, those two land in the same column with nothing to separate them. The symbol is in the markup, in a-offscreen : EUR 8.63 $12.99 Most parsers throw it away, because a currency symbol is inconvenient when you want a float. That is where the data dies. Fix: pin the proxy country to the marketplace you asked for, keep the symbol, resolve it to an ISO code, and check it against what that marketplace should have returned. Flag mismatches instead of shipping them quietly. "Just set the currency cookie" - I checked, and it lies The obvious objection is that you do not need proxies at all: Amazon honours an i18n-prefs cookie. It does. From a European IP: no cookie EUR 5.13 EUR 8.56 EUR 13.69 i18n-prefs=USD $5.99 $9.99 $15.99 Dollars, as requested. Now divide: 5.99 / 5.13 = 1.168 9.99 / 8.56 = 1.167 15.99 / 13.69 = 1.168 Three ratios, identical to three decimals. That is one exchange rate applied to the European catalogue - not United States pricing. For comparison, the same search through an actual US exit returned a different product mix entirely, at $12.99, $27.99, $11.30, $18.99. So the cookie gives you converted European prices wearing a dollar sign. If your question is "what does this cost in the US", the answer it hands you is wrong, and nothing in the response says so. (The ¤cy=USD query parameter, for what it is worth, is ignored outright - prices stayed in euros.) How fast one IP burns out - measured This is the part I could not find numbers for anywhere, so I measured it. A probe hitting Amazon hourly from one datacenter IP. Two search requests per check. That is about as gentle as automated access gets. | hour | fetch methods working (of 4) | records returned (normal: 28) | |---|---|---| | 05:02 | 2 | 28 | | 06:04 | 2 | 28 | | 07:01 | 2 | 14 | | 08:00 | 2 | 28 | | 09:03 | 2 | 14 | | 10:00 | 0 | 14 | | 12:04 | 2 | 28 | | 15:04 | 2 | 14 | | 17:05 | 0 | 14 | Fourteen requests over six hours, two blocked: a 14% block rate at two requests per hour. Individual blocks cleared by themselves within the hour. But the baseline drifted down across the day - what started as "two of four methods work" ended at zero. Separately: I once ran the probe twice within seven minutes. Both requests were blocked, and that IP did not recover for hours. Amazon appears to weigh request density far more heavily than volume. The practical consequence: a session per run is the wrong unit. You want a fresh exit per request, so density per address stays at one. The block that arrives as HTTP 200 Worth restating because my own detector missed it. Amazon's anti-bot interstitial comes back with status 200, Content-Type: text/html , and about 2,265 bytes: function triggerInterstitialChallenge() { var xhr = new XMLHttpRequest(); xhr.open("POST", "/_sec/verify?provider=interstitial", false); None of the classic strings are present - no api-****************@amazon.com , no Enter the characters you see . On amazon.fr the same thing arrives under HTTP 202, which most retry logic does not treat as a failure at all. Size is the reliable signal. A real Amazon search page is 300 KB to 1.6 MB. Anything under ~50 KB is not a page, whatever the status line says. Locale parsing, and two cases that are easy to miss 1,234.56 and 1.234,56 are the same amount. Deleting commas handles the first and destroys the second - 8,63 becomes 863 , silently, and 863 is a plausible price for something. The decimal separator is whichever of . and , comes last, and only when exactly two digits follow it: def to_number(raw): s = raw.replace(" ", "").replace(" ", "") dec = max(s.rfind("."), s.rfind(",")) if dec >= 0 and len(s) - dec - 1 == 2: return float(re.sub(r"[.,]", "", s[:dec]) + "." + s[dec+1:]) return float(re.sub(r"[.,]", "", s)) Two cases that cost me time while testing all 18 marketplaces: - The symbol is not always in front. Poland writes 123,45 zł , Sweden writes1 299 kr . A regex anchored to a leading symbol returns nothing on those markets. My first pass reported 20 products and 0 prices fromamazon.pl , and I assumed the parser had broken. It was matching the wrong shape. - Japan uses a different yen sign. amazon.co.jp writes￥ (U+FFE5, fullwidth), not¥ (U+00A5). A symbol table with the ordinary sign fails to recognise every price on that marketplace, silently. What ties these together None of them raise an error. The run succeeds, the dataset fills, the schema validates. You find out weeks later when someone asks why the German prices look a hundred times too small. So the check that actually caught things was not "did it fail" but "did the shape change". For each run I store the fraction of records where every required field is non-empty, and compare against the median of the last thirty runs. A parser that quietly stops finding prices shows up as price present in 8% of records instead of the usual 95%, long before anyone downstream notices. One thing I got wrong at first: alert on the second consecutive bad run, not the first. At a 14% block rate, alerting on every dip produced about ten messages a day about weather, and I stopped reading them - which defeats the point. I build this as an Apify actor - Amazon International Scraper, 18 marketplaces with the proxy country pinned and an ISO currency code on every price. But the traps above apply to whatever you build with; I would rather you not lose a week to them. Top comments (0)