Google Trends will 429 you into the ground - here's how I got past it

The Token Dance

Trends has no "give me the data for this keyword" endpoint. You go through a handshake. First, explore. You tell it what you want and it hands back a set of widgets, each carrying a one-time token plus the request payload you'll replay:

GET https://trends.google.com/trends/api/explore?hl=en-US&tz=0&req={...}

req is a JSON object describing your comparison items - one keyword, or several to compare, each with a geo and time window:

{
  "comparisonItem": [
    {
      "keyword": "bitcoin",
      "geo": "US",
      "time": "today 12-m"
    }
  ],
  "category": 0,
  "property": ""
}

URL-encode that, drop it into req=, and you're off. The response comes back prefixed with )]}', - that's Google's anti-XSSI guard, sitting in front of every Trends API response so the body can't be executed as a script tag. Strip everything before the first { or your JSON parser chokes on line one.

Then, for each section you want, you call its data endpoint with that widget's token:

GET .../trends/api/widgetdata/multiline?...&token={token}&req={widget.request}

• multiline gives you interest over time
• comparedgeo gives you interest by region
• relatedsearches gives you related queries and topics

Same )]}', prefix on all of them.

Here's the part worth knowing: the token partially signs the request. You can edit some fields of the widget request - bump resolution for finer geo granularity - and it still validates. Touch the wrong field, like userConfig.userType, and you get a flat 401. So no, you can't just promote yourself to a logged-in user by rewriting a string. I tried. It 401s.

The Sections Google Won't Give a Robot

Related topics come back empty. Not sometimes - for every anonymous scraper, always. Your explore response gets tagged userType: USER_TYPE_SCRAPER, and for that user type Google serves an empty rankedList on the entity-typed related-topics section while related queries keep working fine. A logged-in browser gets USER_TYPE_LEGIT_USER and the full payload. Consent cookies don't move the needle. I spent a genuinely embarrassing amount of time convinced I'd broken my own parser before I diffed a browser session against my scraper and saw the userType flip.

So I'll say it plainly, because it's the thing I'm most confident about in this whole space: related topics for anonymous sessions is server-side gated and I don't think it's beatable without a real logged-in cookie. If a competing actor advertises full related-topics anonymously, test it on a fresh keyword before you trust the marketing.

City-level geo is walled off the same way. Country, region/state, and US metro (DMA) resolutions all return real data. Ask for CITY and automated sessions get hasData: false.

One Thing That Is Free: Trending Now

Trending Now skips the token dance entirely. There's a plain RSS feed - https://trends.google.com/trending/rss?geo=US. No token, works anonymously, ~10-20 stories with traffic estimates and the news articles behind each. If trending searches are all you need, you never have to fight the API at all.

The 429 Wall

This is the whole game. Google rate-limits per IP, and it does it hard - a naive loop dies inside a few dozen requests and then your IP is cold for hours. Everything below exists because of this one problem.

A session is one IP paired with one cookie jar. Warm it by hitting a Trends page first - any status code sets the cookie you need. Then reuse that pair for a batch. When a 429 lands, do not retry the same IP. Rotate to a fresh one.

This is where residential proxies earn their money and datacenter IPs don't: datacenter ranges are largely burned against Google and you'll see single-digit success rates on them. Between rotations, back off exponentially, and pace requests inside a session instead of firing them all at once.

The piece that saved my completeness numbers is per-section retry. If multiline succeeds but relatedsearches 429s, retry just that one section on a fresh session - don't throw away a keyword you already have three-quarters of.

async function withSession(fn, { maxRotations = 5 } = {}) {
  for (let i = 0; i < maxRotations; i++) {
    const session = newSession(); // fresh proxy IP + cookie jar
    try {
      await warmup(session); // sets the NID cookie
      return await fn(session);
    } catch (err) {
      if (err.response?.statusCode === 429) {
        await sleep(2 ** i * 1000); // backoff, then rotate
        continue;
      }
      throw err;
    }
  }
  throw new Error('exhausted sessions (429)');
}

A warning from experience: don't develop this against your home IP. Around 100 calls you'll hit a captcha-302 wall that takes hours to clear, and you'll lose the afternoon locked out of the site you're trying to debug. Point it at a small residential pool from the start.

Once the rotation and per-section retry were both in, I ran the actor across 8 test scenarios covering every section type and geo resolution. All 8 came back clean at 100% section completeness, averaging around 6 seconds per keyword end to end. That number only exists because of the per-section retry - without it, one unlucky 429 on the last section drags a "successful" keyword down to partial.

Values Are Relative, and No Scraper Changes That

Trends never gives you absolute search volume. Every series is normalized 0-100 against its own peak in the window you asked for. That's true on trends.google.com itself, so it's not something a scraper can fix or unlock - it's just what the data is. If you want an absolute-ish archive, the only path is to schedule snapshots over time and stitch them together, a daily reading you accumulate rather than a number you fetch.

The Box I Put It In

Token dance, session rotation, exponential backoff, per-section retry, the Trending RSS shortcut - all of it lives in an Apify actor: Google Trends Scraper. It exists because the 429 handling is genuinely tedious to get right and I got tired of rewriting it from scratch for every project. Nothing in it is proprietary magic, though - the flow above is the real thing, and you could build it yourself from this post.

And if you've actually pulled related topics out of an anonymous session, come tell me how. That's the one piece I've filed under unbeatable, and I would love to be wrong.