DEV Community

Google renders your page with last week's JavaScript: a 10-point checklist for assets crawlers can actually fetch

Automated technical SEO checks that catch indexing blocks and deploy regressions - seotto.com

When Google renders a JavaScript page, it doesn't just fetch your HTML. It fetches your scripts, stylesheets and API responses, runs them, and indexes the result. That pipeline has its own caching, its own timing and its own failure modes, and they're different from a user's browser.

Google's own documentation says its rendering service caches resources aggressively and may ignore your caching headers when deciding whether to fetch a file again. So a rendered page can be built from a mix of today's HTML and older JavaScript.

Most of the time that's harmless. Sometimes it means the rendered page is broken, blank or missing content, while everything looks perfect in your browser.

Here's the checklist I'd use to keep rendered pages in line with what users see. It's mostly about build and deploy hygiene, which is why it's a developer's problem more than an SEO one.

Filenames and caching

  1. Use content-hashed filenames for every JS and CSS file
    app.3f9c2a1b.js, not app.js. When the content changes, the URL changes, so no cache anywhere (Google's, a CDN's, a browser's) can serve the old file for the new page. Most bundlers do this by default for production builds. Check that it also applies to lazily loaded chunks, CSS split per route, and any script you add outside the bundler.

  2. Don't reuse a hashed filename for different content
    It sounds obvious, but non-deterministic builds and some custom pipelines produce the same filename for different content, or overwrite files in place. A hashed name is a promise that the file behind it never changes. If it does, caches will keep serving the old version.

  3. Keep old bundles available after a deploy
    This is the one that breaks rendering most often. A deploy replaces the whole assets directory, and the files from the previous build disappear. Any HTML that still references them, whether a page cached at your CDN, a page Google fetched an hour ago and renders later, or a user's open tab, now gets 404s for its scripts. Keep at least the previous few builds' assets online (many teams keep them for days or weeks). Static hosting platforms that keep immutable deployments do this for you; if you deploy by syncing a folder, stop deleting old files on sync.

  4. Serve the HTML with a short cache, and assets with a long one
    HTML should be revalidated often (Cache-Control: no-cache or a short max-age), so new deploys are picked up. Hashed assets can be cached for a year with immutable. The mistake to avoid is the reverse: long-cached HTML pointing at short-lived assets.

Access

  1. Don't block asset paths in robots.txt
    If robots.txt disallows /static/, /assets/, /_next/ or /api/, Google can fetch the HTML but not the code that builds the page. The result is rendered with missing content. Use the URL Inspection tool's "View crawled page" and "More info" panels in Search Console: blocked resources are listed there.

  2. Make sure the WAF and bot rules let Googlebot fetch assets
    Bot protection is often configured on the page routes and forgotten on asset and API routes, or the reverse. A challenge page served instead of main.js breaks rendering just as thoroughly as a 404. Check your WAF logs for Googlebot requests to asset paths that didn't get a 200, verifying that the requests really came from Google.

  3. Check the API calls the page makes during rendering
    Client-rendered content usually comes from API requests made after the page loads. If those require cookies, have strict CORS rules, are rate-limited per IP or are slow, the rendered page may have no content. Google's renderer won't wait forever. Server-side rendering for important content removes this whole category of problem.

Need a check that the next deploy didn't break crawler access? seotto.com watches for indexing blocks and asset regressions after you ship.

Verifying

  1. Compare raw HTML, rendered HTML and what users see
    For your key templates, check three versions: the raw HTML (curl), the rendered HTML from URL Inspection's live test, and the page in a normal browser. The title, main heading, body text, links and canonical tag should match in all three. A short script that extracts those from the first two and diffs them is worth putting in CI.

  2. Watch for asset 404s from crawlers in your logs
    Filter your access or CDN logs for requests from verified Googlebot to .js and .css files that returned 404. A spike right after a deploy is the clearest sign of the stale-bundle problem from item 3. It's a cheap alert to set up, and it catches the issue days before anything shows up in Search Console.

  3. Recheck after major deploys
    After a framework upgrade, a bundler change, or a CDN or hosting migration, run URL Inspection's live test on a handful of key pages, and check the rendered screenshot and HTML. These are the deploys that change filenames, chunking and asset paths all at once.

The short version

Hashed filenames, old bundles kept online, short-cached HTML, nothing blocking asset or API paths, and a regular comparison between raw and rendered HTML for your main templates. Spotting differences between the HTML crawlers fetch and the page users see is one of the checks we focus on at seotto.com, because rendering issues are invisible until you compare the two.

Related reading

  • How to find the URLs Googlebot crawls most but that never get search traffic (access logs + Search Console, in Python)
  • View source vs Inspect element: the SEO bug hiding between raw and rendered HTML
  • JavaScript SEO FAQ: 11 questions frontend devs ask (with straight answers)
  • CSR vs SSR vs SSG vs ISR: an SEO comparison without the framework marketing

Have you seen Google index a page rendered with old JavaScript? What gave it away: a weird snippet, missing content, or a 404 in the logs?

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.