Sitemaps, IndexNow and Getting Crawled Sooner

IndexNow is a one-endpoint protocol: post a list of URLs and a key that is also served as a file on your domain, and participating engines fetch the file to confirm you own the site. Bing and Yandex participate. Google has said it does not. Both halves of that sentence matter before you spend an afternoon on it. What IndexNow is, and who participates The normal discovery model is pull: a crawler decides when to revisit you, based on how often you have changed before. IndexNow inverts it for one moment - you push a notification that a URL is new or changed, and the engine can schedule a crawl instead of waiting to guess. Participants include Bing, Yandex, Seznam, Naver and Yep, and submissions are shared between participating engines, so one POST reaches all of them. Several CDNs offer an integration that submits on your behalf when content changes. Google does not participate. It has said so publicly. That is the single most important fact about IndexNow and it is routinely omitted from tutorials selling it as an indexing accelerator. For most sites Google is the majority of search traffic, so the honest framing is that IndexNow costs one HTTP request per deploy and buys you a minority share of the market slightly sooner. The protocol in full It is genuinely small. One URL is a GET; a batch is a POST with a JSON body. # Single URL GET https://api.indexnow.org/indexnow?url=https://example.com/page&key= # Batch - up to 10,000 URLs per request POST https://api.indexnow.org/indexnow Content-Type: application/json; charset=utf-8 { "host": "example.com", "key": "cbe9a227bd53cf88e15c7a8c69923393", "keyLocation": "https://example.com/cbe9a227bd53cf88e15c7a8c69923393.txt", "urlList": [ "https://example.com/learn/one", "https://example.com/learn/two" ] } | Status | Description | |---|---| | 200 OK | Accepted. It does not mean crawled, and it certainly does not mean indexed. | | 202 Accepted | Accepted, key validation pending. Treat exactly like 200 - this is the normal response on a first submission and is not an error. | | 400 Bad Request | The body is malformed. Usually a missing field or the wrong content type. | | 403 Forbidden | The key file could not be validated. Either the file is genuinely wrong, or it was not reachable at the moment the engine checked - see the race below. | | 422 Unprocessable Entity | The URLs do not all belong to the host you declared, or the key does not match the file. The most common cause is a mixed batch containing a subdomain. | | 429 Too Many Requests | You are submitting too often. This is the response that tells you your submission strategy is wrong. | The key file, which is deliberately public Generate a hexadecimal key of 8 to 128 characters, and serve it as a plain-text file at the root of your domain, named after the key, whose entire contents are the key. https://example.com/cbe9a227bd53cf88e15c7a8c69923393.txt cbe9a227bd53cf88e15c7a8c69923393 # Verify it before you rely on it - including the content type. curl -sSI https://example.com/cbe9a227bd53cf88e15c7a8c69923393.txt curl -sS https://example.com/cbe9a227bd53cf88e15c7a8c69923393.txt This key is not a secret and must not be treated as one. The entire verification mechanism is that anybody can fetch that URL and read it; its purpose is to prove that whoever is submitting URLs also controls the web root. It is one of the few values in a repository that is safe to commit, and it is worth saying out loud once, because a well-trained team will otherwise put it in a secrets manager and then wonder why the fetch fails. If your framework serves static files from a public directory, drop it there. In Next.js that is public/ , and the file is then served verbatim at the root. A working implementation The version below runs at the end of a deploy. It builds the URL list from the same source the sitemap is built from, so the two cannot disagree, checks the key file is actually serving, submits in batches, and records what it has announced. #!/usr/bin/env node // indexnow.mjs - announce new URLs at the end of a deploy. // node indexnow.mjs # dry run: show what would be sent // node indexnow.mjs --send // node indexnow.mjs --send --all # resubmit everything, rarely right import { readFileSync, writeFileSync, existsSync, mkdirSync } from "node:fs"; import { dirname } from "node:path"; const KEY = "cbe9a227bd53cf88e15c7a8c69923393"; const HOST = (process.env.APP_URL ?? "https://example.com").replace(//$/, ""); const SEND = process.argv.includes("--send"); const ALL = process.argv.includes("--all"); // The state file must live OUTSIDE the deployed application directory, or a // deploy that swaps that directory deletes it - and the next run resubmits // every URL you have. That failure is silent, because resubmission returns 200. const STATE = process.env.INDEXNOW_STATE ?? "/var/lib/example/indexnow.json"; // Exactly the URLs a visitor can reach, which is exactly what is in the // sitemap. Build it from the same module the sitemap uses. const urls = buildLiveUrlList(HOST); // your project's function const seen = ALL || !existsSync(STATE) ? new Set() : new Set(JSON.parse(readFileSync(STATE, "utf8"))); const fresh = urls.filter((u) => !seen.has(u)); console.log(live ${urls.length} already told ${urls.length - fresh.length} to submit ${fresh.length}); if (fresh.length === 0) process.exit(0); if (!SEND) { fresh.slice(0, 10).forEach((u) => console.log(" " + u)); process.exit(0); } // Confirm the key file is reachable BEFORE submitting anything. See below. async function keyServing() { const r = await fetch(${HOST}/${KEY}.txt).catch(() => null); return r && r.ok && (await r.text()).trim() === KEY; } if (!(await keyServing())) { console.log("key file not serving yet; waiting 10s"); await new Promise((r) => setTimeout(r, 10_000)); if (!(await keyServing())) { console.log("still not serving - skipping. Nothing is broken."); process.exit(0); } } const BATCH = 500; // the protocol allows 10,000; smaller keeps failures cheap let submitted = 0; for (let i = 0; i 0) { mkdirSync(dirname(STATE), { recursive: true }); writeFileSync(STATE, JSON.stringify([...seen, ...fresh.slice(0, submitted)])); console.log(submitted ${submitted}); } Why submitting everything every time is wrong The obvious implementation posts your whole sitemap on every deploy. It returns 200, so nothing appears to be wrong, and it is the behaviour IndexNow’s own guidance warns against: an engine that concludes you are noisy simply stops acting on your submissions, and there is no notification when that happens. So the script submits only what is new, tracked in a state file. Two details make that work in production and both were learned the hard way. - The state file must survive a deploy. If it lives inside the application directory and your deploy swaps that directory, the file disappears and the next run submits everything. Put it beside your data volume, not beside your code. - Environment variables from your process manager are not available to a deploy script. A script that reads a path from an environment file the web server gets but the shell does not will silently fall back to a default path - which is exactly how a state file ends up inside the application directory in the first place. Read the environment file explicitly, or pass the path in. - A corrupt state file must not fail a deploy. Catch the parse error and treat it as empty. The worst case is one redundant submission; the alternative is a failed release. The key-file race that returns 403 A specific failure worth knowing because it looks exactly like a broken key and is not. If you submit immediately after restarting your service, the engine fetches your key file within seconds - before the service is answering - gets a connection error, and returns 403 for the whole batch. The key is correct. The file is correct. It is a race. The fix is the probe in the script above: fetch your own key file first, and if it is not serving, wait and try once more, then skip the submission entirely rather than reporting a failure. A skipped announcement costs nothing - the URLs are still in your sitemap and the next deploy announces them. Sitemaps, and what lastmod is for IndexNow is a supplement to a sitemap, never a replacement. The sitemap is the durable statement of what exists; IndexNow is a transient notification that something changed. - List only URLs that return 200. A sitemap containing a URL that 404s or redirects is a contradiction you will see reported back to you. - Never list a URL your robots.txt disallows. Same reason. - Use lastmod only if it is true. Search operators have said they use it where a site is consistently accurate; a date generated at render time destroys that consistency permanently. If you do not have a real modification date, omit the field. - Skip changefreq andpriority . Google has said for years that it ignores both. They are two more numbers to keep honest for no return. - Reference the sitemap from robots.txt, and add the line only once the file actually exists - pointing a crawler at a 404 is worse than staying quiet, because it is the one URL in that file a crawler is guaranteed to fetch. - Exclude scheduled-but-unpublished pages. If your content is released on a cadence, the sitemap must contain only what is live today, or you are announcing URLs that 404. What none of this speeds up - Google. It does not participate. Your options there are a correct sitemap, internal links, and the URL Inspection tool in Search Console for the occasional individual URL, which has a small daily quota and is not an automation target. - Indexing. Submission is a hint to crawl. Whether the page is indexed afterwards is a separate decision about whether it is worth indexing, and no protocol influences that. - Ranking. Nothing here is a ranking input. A page crawled sooner ranks exactly as well as the same page crawled later. - AI assistant retrieval. No assistant operator documents consuming IndexNow. The cr