Check Whether Your Sitemap lastmod Dates Are Accurate with Python (Standard Library Only): Compare with dateModified
Where this fits
This is another script in a small standard-library series that reads a sitemap and checks one thing per page. An earlier post on validating an XML sitemap checks that <lastmod> is a valid W3C Datetime. Another on generating a sitemap for a static site writes one. This one asks whether the dates you publish match the pages they describe.
Most sitemap checkers ask one question about <lastmod>: is it a valid date? That is the easy half. The harder half is whether the date is true.
A static site generator that stamps every URL with the build date passes validation. So does a CMS that never updates <lastmod> after an edit. Both are wrong, and Google says it only uses <lastmod> when it is "consistently and verifiably" accurate.
This post builds a small auditor in Python using only the standard library (urllib, urllib.robotparser, xml.etree, html.parser, json, email.utils, datetime, csv, concurrent.futures). It reads your sitemap or sitemap index, fetches every page, and compares each <lastmod> with what the page says about itself: dateModified and datePublished in JSON-LD (including @graph), article:modified_time and similar meta tags, and the Last-Modified response header.
It flags invalid and future dates, sitemap dates older or newer than the page's own date, pages whose own dates disagree, and one <lastmod> value shared by a large share of URLs, which usually means a build stamp.
Google Search Central's Build and submit a sitemap page (showing "Last updated 2026-07-08 UTC") and the sitemaps.org protocol page were opened on 20 Nov 2026, and every statement attributed to them below comes from those pages.
The output in this post comes from a small fictional site ("Fernwood Ceramics") served on localhost, built for this article with planted problems. Appie free-tool descriptions are as of 20 Nov 2026, checked on each tool's page that day.
Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them.
What Google and the protocol actually say
Google. On its sitemap page: "Google ignores <priority> and <changefreq> values. Google uses the <lastmod> value if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate."
And: "The <lastmod> value should reflect the date and time of the last significant update to the page. For example, an update to the main content, the structured data, or links on the page is generally considered significant, however an update to the copyright date is not."
The protocol. sitemaps.org defines <lastmod> as "The date of last modification of the page," in W3C Datetime format, where you can "omit the time portion, if desired, and use YYYY-MM-DD." It adds: "the date must be set to the date the linked page was last modified, not when the sitemap is generated."
And: "this tag is separate from the If-Modified-Since (304) header the server can return, and search engines may use the information from both sources differently."
Two practical readings follow. First, a build-time stamp on every URL is exactly what the protocol says not to do. Second, the Last-Modified header is a separate signal. On many static hosts it reflects when the file was last written, which after a full rebuild is every file, so a header match proves little on its own. The script reports header mismatches as low severity for that reason.
What neither page defines is how Google verifies "the last modification of the page." The script uses the dates a page publishes about itself as the best available proxy. That is my choice, not Google's documented method.
How the audit works
- Read the sitemap (or every child sitemap in an index) and collect each
<loc>with its<lastmod>, if any. - Check robots.txt and skip disallowed URLs.
- Validate each
<lastmod>as W3C Datetime and flag dates more than a day in the future. - Fetch the page.
- Parse JSON-LD blocks and walk the whole tree for
dateModifiedanddatePublished. - Read
article:modified_time,og:updated_timeanditemprop="dateModified"meta tags. - Read the
Last-Modifiedheader. - Take the newest page modification date and compare it with
<lastmod>, with a tolerance (default one day) so time zones and same-day edits do not trigger noise. - After all pages are done, count how many URLs share each
<lastmod>value. - Write a CSV sorted by severity and print a summary.
What the script flags
invalid_lastmod: Severity: high; Meaning: Not a W3C Datetime (for example21/07/2026)future_lastmod: Severity: high; Meaning: More than a day after the time you ran the scriptlastmod_before_page_date: Severity: high; Meaning: The page says it was updated later than the sitemap does. The sitemap is stale, so the update may go unnoticedlastmod_before_published: Severity: high; Meaning: The sitemap date is earlier than the page's own publish datefetch_error: Severity: high; Meaning: The URL in the sitemap did not return a page (404, timeout)lastmod_after_page_date: Severity: medium; Meaning: The sitemap claims a newer change than the page shows. Often a build stamp or a trivial editpage_dates_conflict: Severity: medium; Meaning: JSON-LD and meta tags on the same page give different modification datesmodified_before_published: Severity: medium; Meaning: The page'sdateModifiedis earlier than itsdatePublishedshared_lastmod: Severity: medium; Meaning: One<lastmod>value appears on at least 25% of URLs (and at least 3). Usually a build stampheader_mismatch: Severity: low; Meaning:<lastmod>and theLast-Modifiedheader differ by more than the tolerancemissing_lastmod: Severity: low; Meaning: No<lastmod>. Optional in the protocol, but then Google has nothing to useno_page_date: Severity: info; Meaning: The page publishes no modification date, so<lastmod>cannot be checked against it
The script
Save this as lastmod_audit.py. It needs Python 3.9 or newer and nothing outside the standard library.
#!/usr/bin/env python3
"""Check whether sitemap <lastmod> dates match what each page says about itself.
Standard library only.
Reads a sitemap or sitemap index, fetches every page and compares <lastmod> with the page's own dates (JSON-LD dateModified/datePublished, meta article:modified_time) and the Last-Modified response header.
Writes a CSV report.
Usage: python3 lastmod_audit.py https://example.com/sitemap.xml --out lastmod_report.csv
"""
import argparse, csv, json, re, sys, time
import urllib.request, urllib.error, urllib.robotparser
import xml.etree.ElementTree as ET
from collections import Counter
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timedelta, timezone
from email.utils import parsedate_to_datetime
from html.parser import HTMLParser
from urllib.parse import urlsplit
UA = "lastmod-audit/1.0 (+standard-library script)"
NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
# W3C Datetime: YYYY, YYYY-MM, YYYY-MM-DD, or a full date with time and a zone.
W3C = re.compile(r"^\d{4}(-\d{2}(-\d{2}(T\d{2}:\d{2}(:\d{2}(\.\d+)?)?(Z|[+-]\d{2}:\d{2}))?)?)?$")
SEVERITY = {"invalid_lastmod": "high", "future_lastmod": "high", "lastmod_before_page_date": "high", "lastmod_before_published": "high", "fetch_error": "high", "lastmod_after_page_date": "medium", "page_dates_conflict": "medium", "modified_before_published": "medium", "shared_lastmod": "medium", "header_mismatch": "low", "missing_lastmod": "low", "no_page_date": "info",}
def fetch(url, timeout=15):
req = urllib.request.Request(url, headers={"User-Agent": UA})
with urllib.request.urlopen(req, timeout=timeout) as r:
return r.status, dict(r.headers), r.read()
def parse_w3c(value):
"""Return an aware UTC datetime for a W3C Datetime string, or None if invalid."""
v = (value or "").strip()
if not W3C.match(v):
return None
if len(v) == 4:
v += "-01-01"
elif len(v) == 7:
v += "-01"
v = v.replace("Z", "+00:00")
try:
dt = datetime.fromisoformat(v)
except ValueError:
return None
if dt.tzinfo is None:
# date only: treat as midnight UTC
dt = dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)
def parse_page_date(value):
"""Page dates are looser than W3C (ISO 8601 with or without zone). Try both."""
dt = parse_w3c(value)
if dt:
return dt
try:
dt = datetime.fromisoformat((value or "").strip().replace("Z", "+00:00"))
except ValueError:
return None
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)
def read_sitemap(url, seen=None):
"""Yield (loc, lastmod_text) from a sitemap or sitemap index."""
seen = seen or set()
if url in seen:
return
seen.add(url)
_, _, body = fetch(url)
root = ET.fromstring(body)
if root.tag == NS + "sitemapindex":
for sm in root.findall(NS + "sitemap"):
loc = sm.findtext(NS + "loc", "").strip()
if loc:
yield from read_sitemap(loc, seen)
else:
for u in root.findall(NS + "url"):
loc = u.findtext(NS + "loc", "").strip()
lm = u.findtext(NS + "lastmod")
yield loc, (lm.strip() if lm is not None else None)
class DateParser(HTMLParser):
"""Collect JSON-LD blocks and meta tags that carry modification dates."""
def __init__(self):
super().__init__()
self.in_ld = False
self.ld_chunks = []
self.meta = {}
def handle_starttag(self, tag, attrs):
a = dict(attrs)
if tag == "script" and (a.get("type") or "").lower() == "application/ld+json":
self.in_ld = True
self.ld_chunks.append("")
elif tag == "meta":
key = (a.get("property") or a.get("itemprop") or a.get("name") or "").lower()
if key in ("article:modified_time", "og:updated_time", "datemodified", "article:published_time", "datepublished"):
self.meta[key] = a.get("content", "")
def handle_endtag(self, tag):
if tag == "script":
self.in_ld = False
def handle_data(self, data):
if self.in_ld:
self.ld_chunks[-1] += data
def walk_ld(node, found):
"""Find dateModified and datePublished anywhere in a JSON-LD tree (including @graph)."""
if isinstance(node, dict):
for k in ("dateModified", "datePublished"):
if isinstance(node.get(k), str):
found.setdefault(k, []).append(node[k])
for v in node.values():
walk_ld(v, found)
elif isinstance(node, list):
for v in node:
walk_ld(v, found)
def page_dates(html):
p = DateParser()
p.feed(html)
found = {}
for chunk in p.ld_chunks:
try:
walk_ld(json.loads(chunk), found)
except json.JSONDecodeError:
continue
modified = {}
if found.get("dateModified"):
modified["jsonld"] = max(filter(None, map(parse_page_date, found["dateModified"])), default=None)
for key in ("article:modified_time", "og:updated_time", "datemodified"):
if p.meta.get(key):
modified["meta:" + key] = parse_page_date(p.meta[key])
published = None
pubs = found.get("datePublished", []) + [p.meta[k] for k in ("article:published_time", "datepublished") if p.meta.get(k)]
if pubs:
published = min(filter(None, map(parse_page_date, pubs)), default=None)
return {k: v for k, v in modified.items() if v}, published
def days(a, b):
return abs((a - b).total_seconds()) / 86400
def audit_url(loc, lm_text, tolerance, now, robots):
row = {"url": loc, "lastmod": lm_text or "", "page_modified": "", "page_published": "", "last_modified_header": "", "flags": [], "notes": []}
if robots and not robots.can_fetch(UA, loc):
row["notes"].append("skipped: disallowed by robots.txt")
return row
lm = None
if lm_text is None:
row["flags"].append("missing_lastmod")
else:
lm = parse_w3c(lm_text)
if lm is None:
row["flags"].append("invalid_lastmod")
row["notes"].append(f"not a W3C Datetime: {lm_text}")
elif lm > now + timedelta(days=1):
row["flags"].append("future_lastmod")
try:
status, headers, body = fetch(loc)
except urllib.error.HTTPError as e:
row["flags"].append("fetch_error")
row["notes"].append(f"HTTP {e.code}")
return row
except (urllib.error.URLError, TimeoutError) as e:
row["flags"].append("fetch_error")
row["notes"].append(str(e))
return row
hdr = headers.get("Last-Modified")
hdr_dt = None
if hdr:
try:
hdr_dt = parsedate_to_datetime(hdr).astimezone(timezone.utc)
row["last_modified_header"] = hdr_dt.strftime("%Y-%m-%d")
except (TypeError, ValueError):
pass
modified, published = page_dates(body.decode("utf-8", "replace"))
if published:
row["page_published"] = published.strftime("%Y-%m-%d")
if modified:
vals = sorted(modified.values())
newest = vals[-1]
row["page_modified"] = newest.strftime("%Y-%m-%d")
if days(vals[0], vals[-1]) > tolerance:
row["flags"].append("page_dates_conflict")
row["notes"].append("; ".join(f"{k}={v:%Y-%m-%d}" for k, v in modified.items()))
if published and newest < published - timedelta(days=tolerance):
row["flags"].append("modified_before_published")
if lm:
if lm < newest - timedelta(days=tolerance):
row["flags"].append("lastmod_before_page_date")
elif lm > newest + timedelta(days=tolerance):
row["flags"].append("lastmod_after_page_date")
else:
row["flags"].append("no_page_date")
if lm and published and lm < published - timedelta(days=tolerance):
row["flags"].append("lastmod_before_published")
if lm and hdr_dt and days(lm, hdr_dt) > tolerance:
row["flags"].append("header_mismatch")
return row
def main():
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument
Comments
No comments yet. Start the discussion.