Running SEO Checks in a CI Pipeline: What to Automate

The deploy went green. Every test passed. Two days later, someone in marketing posted a Search Console screenshot in Slack: pages were dropping out of the index.

A template change had shipped a noindex tag that was only ever meant for staging. Nothing in the pipeline caught it, because nothing in the pipeline was looking. Unit tests check that your code behaves. They don't check the HTML that Googlebot actually receives.

This post is a practical plan for running SEO checks in a CI pipeline. It covers which checks to automate, where each one should run, and which ones to leave to humans. Every script below was tested before publishing, and you can drop them into a GitHub Actions setup today. If you've wanted to start automated SEO testing but weren't sure where the line was, this is it.

Why SEO Breaks Quietly in Deploys

SEO regressions are sneaky because nothing errors out. The page loads, the design looks right, and users can click around normally. Meanwhile:

• A staging noindex meta tag or X-Robots-Tag header goes to production.
• robots.txt ships with Disallow: / copied from a preview environment.
• Canonical tags point at the staging host.
• A layout refactor wipes page-specific titles and meta descriptions.
• Routes get renamed with no redirects, so old URLs return 404.
• A JSON-LD block breaks because someone added a trailing comma.
• The sitemap still lists URLs that no longer exist.

Each of these takes seconds to detect with the right check and weeks to notice from traffic graphs. That gap is exactly what CI is good at closing.

The Rule for What to Automate

Not every SEO check belongs in a pipeline. Use one simple rule: Automate checks that are deterministic, fast and binary. A tag is present or it isn't. A URL returns 200 or it doesn't. These make good gates.

Monitor checks that are noisy or depend on real users. Lab performance scores vary between runs, and Core Web Vitals field data comes from real visitors over weeks. Track these, but don't block merges on them.

Keep judgment human. Content quality, search intent, keyword targeting and title copywriting need a person.

The Four Layers at a Glance

• Pull request gate: lint the built HTML for missing or broken tags. Runs in seconds.
• Lighthouse sample: audit a handful of key templates for SEO and performance budgets.
• Post-deploy check: test the live server for robots rules, headers, redirects and sitemap health.
• Scheduled monitoring: catch slow drift such as broken external links and field performance.

To decide which individual rules to codify, start from a reference list rather than a blank page. SEO Site Checker's 70 SEO checks explained groups on-page, performance, security and advanced checks with plain-language explanations, which makes it easy to pick the deterministic ones for your linter.

Layer 1: Gate Pull Requests on the Built HTML

The most important habit is to test the build output, not your source templates. Your framework, your CMS data and your layout components only come together in the final HTML, and that HTML is what search engines see. Point the checks at dist/, out/ or whatever folder your static export produces.

Checks That Belong Here

• Exactly one non-empty <title>, unique across the site, with a warning above 60 characters
• A meta description, with a warning above 160 characters
• A canonical tag that is absolute and points to the production host
• No noindex in meta robots, except on pages you deliberately allow
• One <h1> per page (warning only, since some layouts have good reasons)
• A lang attribute on <html>
• An alt attribute on every image (an empty alt="" is fine for decorative images)
• Basic Open Graph tags
• JSON-LD that parses and includes @context and @type, including inside @graph

A Small Checker Script

This script uses cheerio to parse every HTML file in the build folder. It separates errors, which fail the build, from warnings, which are only printed. Install it with npm install --save-dev cheerio.

// scripts/seo-check.mjs
// Usage: node scripts/seo-check.mjs <build-dir> <production-origin>
import { readdir , readFile } from ' node:fs/promises ' ;
import path from ' node:path ' ;
import * as cheerio from ' cheerio ' ;
const [ dir = ' dist ' , origin = ' https://example.com ' ] = process . argv . slice ( 2 );
const NOINDEX_ALLOWED = new Set ([ ' 404.html ' , ' search/index.html ' ]);
const errors = [];
const warnings = [];
const titles = new Map ();
async function findHtml ( folder ) {
  const files = [];
  for ( const entry of await readdir ( folder , { withFileTypes : true })) {
    const full = path . join ( folder , entry . name );
    if ( entry . isDirectory ()) files . push (...( await findHtml ( full )));
    else if ( entry . name . endsWith ( ' .html ' )) files . push ( full );
  }
  return files ;
}
function checkPage ( page , html ) {
  const $ = cheerio . load ( html );
  const error = ( msg ) => errors . push ( ` ${ page } : ${ msg } ` );
  const warn = ( msg ) => warnings . push ( ` ${ page } : ${ msg } ` );
  // Indexing
  const robots = ( $ ( ' meta[name="robots"] ' ). attr ( ' content ' ) ?? '' ). toLowerCase ();
  if ( robots . includes ( ' noindex ' ) && ! NOINDEX_ALLOWED . has ( page )) error ( ' meta robots contains noindex ' );
  // Title
  const titleTags = $ ( ' head > title ' );
  const title = titleTags . first (). text (). trim ();
  if ( titleTags . length !== 1 || ! title ) {
    error ( `expected one non-empty <title>, found ${ titleTags . length } ` );
  } else {
    if ( title . length > 60 ) warn ( `title is ${ title . length } characters` );
    titles . set ( title , [...( titles . get ( title ) ?? []), page ]);
  }
  // Meta description
  const description = $ ( ' meta[name="description"] ' ). attr ( ' content ' )?. trim ();
  if ( ! description ) error ( ' missing meta description ' );
  else if ( description . length > 160 ) warn ( `meta description is ${ description . length } characters` );
  // Canonical
  const canonical = $ ( ' link[rel="canonical"] ' ). attr ( ' href ' );
  if ( ! canonical ) {
    error ( ' missing canonical tag ' );
  } else {
    try {
      const url = new URL ( canonical );
      if ( url . origin !== origin ) error ( `canonical points to ${ url . origin } ` );
    } catch { error ( `canonical is not an absolute URL: ${ canonical } ` ); }
  }
  // Headings, language, images, social
  const h1Count = $ ( ' h1 ' ). length ;
  if ( h1Count !== 1 ) warn ( `found ${ h1Count } <h1> elements` );
  if ( ! $ ( ' html ' ). attr ( ' lang ' )) warn ( ' missing lang attribute on <html> ' );
  $ ( ' img:not([alt]) ' ). each (( _ , img ) => warn ( `image without alt: ${ $ ( img ). attr ( ' src ' )} ` ));
  if ( ! $ ( ' meta[property="og:title"] ' ). length ) warn ( ' missing og:title ' );
  // Structured data
  $ ( ' script[type="application/ld+json"] ' ). each (( i , el ) => {
    let data ;
    try { data = JSON . parse ( $ ( el ). text ()); }
    catch { return error ( `JSON-LD block ${ i + 1 } is not valid JSON` ); }
    for ( const block of Array . isArray ( data ) ? data : [ data ]) {
      if ( ! block [ ' @context ' ]) error ( `JSON-LD block ${ i + 1 } has no @context` );
      const items = block [ ' @graph ' ] ?? [ block ];
      if ( items . some (( item ) => ! item [ ' @type ' ])) error ( `JSON-LD block ${ i + 1 } has an item without @type` );
    }
  });
}
for ( const file of await findHtml ( dir )) {
  checkPage ( path . relative ( dir , file ). split ( path . sep ). join ( ' / ' ), await readFile ( file , ' utf8 ' ));
}
for ( const [ title , pages ] of titles ) {
  if ( pages . length > 1 ) errors . push ( `duplicate title " ${ title } " on: ${ pages . join ( ' , ' )} ` );
}
warnings . forEach (( w ) => console . warn ( `warning ${ w } ` ));
errors . forEach (( e ) => console . error ( `error ${ e } ` ));
console . log ( `\n ${ errors . length } errors, ${ warnings . length } warnings` );
process . exit ( errors . length ? 1 : 0 );

Run against a sample build with a few planted mistakes, the output looks like this:

warningblog/post/index.html: found 2 <h1> elements
warningblog/post/index.html: missing lang attribute on <html>
warningblog/post/index.html: image without alt: b.png
warningblog/post/index.html: missing og:title
errorabout/index.html: canonical is not an absolute URL: /about/
errorblog/post/index.html: meta robots contains noindex
errorblog/post/index.html: missing meta description
errorblog/post/index.html: canonical points to https://staging.example.com
errorblog/post/index.html: JSON-LD block 1 is not valid JSON
errorduplicate title "Home | Example" on: blog/post/index.html, index.html
6 errors, 4 warnings

Every message names the page and the rule, so whoever opened the PR knows exactly what to fix.

Wire It Into GitHub Actions

# .github/workflows/seo-pr.yml
name : SEO checks
on : pull_request :
jobs : seo :
runs-on : ubuntu-latest
steps : - uses : actions/checkout@v4
- uses : actions/setup-node@v4
with : node-version : 22
cache : npm
- run : npm ci
- run : npm run build
- name : Lint built HTML
run : node scripts/seo-check.mjs dist https://example.com
- name : Check internal links
uses : lycheeverse/lychee-action@v2
with : args : --offline --no-progress --root-dir "${{ github.workspace }}/dist" "dist/**/*.html"

The lychee link checker runs in offline mode here, so it only checks links between your own files. That keeps it fast and immune to third-party sites being slow. External links get checked later, on a schedule.

Keep this whole job under about a minute. If an SEO gate becomes the slowest step in CI, people start looking for ways around it.

Layer 2: Lighthouse CI on Key Templates, Not Every Page

Lighthouse is useful, but it's slow, and its performance scores change from run to run. Running it on every page of every PR is a recipe for flaky builds. Instead, audit three to five representative templates: the homepage, an article, a product or service page and a listing page.

Lighthouse CI can serve your static build itself and assert on the results:

{ "ci" : { "collect" : { "staticDistDir" : "./dist" , "url" : [ "http://localhost/index.html" , "http://localhost/blog/sample-post/index.html" , "http://localhost/pricing/index.html" ], "numberOfRuns" : 3 }, "assert" : { "assertions" : { "categories:seo" : [ "error" , { "minScore" : 0.9 }], "categories:performance" : [ "warn" , { "minScore" : 0.8 }], "largest-contentful-paint" : [ "warn" , { "maxNumericValue" : 2500 }], "cumulative-layout-shift" : [ "warn" , { "maxNumericValue" : 0.1 }] } }, "upload" : { "target" : "filesystem" , "outputDir" : "./lhci-reports" } } }

Save that as lighthouserc.json and add one step to the workflow: npx @lhci/cli autorun.

Notice the split. The SEO category is an error, because its audits are mostly deterministic (crawlable links, valid hreflang, a document title and so on). Performance is a warning, because CI runners vary in speed. Three runs per URL lets Lighthouse CI use the median result, which smooths out the noise.

Layer 3: Check Production Right After Deploy

Some of the most damaging problems only exist on the live server. Response headers, robots.txt, redirects and host settings often come from CDN rules, server config or a hosting dashboard rather than your repository. A PR check can't see them, so test production immediately after every deploy.

robots.txt and X-Robots-Tag

Fail the run if robots.txt blocks everything for User-agent: *, or if key pages send noindex through an X-Robots-Tag header or a meta tag. The header is easy to miss because it never appears in the HTML. Google's robots meta tag and X-Robots-Tag docs explain how both are interpreted.

Be precise about Disallow: /. Plenty of sites block specific bots, such as AI crawlers, on purpose, so the check should only fail when the rule applies to every crawler.

A Redirect Map Test

Keep a redirects.csv file in the repository with two columns, from and to. After each deploy, assert that every old URL returns a single 301 or 308 straight to the expected target, and that the target returns 200. This catches missing redirects, temporary 302s, wrong destinations and chains.

Sitemap Sanity

Request every URL in the sitemap and expect a 200. Then compare the URL count against a stored baseline, and fail if it drops sharply. A sitemap that suddenly loses a large share of its URLs usually means a routing change or a broken data source, not a deliberate cleanup.

The Post-Deploy Script

// scripts/post-deploy-check.mjs
// Usage: node scripts/post-deploy-check.mjs https://example.com
import { readFile , writeFile } from ' node:fs/promises ' ;
const origin = ( process . argv [ 2 ] ?? ' https://example.com ' ). replace ( / \/ $/ , '' );
const KEY_PATHS = [ ' / ' , ' /blog/ ' , ' /pricing/ ' ];
const MAX_SITEMAP_DROP = 0.1 ; // fail if the sitemap shrinks by more than 10%
const failures = [];
const fail = ( msg ) => failures . push ( msg );
// 1. robots.txt must not block everything for all crawlers
function blocksAllCrawlers ( txt ) {
  let agents = [];
  let inRules = false ;
  for ( const raw of txt . split ( / \r?\n / )) {
    const line = raw . replace ( /#.*/ , '' ). trim ();
    const sep = line . indexOf ( ' : ' );
    if ( sep === - 1 ) continue ;
    const key = line . slice ( 0 , sep ). trim (). toLowerCase ();
    const value = line . slice ( sep + 1 ). trim ();
    if ( key === ' user-agent ' ) {
      if ( inRules ) { agents = []; inRules = false ; }
      agents . push ( value );
    } else if ( key === ' allow ' || key === ' disallow ' ) {
      inRules = true ;
      if ( key === ' disallow ' && value === ' / ' && agents . includes ( ' * ' )) return true ;
    }
  }
  return false ;
}
const robotsRes = await fetch ( ` ${ origin } /robots.txt` );
if ( ! robotsRes . ok ) fail ( `robots.txt returned ${ robotsRes . status } ` );
else if ( blocksAllCrawlers ( await robotsRes . text ())) fail ( ' robots.txt has "Disallow: /" for User-agent: * ' );
// 2. Key pages: 200, no noindex header, no noindex meta tag
for ( const p of KEY_PATHS ) {
  const res = await fetch ( origin + p );
  if ( res . status !== 200 ) { fail ( ` ${ p } returned ${ res . status } ` ); continue ; }
  if ( /noindex/i . test ( res . headers . get ( ' x-robots-tag ' ) ?? '' )) fail ( ` ${ p } send