Keyword Research with AI Agents and Google Search Console

Keyword Research with AI Agents and Google Search Console Manual keyword research is slow and disconnected from real user behavior. I use AI agents connected to Google Search Console data to automate keyword clustering, intent analysis, and content gap detection for client websites. This article shows the complete workflow: from raw GSC data to actionable keyword clusters that drive content decisions. The problem with manual keyword research Traditional keyword research relies on guesswork. You look at competitor sites, use keyword tools with volume estimates, and try to predict what users might search for. But if you already have a website, Google is telling you exactly what people search for - and you're not listening. Google Search Console provides actual search query data, grouped by page, with impressions, clicks, and average position. This is the real signal. The problem is extracting actionable insights from thousands of rows of query data. That's where AI agents come in. The architecture The system consists of three components: - Data source: Google Search Console Search Analytics API - AI agent: A coding agent with access to the GSC data and the ability to execute scripts - Cluster output: A JSON mapping of queries to pages, intents, and optimization opportunities The agent runs a Node.js script that authenticates with Google Cloud, fetches 12 months of search query data, and processes it into keyword clusters organized by page and intent. Step 1: Setting up GSC access Before anything else, you need API access to Search Console data. This requires: - A Google Cloud project with the Search Console API enabled - Application Default Credentials (ADC) configured with gcloud auth application-default login - The Search Console property added to your account # Configure ADC credentials gcloud auth application-default login # Set the quota project (required since 2026) export GOOGLE_CLOUD_QUOTA_PROJECT=your-project-id Step 2: The keyword cluster script The core of this workflow is a Node.js script that fetches GSC data and processes it. Here's how it works: // Authenticate with GSC API async function mintAdcToken() { const result = await execFileAsync('gcloud', [ 'auth', 'application-default', 'print-access-token' ]); return result.stdout.trim(); } // Fetch query data for a specific period async function fetchGscRows({ startDate, endDate }) { const accessToken = await mintAdcToken(); const response = await fetch( 'https://searchconsole.googleapis.com/webmasters/v3/sites/' + encodeURIComponent('sc-domain:your-site.com') + '/searchAnalytics/query', { method: 'POST', headers: { authorization: Bearer ${accessToken}, 'content-type': 'application/json', 'x-goog-user-project': process.env.GOOGLE_CLOUD_QUOTA_PROJECT }, body: JSON.stringify({ startDate, endDate, dimensions: ['query', 'page'], dataState: 'final', rowLimit: 25000 }) } ); return (await response.json()).rows; } The script fetches up to 25,000 rows of query data spanning the last 12 months. It uses pagination to handle larger datasets. Step 3: Processing the data into clusters Once the data is fetched, the processing happens in several stages: Map routes to pages: Each query result is mapped to a canonical route, stripping fragments and query parameters. Aggregate by page: For each page, all queries are collected along with their impressions, clicks, and positions. Identify intent clusters: Queries are grouped by intent. "What is X" queries are informative. "Buy X" or "hire X" queries are commercial. The intent is inferred from both the query phrasing and the page it leads to. Detect cannibalization: When multiple pages rank for similar queries, the script flags potential cannibalization where pages compete for the same search terms. Here's a simplified version of the clustering logic: function clusterQueries(page, queries) { return { route: page, pageMetrics: calculatePageMetrics(queries), primaryTerm: queries[0].query, // Highest click volume cluster: queries.map((q, i) => ({ query: q.query, clicks: q.clicks, impressions: q.impressions, ctr: q.clicks / q.impressions, role: i === 0 ? 'primary' : (i <= 9 ? 'secondary' : 'long-tail'), sharedWith: detectCannibalization(q.query, allPages) })) }; } Step 4: Using the clusters for content decisions The output is a JSON file that maps each page to its keyword clusters. This becomes the foundation for all content and SEO decisions. Optimization targets: Pages with high impressions but low clicks are prime optimization targets. The title, meta description, or content may need improvement. Internal linking strategy: Use the cluster data to identify which pages should link to which. If a page ranks for "website development services" and another ranks for "hire a web developer," they should link to each other. Content gap identification: Queries with impressions but no corresponding page suggest new content opportunities. The queries tell you what users want but don't find. Commercial vs. informational intent: The intent labels help decide whether a page should be a blog post (informative) or a landing page (commercial). Example: real data from our site Here's a real example from our Search Console data (August 2026): | Query | Impressions | Position | Current page | |---|---|---|---| | website laten maken | 2,019 | 16.7 | /blog/wat-kost-een-website-laten-maken | | goedkope website laten maken | 1,232 | 21.3 | /blog/goedkoop-website-laten-maken | | een website laten maken | 401 | 19.2 | /blog/wat-kost-een-website-laten-maken | | astro website laten maken | 15 | 7.0 | /astro-website-laten-maken | (These queries are in Dutch - they come from a Dutch-language client site we manage. The workflow itself works the same in any language.) What does this tell us? - The commercial head term is being carried by an informational price blog - suboptimal - "goedkope website laten maken" appears on two different blogs - classic cannibalization - "astro website laten maken" has a clear owner and ranks well (position 7.0) - Opportunity: Create a dedicated commercial landing page for "website laten maken" This kind of insight would take hours to discover manually. The script generates it in minutes. Automating the workflow with AI agents The real power comes from automating this workflow. With an AI coding agent, you can: - Schedule regular runs: Run the script weekly or monthly to track changes - Generate reports: Have the agent analyze the data and generate optimization recommendations - Implement changes: Have the agent actually update titles, meta descriptions, and internal links based on the cluster data - Track results: Compare before and after data to measure the impact of changes The agent doesn't replace your SEO judgment - it amplifies it. You still decide the strategy, but the agent handles the data analysis and implementation. Practical tips - Always compare fixed periods: When measuring changes, use complete, non-overlapping periods. Don't compare this week with last week if they have different numbers of days. - Track baseline metrics: Before making any SEO changes, record the current impressions, clicks, and positions for the affected pages and queries. - Give changes time: SEO changes can take 2-4 weeks to show in search data. Don't make rapid changes and expect immediate results. - Use the API, not the UI: The Search Analytics API gives you more data and more control than the web interface. It's worth the extra setup. Conclusion Keyword research doesn't have to be manual guesswork. By connecting AI agents to your real Google Search Console data, you can build a data-driven SEO workflow that continuously optimizes based on actual user behavior. The key insight: you already have the data you need. You just need the right tools to process it. This workflow is based on our experience running keyword research for client websites. The scripts described here are used in production on straffesites.com. New to keyword research? Start with How to do keyword research in 6 clear steps on the Straffe Sites blog. Top comments (0)