A 50-capability map for governed web crawling and AI agents

Giving an agent “web access” sounds like one feature. In practice, it is a stack of separate decisions: - How does the system discover URLs? - Which destinations can it contact? - Does it need a browser, or is static HTTP enough? - What turns the response into agent-ready data? - Where are request, byte, depth, and time limits enforced? - What evidence comes back with the extracted content? Treating all of that as one unrestricted browser capability makes systems difficult to reason about. A better approach is to choose the smallest acquisition surface that completes the job, then make its authority explicit. This article maps 50 current Cockroach Crawler capabilities into seven jobs. It is also a practical checklist you can use with another crawler: if a capability matters to your workflow, identify its input contract, output contract, failure behavior, and authority boundary before an agent depends on it. Disclosure: I’m Ajnas N B, the developer of Cockroach Crawler. The project is open source under the MIT license. Start with a finite crawl contract The next channel currently contains the reviewed 0.7.0-rc.1 prerelease. A bounded documentation crawl can start like this: npm install cockroach-crawler@next import { crawlDetailed } from "cockroach-crawler"; const result = await crawlDetailed({ seeds: ["https://docs.example.com"], allowedOrigins: ["https://docs.example.com"], include: ["/guides/", "/reference/"], exclude: ["/archive/"], traversal: "bfs", obeyRobots: true, maxPages: 25, maxRequests: 120, maxDepth: 4, maxTotalBytes: 10_000_000, maxDurationMs: 60_000, concurrency: 4 }); for (const page of result.pages) { console.log(page.url, page.contentHash, page.markdown.length); } The important part is not the number of options. It is ownership: the creator of the agent sets the origins and ceilings. Model-facing input can narrow that contract, but it should not be able to expand it. 1. Crawl and discover - 15 capabilities These capabilities decide what enters the queue, what is contacted, and when the job stops. - Static HTTP crawling - fetch public HTTP(S) pages without starting a browser. - Multiple seeds - begin one bounded job from several explicit entry points. - Breadth-first traversal - cover each depth level before going deeper. - Depth-first traversal - follow the newest admitted path first. - Best-first traversal - rank admitted links against a bounded relevance query. - Adaptive relevance traversal - reprioritize the queue as relevant page text appears. - Sitemap discovery - read robots-declared and conventional sitemap locations, including nested indexes. - Robots enforcement - evaluate robots policy before page contact and preserve the decision. - Include and exclude filters - admit only the paths that belong to the job. - Validated redirects - inspect and admit every redirect destination before following it. - Concurrency and politeness - combine exact concurrent work with per-origin delays and global ceilings. - Deadlines and cancellation - stop by wall-clock budget or AbortSignal . - Persistent cache - reuse hash-verified results inside an explicit namespace, TTL, entry, and byte budget. - Compact fetch-validated site maps - return URL metadata without retaining complete page bodies. - Searchable fetch-validated site maps - rank only entries already admitted and fetched under the crawl policy. For discovery work, the key distinction is between ranking and authority. A relevance score may reorder already admitted links; it must not broaden the origin policy or resource budget. 2. Render and capture - 9 capabilities Static HTTP should remain the default when it works. Browser execution is useful when content genuinely depends on client-side rendering or bounded interaction. - JavaScript rendering through optional Chromium. - Selector waits and bounded clicks for explicit page states and interactions. - Infinite and virtual scroll with finite steps and stability checks. - Open Shadow DOM flattening into a bounded extraction snapshot. - Readable same-origin iframe flattening while preserving cross-origin isolation. - Full-page screenshots with format, size, and SHA-256 evidence metadata. - PDF generation with explicit print settings. - Trusted operator page hooks that are reviewed configuration, not model input. - Explicit persistent browser profiles using a dedicated directory rather than discovering a personal browser profile. Browser mode is not a process sandbox. Host isolation, egress policy, CPU and memory limits, and sensitive-data separation still belong to the deployment. 3. Extract agent-ready data - 8 capabilities Retrieval is not finished when bytes arrive. Agents need a bounded record that preserves enough source identity to verify or revisit the result. - Readable Markdown through the dependency-light core or opt-in Node quality backend. - CSS schema extraction for visible text, cleaned HTML, and named attributes. - XPath extraction for deterministic fields in inactive markup. - Restricted regex extraction with safe flags and hard input, item, value, and total ceilings. - Optional host-model JSON Schema extraction where returned JSON must validate against the supplied schema. - Local PDF parsing with signature, page, byte, and text ceilings. - Links and page metadata including canonical URL, title, description, language, status, ETag, and Last-Modified. - Evidence hashes and retrieval provenance including SHA-256, fetch time, parent, depth, and redirect history. Deterministic extraction and model-assisted extraction are different contracts. If a host model is used, its output should be treated as untrusted until it passes the supplied schema and size limits. 4. Reach public sources - 6 capabilities Provider integrations should say what access state they require before dispatch. “Supported” is not enough if the operator cannot tell whether a route is public, credentialed, session-backed, or unavailable. - Public GitHub repository and issue reads. - YouTube search and metadata without a developer API key through an optional reviewed route. - Official YouTube, X, and Reddit provider adapters. - Optional read-only session providers for X, Reddit, Facebook, Instagram, LinkedIn, and Xiaohongshu. - Offline RSS and Atom parsing. - Provider doctor, capability reporting, and deterministic routing. Run the doctor before choosing a source route: npx cockroach-sources doctor --json npx cockroach-reach doctor --json Optional session providers are operator-installed read routes. They do not expose posting, liking, following, messaging, deleting, cookie extraction, or personal profile discovery. 5. Connect agents - 3 capabilities - Strict creator-bounded agent tool whose model input may narrow but cannot broaden host-owned origins and budgets. - Native MCP stdio server with crawl, map, extraction, and machine-readable capability surfaces. - Optional Maqam policy, approval, trace, and evidence integration for registered crawler operations. A minimal MCP launch keeps authority in environment configuration: COCKROACH_ALLOWED_ORIGINS=https://docs.example.com \ COCKROACH_MAX_PAGES=10 \ npx cockroach-mcp 6. Deploy and operate - 4 capabilities - Authenticated Node.js and Docker API for health, playground, crawl, map, and extraction routes. - Responsive dashboard and browser playground for local inspection. - Bounded process-local asynchronous jobs with concurrency, pending, retained-result, and result-byte ceilings. - Fixed-origin Cloudflare Worker profile for a small deployment-configured HTTPS fetch tier. The process-local queue is intentionally not presented as a durable distributed queue. If a workflow needs cross-machine durability, retries across restarts, or independent worker scaling, connect external infrastructure rather than pretending an in-memory queue provides it. 7. Keep authority bounded - 5 capabilities - Public-network admission and SSRF defenses that reject unsafe schemes, credentials, private ranges, and metadata destinations. - DNS pinning and explicit origin policy for the Node transport. - Exact resource ceilings across pages, requests, queue, depth, bytes, retries, redirects, callbacks, and duration. - Fixed self-hosted proxy-gateway adapter that does not accept model-selected endpoints or credentials. - Challenge-aware provider escalation that stops without access-control bypass. The fixed-origin Worker profile is a smaller deployment tier and does not provide the Node transport’s DNS-resolution and pinning guarantees. That boundary matters when deciding where a job may run. How to choose the smallest useful surface Use this order: - Start with static HTTP and deterministic extraction. - Add sitemap or relevance traversal only when the queue needs better discovery. - Add browser rendering only when the target content cannot be obtained correctly without it. - Add bounded interaction only for a specific, reviewed state transition. - Add provider adapters only after capability inspection reports the expected access state. - Give an agent the strict tool or MCP surface only after the host fixes origins and budgets. - Preserve hashes, redirects, warnings, and retrieval metadata beside the content. This makes failures easier to interpret. A robots refusal, origin denial, byte ceiling, extraction warning, provider-unavailable state, and browser challenge are not the same failure and should not collapse into “the crawler returned nothing.” The current extraction measurement The published 0.7.0-rc.1 quality path reports the following on the 511-page observed WCEB partition: - precision: 0.894101 - recall: 0.926022 - macro F1: 0.890524 That partition is labeled observed development evidence because the project had previously iterated against it. It is useful for reproducing the published candidate, but it is not an untouched confirmatory test set and it does not support a universal “0.90” or best-crawler claim. The complete benchmark method, artifacts, hashes, and wider 1,497-page developm