Give Your CI Pipeline an Error Budget Before Adding AI Triage

Service-Level Indicators for CI Health

An AI-generated explanation cannot rescue an unhealthy CI pipeline. If jobs wait unpredictably, runners disappear, retries hide flakes, or logs lack revision metadata, the model is summarizing unreliable evidence.

Start with four service-level indicators:

• queue_seconds = job_started_at - job_queued_at
• execution_seconds = job_finished_at - job_started_at
• infra_failure_rate = infrastructure_failures / completed_jobs
• retry_recovery_rate = passed_after_retry / retried_jobs

Split failures into code, test_flake, infrastructure, dependency, policy, and unknown. Do not let unknown become permanent.

Practical Objectives

For a protected-branch workflow, one starting target could be:

• 95% of jobs begin within 120 seconds
• 95% finish within 15 minutes
• Fewer than 1% end in infrastructure failure
• Fewer than 2% pass only after an unchanged retry
• Every failure includes repository, revision, workflow version, runner image, attempt, and artifact links

These are example thresholds, not universal standards. Derive yours from developer wait tolerance, release risk, runner capacity, and current distributions.

If a week has 1,000 protected-branch runs and the infrastructure-failure objective allows 10, the eleventh failure should change priorities. Pause runner-image changes, investigate the dominant signature, or reduce concurrency until recovery.

Instrumentation and Retry Handling

Instrument queue and execution separately. Scaling executors cannot fix serialized dependency downloads; caching cannot fix insufficient runner capacity.

Record retries as new attempts linked to the original job, never overwrite the initial failure.

Structured Evidence for AI Triage

Before AI triage, require structured evidence:

{
  "revision": "abc123",
  "workflow_revision": "def456",
  "runner_image": "ubuntu-24.04@sha256:...",
  "attempt": 2,
  "previous_attempt": 1,
  "exit_code": 1,
  "failure_class": "test_flake"
}

Then an AI summary can cite stable logs and metadata. Evaluate it on classification accuracy, missing-evidence rate, analyst correction time, and dangerous false reassurance-not prose quality.

Context and Disclosure

The public MonkeyCode repository describes AI tasks, development environments, and automated PR/MR review. CI evidence can feed that category of workflow, but this article reports no MonkeyCode deployment measurement or integration test.

Disclosure: I contribute to the MonkeyCode project. The public repository provides context; the SLO framework is independent.

Operate the evidence first. Automate the explanation second.