DEV Community 2h ago

Route CI/CD alerts to an agent that triages by email

CI alert email is noisy, and most of it is ignorable. If you've ever owned a pipeline at any scale, you know the shape of it: a flaky integration test trips, GitHub Actions emails you, a retry passes, and now there are three messages in your inbox about a problem that fixed itself. Multiply that by every branch, every nightly, every deploy, and the signal you actually care about - the one real failure - is buried under forty notifications that all look identical until you open them.

The usual fix is to wire alerts into Slack and add a bot that reacts to emoji. That works right up until the channel becomes the new noisy inbox. The other usual fix is to point an LLM at a human's mailbox and let it "summarize your morning." That's a demo, not a system - the moment you want the summarizer to reply, to be a participant in the alert thread that on-call is reading, you need it to own an address, not borrow yours.

That's the angle here. Give the triage agent its own inbox. Your CI system already knows how to email failures somewhere - point it at an address the agent controls, let the agent cluster and summarize the incoming alerts, and have it reply in the same thread with a probable root cause so whoever's on-call reads one message instead of forty.

I work on the Nylas CLI, so the terminal commands below are the exact ones I reach for when I'm building this. I'll show the raw API call alongside each one, because in production this lives in a service, not a shell.

What an Agent Account actually is

An Agent Account is a Nylas grant with its own email address. That's the whole trick. It's not a new product surface you have to learn - it's a grant_id that happens to belong to a programmatic mailbox instead of a human's connected Google or Microsoft account.

Every grant-scoped endpoint you already know works against it unchanged: GET /v3/grants/{grant_id}/messages, POST .../messages/send, threads, folders, all of it. So the data plane is nothing new. You point your CI notifier at ci-alerts@yourcompany.com, the agent receives mail there like any inbox, and you read and reply with the same endpoints you'd use for a connected account.

One honest caveat up front: this is alert triage and summarization over email. It is not a replacement for your observability stack. The agent doesn't have your traces, your metrics, or your deploy history unless you give them to it. What it does well is take the firehose of alert emails - which is genuinely noisy and mostly redundant - collapse it into clusters, and hand on-call a probable cause to start from. Treat its root-cause guess as a hypothesis, not a verdict.

Provision the inbox

Create the account. The CLI wraps POST /v3/connect/custom with provider: nylas:

nylas agent account create ci-alerts@yourcompany.com --name "CI Triage Bot"

The raw call, if you're provisioning from a service:

curl -X POST "https://api.us.nylas.com/v3/connect/custom" \
  -H "Authorization: Bearer $NYLAS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "nylas",
    "name": "CI Triage Bot",
    "settings": {
      "email": "ci-alerts@yourcompany.com"
    }
  }'

The email has to be on a registered domain - either your own custom domain or a Nylas *.nylas.email trial subdomain. There's no OAuth, no refresh token, no consent screen, because nobody's connecting an external account; you're minting one. The API auto-creates a default workspace and policy for the account, which matters later when we route noisy senders.

Now go into your CI system's notification settings - GitHub Actions, Jenkins, CircleCI, whatever you run - and set the alert recipient to ci-alerts@yourcompany.com. From the agent's side, an alert is just inbound mail.

Receive alerts with a webhook

Inbound mail fires the standard message.created webhook. The thing to internalize: webhooks are application-scoped, not grant-scoped. You subscribe once at the app level, and events for every grant in your app land at the same endpoint, each payload carrying a grant_id you filter on. You don't register a webhook per Agent Account.

Subscribe with the CLI:

nylas webhook create \
  --url https://triage.yourcompany.com/nylas/webhook \
  --triggers message.created

Or directly:

curl -X POST "https://api.us.nylas.com/v3/webhooks" \
  -H "Authorization: Bearer $NYLAS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "trigger_types": ["message.created"],
    "webhook_url": "https://triage.yourcompany.com/nylas/webhook",
    "description": "CI alert triage"
  }'

Two things every webhook handler needs to get right, and they bite people who skip them.

Verify the signature. Nylas signs each delivery with an X-Nylas-Signature header - a hex HMAC-SHA256 of the raw request body using your webhook secret. Compute the same HMAC over the raw bytes and compare. If you use a constant-time compare like Node's crypto.timingSafeEqual, guard that both buffers are the same length first, because it throws on a length mismatch. The CLI ships nylas webhook verify to check a payload locally while you're developing.

Deduplicate. Nylas guarantees at-least-once delivery - the same event can arrive up to three times. Dedupe on the top-level notification id, which stays constant across every retry of one event. That's your delivery-dedup key. The inner data.object.id is the message id; you can additionally guard on it so you never act twice on the same alert, even if your CI somehow sent two near-identical emails.

That dedup discipline is half the value of this whole system. CI alert noise is duplication. If your handler is idempotent on the notification id and fingerprints the underlying alert, you've already killed most of the noise before the LLM does anything clever.

Fetch the alert body

The message.created payload gives you enough to route - sender, subject, thread id - but don't rely on the webhook payload for the body. The Nylas docs themselves are inconsistent on whether the body comes inline, and there's a real edge: when a message exceeds ~1 MB the trigger becomes message.created.truncated and the body is dropped. CI alerts with full stack traces and log tails get big.

So the safe pattern is: branch on message.created.truncated, and fetch the full message by id whenever you actually need the body.

curl "https://api.us.nylas.com/v3/grants/$GRANT_ID/messages/$MESSAGE_ID" \
  -H "Authorization: Bearer $NYLAS_API_KEY"

The CLI equivalent reads the message and renders the body:

nylas email read $MESSAGE_ID $GRANT_ID

Now you have the raw alert text - the failing job name, the stage, the stack trace, the log excerpt your CI tool stuffed into the email. That's the input to the part Nylas doesn't do for you.

Cluster and summarize - this is your code

Here's the line I want to be honest about: Nylas delivers the alert mail. It does not analyze it. The clustering, the summarization, and the root-cause guess are all your application code and your LLM. Nylas is the inbox and the transport, not the brain.

A workable shape:

Fingerprint the failure. Pull stable signal out of the body - repo, workflow, job, failing step, the first line of the stack trace or assertion. Hash a normalized version of that into a fingerprint. This is deterministic string work; don't waste an LLM call on it.

Cluster on the fingerprint. Keep state in your own database - Agent Accounts don't support custom metadata on messages, so you can't stash a cluster id on the Nylas message and filter by it later. Your DB owns the mapping from fingerprint to "open incident." Five alerts with the same fingerprint in ten minutes is one incident, not five.

Summarize the cluster with the LLM. Feed the model the deduped set - "the integration-tests job on main failed 6 times in 12 minutes, here are the three distinct error signatures" - and ask for a tight summary plus a probable root cause. Keep the prompt deterministic: low temperature, a fixed output shape (summary, suspected cause, confidence), and validate it before you trust it.

The model is doing exactly the kind of judgment work it's good at - reading messy human-and-machine text and proposing a cause. It is not doing routing or counting; your code does that, because code is deterministic and the model isn't. If the alert text says "connection refused to postgres," the model can reasonably suggest the DB container didn't come up. That's a hypothesis on-call can confirm in seconds - which is the whole point.

Reply in-thread with the root cause

This is where the Agent Account earns its keep. Because the agent owns the address, its reply threads naturally with the original alert - Nylas groups it using the In-Reply-To and References headers, so on-call sees your summary attached to the exact alert it explains, not floating in a separate channel.

The CLI reply preserves the thread automatically; it fetches the original to set the recipient and subject:

nylas email reply $MESSAGE_ID $GRANT_ID \
  --body "Clustered 6 failures of integration-tests on main (last 12 min) into one incident. Probable cause: postgres service container failed health check before tests ran - 'connection refused' on :5432 in 5 of 6 runs. Suspect the DB image bump in #4821. Confidence: medium."

The raw call sends a reply by setting reply_to_message_id, which is what keeps it in the thread:

curl -X POST "https://api.us.nylas.com/v3/grants/$GRANT_ID/messages/send" \
  -H "Authorization: Bearer $NYLAS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "reply_to_message_id": "'"$MESSAGE_ID"'",
    "to": [{ "email": "oncall@yourcompany.com" }],
    "body": "Clustered 6 failures of integration-tests on main into one incident. Probable cause: postgres health check failed before tests ran. Suspect the DB image bump in #4821. Confidence: medium."
  }'

Set to to wherever on-call actually reads - a rotation alias, the team list, whatever your PagerDuty schedule resolves to. The reply lands in the alert thread and in on-call's inbox, which is the behavior you want: one authoritative summary message per incident, threaded under the noise it replaces.

Marking the original alert read, if you want the agent's inbox to reflect what it's already processed, is a separate operation - a GET fetches, it doesn't mark anything read:

nylas email mark read $MESSAGE_ID $GRANT_ID

curl -X PUT "https://api.us.nylas.com/v3/grants/$GRANT_ID/messages/$MESSAGE_ID" \
  -H "Authorization: Bearer $NYLAS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "unread": false }'

Optionally route a chatty notifier into its own folder

Some CI notifiers are chattier than others, and you may want to keep low-value automated mail out of the agent's inbox entirely - successful-build confirmations, for instance. You can do that at the platform level with a Rule, before your handler ever sees it.

One important constraint: inbound rules match on the sender only. They accept from.address, from.domain, and from.tld with operators like is, is_not, contains, and in_list. They cannot match on subject or body. So "route anything with [SUCCESS] in the subject" is not a rule - that's your app code after the webhook (fetch, classify, then nylas email move the message). But "route everything from noreply@ci-provider.com into a firehose folder" is exactly a rule's job.

Create the folder first and capture its id - the assign_to_folder action takes a folder ID, not a folder name:

curl -X POST "https://api.us.nylas.com/v3/grants/$GRANT_ID/folders" \
  -H "Authorization: Bearer $NYLAS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "name": "ci-firehose" }'
# capture the returned folder id into CI_FOLDER_ID

Then create the rule, pointing the action at that id:

nylas agent rule create \
  --name "Route CI firehose" \
  --condition from.domain,is,ci-provider.com \
  --action assign_to_folder=$CI_FOLDER_ID

If you create the rule through the raw /v3/rules API instead, the action carries that same folder ID as its value - {"type": "assign_to_folder", "value": "<CI_FOLDER_ID>"} - never the folder name.

A rule does nothing until it's attached to a workspace via rule_ids. The CLI's agent rule create attaches it to the account's default workspace for you; the raw flow is create the rule, then patch the workspace:

curl -X PATCH "https://api.us.nylas.com/v3/workspaces/$WORKSPACE_ID" \
  -H "Authorization: Bearer $NYLAS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "rule_ids": ["'"$RULE_ID"'"] }'

If you ever need to re-attach or swap rules from the CLI, that's nylas workspace update <workspace-id> --rules-ids rule1,rule2.

Reach for this only when a sender is reliably noise. The interesting routing - by failure type, by severity, by repo - is content-based, and that belongs in your code.

Guardrails worth setting before you ship

Idempotency is non-negotiable. Dedup on the notification id, fingerprint the alert, keep incident state in your DB. Without it, retries and re-deliveries turn your "less noise" system into more noise.
The root cause is a hypothesis. Label it that way in the reply - include a confidence level. On-call should treat it as a starting point, not a closed ticket. Over-confident summaries erode trust fast.
Watch the free-plan limits. Free accounts cap at 200 messages per account per day, with a 30-day inbox retention. A busy pipeline can blow past 200 alert emails on a bad day, so size your plan to your real alert volume.
New domains warm up. If you stand this up on a fresh custom domain, deliverability ramps over roughly four weeks. For an internal triage inbox that's rarely an issue, but it's worth knowing if replies seem slow to land early on.
Keep fetch and mark-read separate. Reading a message via GET never changes its flags. If you want read-state to mean "the agent has processed this," set it explicitly with the PUT / nylas email mark read.

What's next

Once the basic loop works - receive, cluster, reply - the natural extensions are all just more of the same grant-scoped endpoints you've already wired:

Enrich the root-cause guess by pulling deploy or commit context from your own systems and including it in the summary. Route alerts by severity into different folders or trigger different reply templates. Build a dashboard that shows open incidents per cluster, so on-call can see the state of the firehose at a glance. The pattern scales because the primitives - an inbox, a webhook, a send endpoint - are the same ones you'd use for a human account, just automated.

Read on DEV Community ↗ ← Back to News

Route CI/CD alerts to an agent that triages by email