EEM 101: On-Box Automation That Runs Even When Your NMS Doesn't

What EEM Actually Is

Embedded Event Manager is an event-driven automation framework built into Cisco IOS, IOS-XE, and NX-OS. Strip away the jargon and it's a very simple idea: When something happens, do something about it - automatically, on the device itself.

That "something happens" is an event. That "do something" is one or more actions. Bundle an event with its actions and you have an applet - the basic unit of EEM. That's the whole model.

The power comes from how many different things can be an event, and how much an action can do.

Events EEM can watch for include:

• A syslog message matching a pattern (an interface flapping, an HSRP state change, a config being saved)
• An SNMP OID crossing a threshold (CPU over 85%, a power supply going absent, temperature rising)
• A CLI command being entered (someone typing write erase - catch it before it runs)
• A timer (every 60 seconds, or a cron schedule like "every night at 2 AM")
• An interface counter (input errors climbing, a specific counter rate)
• A track object changing state (an IP SLA probe failing, a route disappearing)
• None - meaning you trigger it manually or from another applet (useful for testing and for chaining logic)

Actions an applet can take include:

• Run any CLI command - show commands, config changes, clears
• Write a syslog message (leave a breadcrumb that this applet fired)
• Send an SNMP trap to your monitoring/SIEM
• Send an email (on platforms/configs that support it)
• Write to a file on flash/bootflash (capture diagnostics with a timestamp)
• Do arithmetic and conditionals (if/else, counters, variables) for logic beyond a straight command list

Mix and match those, and you can express an enormous range of behavior - from "recover this port" to "when CPU spikes at night, snapshot the process table to a timestamped file and trap the SIEM."

Why "On the Box" Changes Everything

You might reasonably ask: I already have monitoring, automation, maybe Ansible or a NetOps platform. Why do I need automation inside the device?

Because external automation has a fatal dependency: it has to be able to reach the device. And the exact moments you most want automation are the moments that dependency is most likely to be broken.

• The WAN is down. Your central automation server can't reach the branch router. But the branch router can still see its own interfaces flapping and react locally.
• The control plane is slammed. CPU is pegged, the box is barely responding to SSH - but EEM, running on the device, can capture the diagnostics you'd otherwise never get because you can't even log in.
• It's 3 AM and the problem lasts 90 seconds. No human is watching. By the time an alert reaches someone and they connect, the transient is gone. EEM was already there, in the moment.
• The failure is the network itself. When the thing that's broken is connectivity, any automation that depends on connectivity is out of the game. On-box automation isn't.

This is the core reason EEM matters and the reason this whole series exists: it's the automation that keeps working when everything else can't reach the patient.

External tooling is fantastic for orchestration, provisioning, and scale. EEM is for the fast, local, autonomous reaction that has to happen right there, right now, regardless of what's happening everywhere else. They're complementary. This series is about the layer most people are missing.

Anatomy of an Applet

Let's read a real one before we build. Here's the classic err-disabled recovery applet - the one everybody has - annotated so you can see every piece:

event manager applet ERRDISABLE-RECOVERY
 event syslog pattern "%PM-4-ERR_DISABLE" ratelimit 60
 action 1.0 syslog msg "EEM: err-disable detected, attempting recovery"
 action 2.0 cli command "enable"
 action 3.0 cli command "clear errdisable interface"
 action 4.0 syslog msg "EEM: err-disable recovery action completed"

Breaking it down:

• event manager applet ERRDISABLE-RECOVERY - defines the applet and names it. The name is how you'll see it in show event manager output later, so make it descriptive.
• event syslog pattern "%PM-4-ERR_DISABLE" - the trigger. EEM watches the logging buffer, and when a message containing that pattern appears, the applet fires. (This is why logging has to be enabled and reaching the buffer - no log, no trigger.)
• ratelimit 60 - a guardrail. The applet can't fire more than once per 60 seconds, so a storm of err-disable events can't make it thrash. Almost every applet you write should have a ratelimit. More on why in a moment.
• action 1.0 ... 4.0 - the response, executed in order. The numbering (1.0, 2.0…) sets sequence and lets you insert steps later (1.5) without renumbering.
• action ... syslog msg - breadcrumbs. These write a line into the log every time the applet runs. They cost nothing and they're the difference between "I think the applet is working" and "here's exactly when and why it fired." Always leave breadcrumbs.

That's the anatomy. Every applet in this series is a variation on this shape: a trigger, a guarded response, and breadcrumbs so you can see what happened.

Your First Three Applets

Enough theory. Here are three applets that deliver value immediately - each illustrating a different event type so you get a feel for the range. Deploy them in a lab first, then a maintenance window.

1. Err-disabled recovery (syslog-triggered) - but done right

The standard applet recovers every err-disabled port blindly, which isn't always what you want - sometimes a port is err-disabled for a good reason (a security violation you should investigate). A more thoughtful version logs loudly and recovers, so at least you have a record:

event manager applet ERRDISABLE-RECOVERY
 event syslog pattern "%PM-4-ERR_DISABLE" ratelimit 30
 action 1.0 syslog priority warnings msg "EEM: err-disable event detected - review recommended"
 action 2.0 cli command "enable"
 action 3.0 cli command "show interfaces status err-disabled"
 action 4.0 cli command "clear errdisable interface"
 action 5.0 syslog priority notifications msg "EEM: err-disable recovery attempted"

Note action 3.0 - it runs a show so the output lands in the log/history, giving you a record of which ports were affected before you cleared them. Small touch, big help during a post-incident review.

2. Catch a dangerous command before it runs (CLI-triggered)

This one shows off a completely different trigger - watching for a command being typed. Here we intercept write erase and force a conscious confirmation, turning a fat-finger catastrophe into a deliberate act:

event manager applet CONFIRM-WRITE-ERASE
 event cli pattern "write erase" sync yes
 action 1.0 syslog priority critical msg "EEM: write erase was entered by a user - intercepted"
 action 2.0 puts "WARNING: 'write erase' intercepted by EEM."
 action 3.0 puts "This will erase startup-config. Type the command again within 10s to proceed, or wait to abort."
 set 4.0 _exit_status "0"

The sync yes means the applet runs synchronously before the command is allowed to execute, so it can gate it. Even in its simplest form - just logging who ran a destructive command with a critical-severity message to your SIEM - this is enormously useful for accountability. (We go deep on this guardrail pattern in Part 4.)

3. A scheduled config backup (timer-triggered)

A third trigger type: time. This applet fires on a cron schedule and pushes the running-config to a remote server every night - no external scheduler needed, the device backs itself up:

event manager applet NIGHTLY-CONFIG-BACKUP
 event timer cron cron-entry "0 2 * * *"
 action 1.0 syslog priority notifications msg "EEM: starting nightly config backup"
 action 2.0 cli command "enable"
 action 3.0 cli command "copy running-config tftp://192.168.37.50/$HOSTNAME-backup.cfg" pattern "\?"
 action 4.0 cli command ""
 action 5.0 syslog priority notifications msg "EEM: nightly config backup completed"

The cron-entry "0 2 * * *" means 2:00 AM daily. The $HOSTNAME built-in variable names the file per device automatically. The pattern "\?" / empty-command pair handles the interactive confirmation prompt that copy throws - a common gotcha when scripting IOS copy operations. (Use SCP/SFTP instead of TFTP in any real environment; TFTP is here only for a readable example.)

Three applets, three different triggers - syslog, CLI, timer - and you've already seen most of the EEM event model in action.

The Guardrails to Internalize Now

Because EEM runs with the device's own privileges and can change configuration, a careless applet is a genuine risk. Build these habits from applet number one:

• Always ratelimit. An applet triggered by a flapping condition, without a rate limit, becomes part of the problem - thrashing an interface or hammering the CPU. Cap how often it can fire.
• Always leave syslog breadcrumbs. Every applet should announce when it fires and what it did. Silent automation is un-debuggable automation. The absence of an expected breadcrumb is itself a signal.
• Test in a lab, then a window. An applet that changes config is a change. Treat it like one. Never let your first test of a config-modifying applet be on production.
• Don't mask root cause. EEM that "fixes" a symptom can hide a chronic problem. Auto-recovering a port that keeps err-disabling isn't a fix - it's a snooze button. Use EEM to buy time and capture evidence, then fix the real thing.

These four aren't bureaucracy - they're what separates automation that saves you from automation that pages you at 3 AM for a fire it started.

Verifying Your Applets

Two commands you'll use constantly throughout this series:

• show event manager policy registered - Confirms which applets are registered and armed. If your applet isn't listed here, it isn't running.
• show event manager history events - Shows every time an applet fired and what event triggered it. This is your window into whether the automation is actually working - pair it with the syslog breadcrumbs and you can reconstruct exactly what happened and when.

To test an applet on demand without waiting for its real trigger, you can build a version with event none and run it manually with event manager run <applet-name> - handy for validating your action logic before wiring it to a live event.

Where This Series Goes Next

You now have the model - event plus action - and three working applets spanning three trigger types. That's the foundation everything else builds on.

• Part 2 - Self-Healing Networks: applets that detect and fix problems autonomously, so the network recovers before you even wake up.
• Part 3 - The 3 AM Problem: using EEM to capture the diagnostics for transient issues you'd otherwise never catch.
• Part 4 - EEM as a Guardrail: dangerous-command interception, config-change detection, and baseline-drift alerting - compliance-grade control at the device level.
• Part 5 - A Complete Real-World Build: a full two-router deployment automating an HSRP-ineligible interface, including the syslog regex that catches every HSRP transition (the one almost everyone gets wrong).

The single idea to carry forward: EEM is the automation that keeps working when everything else can't reach the device. Once you start thinking that way, you'll see uses for it everywhere.

What's the one applet you already run - and what would you automate on-box if you had time to build it? I'm collecting real-world use cases as this series runs.

Visit Website: Digital Security Lab