My Self-Hosted AI Assistant Kept Overwriting Its Own API Keys: A Zero-Trust Postmortem

🦞 Code for this post: sanitized watchdog + config-repair scripts at github.com/backitupboys/ai-agent-zero-trust

Part 1 of a series on running AI agents on real infrastructure without regretting it.

I've spent 20+ years administering infrastructure - Windows NT to Win2000 to well, you get it... PCAnywhere 😬... VMware, Nutanix, Azure, security hardening. I thought I understood untrusted actors on a network. Then I gave an AI agent write access to its own config files, and it taught me the lesson again from scratch.

The setup

I run a self-hosted AI assistant - I call it "The Doctor" (Voyager and Tardis ref.) - in an OpenClaw container on a Hostinger VPS. It's a hobby project: part experiment, part obsession, part fun excuse to learn agentic architecture hands-on. The assistant talks to LLMs through an API gateway, authenticated with an API key stored in its config. I mean, if you're here, you prolly know already... Standard stuff.

The incident

After granting magic access to "do what it needs to be better," the assistant started throwing persistent 401 authentication errors. I checked the config - the key was wrong. Fixed it. Working again. Next boot: 401s. Had to ask Claude, what the heck was happening....!!!??? Key was wrong again.

Here's the part that took me longer to accept than it should have: the assistant was overwriting its own API keys on startup. Its initialization routine was "helping" by rewriting config files - including the credentials block - with stale or generated values. My own agent was hard coding expired API's. Every time it booted, it re-broke its authentication.

I wasn't fighting an attacker. I was fighting a well-intentioned process with too much write access and no concept of oops.

The fix (short-term)

• Confirmed the failure mode - diffed the config before and after boot to prove the agent was the writer; not me, not corruption, not the provider, using Claude.
• Asked Claude to fix it. Claude then:
  • Wrote Python patch scripts to safely repair the config so I didn't have to hand-edit any live files at 1 a.m. (config_patch.py)
  • Deployed a watchdog - a cron job that validates the API key block on a schedule and restores the known-good value if the agent overwrites it accidentally ;) (keyguard_watchdog.sh)

The watchdog worked. It was also a band-aid, and I knew while I was writing it, well, I felt it, basically: I was fighting a well-intentioned process with too much misguided (my fault) 🤦‍♂️ write access. The irony was, I was using agentic AI to fix my agentic AI 🙄. This no longer seemed like a constructive project. This was beginning to become a costly ($75 in a week) loop of over-complicating things.

The real lesson

The long-term fix was a mindset shift, and it's the reason I'm writing this up - because I think most people deploying AI agents haven't hit this wall yet: An AI agent should be implemented exactly like any other untrusted process in a zero-trust model - no matter how helpful it is, and no matter that you built it.

Everything I'd apply to a third-party service applies to my own assistant:

• Least privilege for the agent itself. The assistant never needed write access to its own credentials. Although I admit... I really wanted it to, cause freedom for me etc... Known-good secrets now live outside the agent's writable surface, in a root-owned reference file the agent cannot modify, that I had it create. I wanted an easier (for me) dashboard to give it the API's and other "secrets." The agent can read them; it just cannot make changes to it.

• Immutable configuration. Configs the agent depends on but shouldn't manage are read-only to the agent. If a process doesn't need to write it, it can't write it. This is CIS-hardening thinking applied one layer up the stack.

• Detection, not just prevention. The watchdog evolved from "auto-repair the key" to "alert me when anything writes to files nothing should be writing to." File integrity monitoring for a hobby VPS sounds like overkill until your own agent is the insider threat :/.

• Assume key exposure, rotate accordingly. During the debugging session, keys got pasted into places keys get pasted when it's latenight zzz. They were rotated... um eventually. If a credential has touched a chat window, a log, or a screenshot, it's burned - that rule doesn't relax because it's a personal project. (shhhh) 🤫🤫

Why this matters beyond my cloud basement VPS

Every company is currently wiring AI agents into real infrastructure - ticketing systems, runbooks, CI pipelines, cloud consoles. The industry conversation is mostly about what agents can do. The operational conversation needs to be about what they can mess up.

My assistant wasn't/isn't malicious (we hope). It wasn't compromised. It was doing exactly what its code said, with more authority than it needed (sadly) - which is the same root cause behind most insider incidents I've seen. The fact that the insider is a language model changes the tooling, not the principle: If modern architectures assume no implicit trust for users and devices, AI agents should be governed tools within that system - not exempt operators. That's the design principle I now start from, at home and at work.

What I'd do differently from day one

• Secrets injected at runtime, never stored in agent-writable config.
• Agent runs as a dedicated low-privilege user with an explicit write allowlist.
• File integrity monitoring on config paths from the start.
• A known-good config in version control, so recovery is git checkout, not archaeology.
• Rotate any key the moment it touches a screen that scrolls.

None of that is exotic. It's the same discipline we apply to production servers - the only new part was admitting my helpful little assistant deserved the same suspicion as everything else on the wire.

What's in this repo

ai-agent-zero-trust/
├── README.md            # this postmortem
├── .gitignore           # keeps secrets out of the repo
├── keyguard_watchdog.sh # cron watchdog: detect + restore + alert
└── config_patch.py      # safe, atomic config repair

Both scripts are sanitized reference implementations of the approach described above - generic paths, placeholder variable names, no real credentials, no real hostnames. Adapt to your own stack; read every line before running anything from the internet as root, including this. (Especially this. That's the whole point of the article.)

Usage

# 1. Store the known-good key OUTSIDE the agent's writable surface (root-owned, 600)
echo "sk-your-real-key" | sudo tee /etc/keyguard/reference.key
sudo chmod 600 /etc/keyguard/reference.key

# 2. Test the watchdog manually
sudo ./keyguard_watchdog.sh

# 3. Schedule it (every 5 minutes)
# crontab -e (as root):
*/5 * * * * /opt/ai-agent-zero-trust/keyguard_watchdog.sh >> /var/log/keyguard.log 2>&1

I'm a San Antonio-based systems administrator working at the intersection of infrastructure, security, and AI operations. Next up: how I turned 668 screenshots of after-hours incident response into an automated evidence pipeline with PowerShell and Python.