The three lives of my JWT signing key

One private key signs every token our auth server issues. Lose it and an attacker mints a valid token for any user, no password required, and your logs show a clean login. It is the single most valuable secret in the system, and over its life it has moved twice: it was born inside the application process, then exiled to a vault it can no longer leave, then reincarnated as a different kind of key entirely.

Each move happened on a live issuer with tokens already in flight and validators caching state we don't control. This is what those moves actually take, and the one that taught us the most was the one we didn't announce.

Life one: the key the server kept in its pocket

In the beginning the key lived where it was used. The server generated an RSA-2048 key, held it in memory, and signed its tokens with RS256. This is the default almost everywhere, and it is fine right up until you say the word custody out loud.

The private key was reachable by anything that could read the process's memory, its configuration, or a backup of either. We wanted to be able to tell customers that a leaked database exposes nothing useful, and a signing key sitting one process away from that database made the claim a lie. The key had to leave the building.

Life two: the key that moved into a vault it can't leave

We introduced a seam - an ISignatureProvider the server calls whenever it needs a signature - and put a KMS behind it. The key is now generated inside Vault's transit engine and never exported. The application doesn't hold it; it sends the bytes to be signed to the vault and gets a signature back. It can use the key and it cannot exfiltrate it. That distinction is the whole point.

A memory dump, a spilled config file, a leaked backup - none of them contain the key anymore, because the key was never in any of those places. Compromising the application now buys an attacker the ability to ask the vault to sign, which we can rate-limit, audit, and revoke. It no longer buys them the key itself, which we could do nothing about once it was gone.

Here is the part we didn't put in the release notes. The same commit that moved the key into the vault also quietly halved our PBKDF2 iteration count, from 100,000 down to 50,000. One diff, two security changes pointing in opposite directions: the headline was "sign in the KMS," and riding underneath it was a password-hashing work factor cut in half, in a line nobody was looking at because the story was about something else.

Both are security code, so both got reviewed as a single "more secure" change and waved through on the strength of the good half. The fix was one line. The lesson wasn't. A diff labelled more secure is still a diff, and the security-sensitive constants inside it - iteration counts, key sizes, timeouts, algorithm names - don't inherit a halo from the commit message. We now treat a work-factor number the same way we treat a choice of algorithm: something asserted and checked, not something you trust because the surrounding change was a good idea.

Life three: the key that got smaller to get faster

With signing behind a KMS, every signature is a network round-trip, and RSA signing is the expensive kind. So the key changed species: RS256 over RSA-2048 became ES256 over the P-256 curve. Elliptic-curve signing is roughly an order of magnitude cheaper than RSA, a P-256 signature is 64 bytes where RSA-2048's is 256, and the public key set validators download shrinks accordingly. Smaller key, smaller signature, smaller JWKS, faster tokens - all from picking a better curve.

The catch is that you cannot flip-swap the signing algorithm on a live issuer. Every token already issued was signed RS256 and has to keep validating until it expires. Every validator has your old public key cached. Change the algorithm in one move and you invalidate every token in flight and every cached key set at the same instant - a self-inflicted outage dressed up as an upgrade.

What worked was to stop pretending there was ever only one key. The key store went algorithm-agnostic: it holds the RSA key and the EC key at the same time, and publishes both in the JWKS, each under its own key id and algorithm. A separate active-key picker decides which key signs new tokens, and it rotates itself off the legacy RSA key the moment an EC key exists - no config flag, no deploy timed to line up with a rotation.

Through the overlap, discovery advertises both public keys, so tokens signed with either one keep validating. Once the last RS256 token has expired, discovery advertises only ES256 and validation pins ValidAlgorithms = ES256, which also slams the door on algorithm-confusion attacks that try to talk a validator into accepting the wrong scheme. The migration is a crossfade the issuer performs on itself, not a switch anyone throws.

The lesson, factored out

A signing key looks like a constant: one secret, set once, referenced forever. Ours wasn't. In its life it changed custody, from the process to a KMS, and it changed species, from RSA to elliptic curve, and both moves had to be made with tokens in flight and validators caching things we don't own.

The property that made each move survivable was the same one both times: the system never assumed there was exactly one key, one algorithm, or one home. Build the key store to hold several keys and to honestly advertise which ones it holds, and rotation - of where the key lives or of what kind of key it is - stops being an outage you schedule and becomes a property the issuer manages on its own.

If you run auth, the thing that signs your tokens should be the most boring, most inspectable, least clever object you own. Ours took three lives to get there. It was worth every one.