DEV Community

Your backup quietly resurrects the users you deleted

The Problem: Incremental Backups Can't See Deletes

Every incremental backup makes the same quiet assumption: to back up what changed, find the rows that changed. On a key-value store like Azure Table Storage or DynamoDB, "what changed" means "rows whose last-modified timestamp is newer than my last watermark." Walk the table, grab everything newer, write it out. Fast, cheap, correct. Correct for writes.

Now delete a row. The row does not get a "deleted" flag. It does not get a newer timestamp. It does not move to a recycle bin. It just stops existing. These stores have no delete markers and no change feed for deletions, so a deleted row leaves behind exactly nothing.

Your incremental backup, which finds changes by scanning for newer timestamps, scans right past the empty space where the row used to be and finds nothing. There is nothing to find. The delete is invisible. Which means your backup still contains the row.

Where Data Loss Becomes a Security Story

Think about what a delete usually means in an auth system. You did not delete that user for fun:

  • You offboarded an employee.
  • You removed an account whose password showed up in a breach dump.
  • You revoked an OAuth grant after a token leaked.
  • You pulled an admin's access the day they left.

Every one of those is a security decision, and none of them survive a backup that cannot see deletes.

So you restore. Maybe from a real disaster, maybe just to a staging clone. The backup does exactly what you asked: it puts back every row it knows about. The offboarded employee is a user again. The breached password is valid again. The revoked grant works again. The removed admin has admin again.

Your restore did not lose data. It reverted your security decisions, silently, and handed you a system that looks healthy and is quietly compromised. A backup that forgets a delete is worse than no backup, because you trust it.

The Fix: Treat Deletes as Events

The fix is to stop treating a delete as the absence of a row and start treating it as an event. Deletes are data.

So we give deletes their own table. Every delete in the system goes through an injected ITombstoneWriter that records a tombstone: the key, and the time it died. There is no code path that deletes a row without leaving a tombstone, because the delete and the tombstone are one operation.

An incremental backup is then two parts captured at a single watermark:

  1. The upserts (rows with a newer timestamp)
  2. The tombstones (deletes since the last watermark)

Restore replays both in time order, so a delete applies just like a write, and a key that was deleted and later recreated resolves to whichever happened last. The empty space finally has a record attached to it.

That is the whole trick, and it is small. The reason it matters is not.

The Shape Outlives Any Single Store

The shape here outlives Table Storage and DynamoDB. Any system that models a delete as "the row is gone" cannot back up, replicate, or sync deletes, because there is nothing to carry. It shows up in:

  • Naive event replays that only replay creates and updates
  • Caches that expire but never invalidate
  • Read replicas that drift because the delete never propagated

And it is most dangerous exactly where deletes are how you enforce security, which in an identity system is everywhere: revocation, offboarding, rotation, lockout.

Audit Your Own Backups

If your durability story only tracks the things that exist, it will faithfully preserve the things you spent effort making not exist.

So audit your own backups with one question: if I delete a user right now, take a backup, and restore it, is that user gone?

If the honest answer is "I am not sure," your backup is a time machine pointed the wrong way. It does not protect you from your mistakes. It resurrects them: the password you rotated, the employee you offboarded, the token you revoked.

Deletes are data. Back them up like they are.

If you would rather your backups honor a delete without you having to prove they do, that is what a platform is for. Authagonal tombstones every delete, so a restore never brings back a revoked credential.

Comments

No comments yet. Start the discussion.