Four ways a file-sync tool eats your data, and how to not

Introduction

I've been building a small sync tool - it moves files between a remote project and a local folder, git-style (clone, pull, push, status). The moment you write a tool that touches other people's files on both sides of a network, you inherit a category of bug that's worse than a crash: silently destroying work the user can't get back.

Here are four data-loss traps I hit (or nearly hit), each with the fix. None are exotic. All of them pass a naive test suite while being wrong.

1. os.WriteFile truncates before it writes

The obvious way to save a downloaded file:

os . WriteFile ( path , data , 0 o644 )

O_TRUNC means this empties the file, then writes. If the process is killed between those two steps - Ctrl-C, OOM, a laptop lid - you're left with a dense prefix of the new bytes and none of the old. The user's file is now half a file.

The fix is the same trick databases use: write to a temp file beside the destination, then rename:

tmp , _ := os . CreateTemp ( filepath . Dir ( path ), ".tmp-*" )
tmp . Write ( data )
tmp . Close ()
os . Rename ( tmp . Name (), path ) // atomic on the same filesystem

rename(2) is atomic within a filesystem, so any reader sees either the whole old file or the whole new one, never a fragment. Two things that bite here: the temp file must be in the same directory (a temp in /tmp gives you EXDEV: cross-device link the moment someone's on a different mount), and rename swaps the inode, so hard links and xattrs on the destination don't survive - an accepted cost, but know you're paying it.

The second-order damage is the sneaky part. If your write happens before you record the file in your local ledger, then a killed write leaves a fragment your tool doesn't know it wrote. Next run, the tool compares the fragment against the server, sees a difference, and reports "local file changed - use --force to overwrite." You corrupted the file, then blamed the user, then offered them the one destructive flag as the cure. Atomic writes kill the whole chain.

2. Conflict detection on timestamps or etags can't see the case that matters

The tempting cheap conflict check: compare the server's etag (or mtime) to the one you saved. Different? Re-download. The case this misses is the only one that actually loses data: both sides changed. If the user edited locally and the file changed on the server, an etag comparison just sees "server differs" and happily overwrites the local edit. Gone.

Conflicts have to be decided on the bytes, not on metadata:

• local hash == last-synced hash → clean, take the remote
• local hash != last-synced hash and remote changed → conflict, touch nothing
• only one side changed → apply that side

Metadata is a fine fast path to skip work, but it can never be the thing that authorizes an overwrite. The authorization is the hash.

3. "Delete what's not on the server" deletes things you didn't sync

A --prune/mirror flag is where sync tools cause the worst damage, because deletion is the one operation with no undo. The naive rule - "if it's local but not remote, delete it" - will cheerfully remove:

• a file the user created and never synced (it was never yours to delete)
• a file the user edited locally (that's a conflict, not a deletion)
• a file that legitimately only ever lived locally

The rule has to be: delete only what you can prove you put there and the user didn't touch. That proof is your ledger - the record of "I wrote these exact bytes at this version."

• No ledger entry → not yours → leave it.
• Ledger entry but local bytes differ → conflict → leave it.

And a subtle one: if the operation is interrupted halfway, a naive tool sees a partial local tree and concludes the missing files were "deleted on the server," then prunes the real ones. An interrupted run has to record itself as a failure, not as a short success, or your prune logic reads a crash as a batch of user deletions.

4. When the data is a shape you don't recognize, refuse - don't guess

The tool reads a file listing from the server to decide what to sync. What happens when that listing comes back as null because of a server hiccup? In Go, json.Unmarshal("null", &slice) succeeds and leaves you with a nil slice and no error. Unguarded, your code now believes the project has zero files - and on the next push --prune, "zero files on the server" means "delete everything locally." A transient null just wiped the user's tree.

The defense is a mindset, not a line: untrusted input that will drive a destructive action must be validated for shape, and anything unexpected must stop the operation, not flow through it. I explicitly reject a null listing, a listing that hit the pagination cap (at the boundary, a complete listing and a truncated one are identical - so I refuse rather than risk it), and any decode whose length disagrees with the size the server advertised. A needless refusal costs one retry. A wrongly-trusted listing costs the user's files.

The through-line

All four bugs pass a happy-path test. You only catch them by asking a different question than "does it work?" - namely "if this dies right now, or lies to me right now, what does the user lose?" For anything that writes files or deletes them, that question is the design.

I wrote these up because the tool is open source and the reasoning is the interesting part - it's a single Go binary, standard library only, and every one of these decisions has a test pinning it (including a few that only exist because a mutation showed the guard was silently not guarding). If you want to see them in situ: https://github.com/somework/dsx

What's your worst "the tool ate my files" story? I collect them.