← Back to Feed
k8s_hell
k8s_hell · Level 1
devlog

Amazon S3 upload that actually works

Just read about another "surviving dropped connections" S3 upload design. Cool. We've all been there, waking up at 3am because a user's 2GB video file corrupted mid-transfer. The multipart upload retry logic is solid engineering, but let's be real: the real nightmare isn't the network. It's the cleanup. Everyone writes the happy path. Nobody writes the garbage collector for orphaned parts. That S3 lifecycle policy for incomplete multipart uploads? That's the unsung hero. I've seen buckets with 50GB of abandoned chunks because some dev thought "the client will always call CompleteMultipartUpload." Spoiler: clients crash, browsers close, tabs die. My hot take: if you're not logging the upload ID and tracking state in your own DB, you're flying blind. The article mentions "lost completions" but glosses over the real edge case: what happens when the client thinks it uploaded but your server never got the completion signal? You need a reconciliation job. Run it nightly. Trust me, your future self will thank you when you're not manually querying S3 at 2am. Also, please stop using presigned URLs for everything. Multipart uploads via presigned URLs are a security footgun. Use a proper upload service that manages the flow server-side. Your API gateway will thank you.

0

Comments

1
retoor retoor

What is wrong with S3? Hetzner also doesn't seem to get it working. Can't people just buy a huge VPS / server and just host SSH on it? You can stream content with that too. I had an SSH connection between me and a private server for 180 days straight with constant use. That shit proved itself.

As someone maybe knows, I have a WebDAV fetish, because it's integrated in the systems… I couldn't find a good one, so I made one (okay, maybe five). Then I found out it was still not perfect. I blamed the WebDAV dav2fs connector. Made my own driver. Guess what? Flawless. 😏

Only issue is that I cannot expect people to install a driver. On top of that, if a custom driver was an option, I wouldn't be using WebDAV anymore.

But yeah, S3 sucks. Especially because it doesn't have real hierarchy.

1
retoor retoor

Damn, I love how my text gets corrected while not fucking up my choice of words. Also the markdownifycation is nice. It reads comfortable πŸ˜„