Private Access for Uploaded Documents in Startup Apps: A Quarantine-First Design
Short answer: keep uploaded documents in a private bucket, make the application the authorization authority, and issue short-lived signed URLs only after it confirms ownership and document state.
That decision is less about picking the βsimplestβ S3-compatible alternative and more about keeping the trust boundary small. The bucket stores bytes. The Python app decides who can ask for them. A quarantine step prevents an upload that merely exists from becoming a document the rest of the product can process.
What should a startup app verify before granting private document access?
The data flow is straightforward. A browser asks the API for an upload intent; the API authenticates the user, creates an opaque object key, records a pending row, and returns a constrained upload grant. A worker checks the completed object for size, media type, and application policy before marking it ready. A download follows the reverse path: the API checks the caller against the row, then signs a GET for a short window.
A signed URL is a bearer capability. Anyone who gets the string can use it until expiry, so query strings belong in neither analytics events nor ordinary request logs. Store the stable object key, never the URL, and mint a fresh URL after each authorization decision. Keep the bucket private even if every current caller is βtrusted.β No public shortcut.
My eval harness treats these as invariants:
- Alice can retrieve her ready document;
- Bob cannot obtain Aliceβs URL;
- an expired URL is rejected;
- a pending object is not downloadable;
- duplicate completion events are harmless;
- and deleting the application row stops new URL issuance.
I add a zero-byte upload, a filename containing quotes and non-ASCII characters, a client retry, and a 25 MB boundary case. Your mileage may vary on expiry duration because mobile networks differ, but the ownership check should not.
The application owns identity, metadata, lifecycle, and audit events. The object service owns durable bytes and signature validation. A worker can inspect content, but it never grants user access. This separation also makes an S3-compatible alternative useful: the protocol is an adapter boundary, not the policy engine.
Put a small Python boundary in front of the object service
The example below uses the standard S3 signing shape so the application boundary stays visible. The endpoint is configuration; business code deals in document IDs. Direct browser uploads keep large bodies out of the API process.
import os
import sqlite3
import uuid
from contextlib import closing
import boto3
from fastapi import Depends, FastAPI, HTTPException
from pydantic import BaseModel
app = FastAPI()
s3 = boto3.client(
"s3",
endpoint_url=os.environ.get("S3_ENDPOINT_URL"),
region_name=os.environ.get("S3_REGION", "us-east-1"),
)
BUCKET = os.environ["DOCUMENT_BUCKET"]
DB_PATH = os.environ.get("DOCUMENT_DB", "documents.db")
def current_user_id() -> str:
# Replace with the app's verified session or token identity.
return "user_123"
def initialize() -> None:
with closing(sqlite3.connect(DB_PATH)) as db:
db.execute(
"CREATE TABLE IF NOT EXISTS documents "
"(id TEXT PRIMARY KEY, owner_id TEXT, object_key TEXT, status TEXT)"
)
class UploadRequest(BaseModel):
content_type: str
@app.post("/documents/upload-intent")
def create_upload_intent(request: UploadRequest, user_id: str = Depends(current_user_id)) -> dict:
document_id = str(uuid.uuid4())
object_key = f"users/{user_id}/{document_id}"
with closing(sqlite3.connect(DB_PATH)) as db:
db.execute(
"INSERT INTO documents VALUES (?, ?, ?, 'pending')",
(document_id, user_id, object_key),
)
db.commit()
upload = s3.generate_presigned_post(
Bucket=BUCKET,
Key=object_key,
Fields={"Content-Type": request.content_type},
Conditions=[
{"Content-Type": request.content_type},
["content-length-range", 1, 25 * 1024 * 1024],
],
ExpiresIn=300,
)
return {"document_id": document_id, "upload": upload}
@app.get("/documents/{document_id}/download-intent")
def create_download_intent(document_id: str, user_id: str = Depends(current_user_id)) -> dict:
with closing(sqlite3.connect(DB_PATH)) as db:
row = db.execute(
"SELECT object_key, status FROM documents "
"WHERE id = ? AND owner_id = ?",
(document_id, user_id),
).fetchone()
if row is None or row[1] != "ready":
raise HTTPException(status_code=404, detail="Document not found")
url = s3.generate_presigned_url(
"get_object",
Params={
"Bucket": BUCKET,
"Key": row[0],
"ResponseContentDisposition": 'attachment; filename="document"',
},
ExpiresIn=120,
)
return {"url": url}
initialize()
The completion handler or verification worker should make the pending-to-ready transition idempotent after confirming that the object exists and satisfies policy. A real identity dependency must derive user_id from a verified credential, never from request JSON. Returning the same 404 for a missing row and a row owned by someone else avoids turning document IDs into an ownership probe.
Keep it small.
How do private document pipelines fail under retry pressure?
Successful PUT and GET tests are only the first mile. I record request count, transferred bytes, stored bytes by lifecycle state, signing latency, verification delay, and abandoned pending uploads. Then I replay retries with forced client timeouts.
An upload can succeed while the browser misses the response, so completion must be idempotent and reconciliation must find old pending rows. Deletion gets the same treatment: record intent, remove the object, verify the result, and retain only the audit data policy requires.
I learned the cost side with a painful eval run. I expected roughly $70 for a document-processing experiment and received a $312 bill because my retry loop downloaded the same source documents before every prompt evaluation. Token spend got my attention first; repeated object transfer was the quiet multiplier. I changed the harness to hash source objects, cache extracted text, cap retries, and report bytes per successful eval case. Now a green accuracy score ships with a cost trace.
Measure it. Prices change, and I'm not sure why teams paste a single storage rate into architecture documents without recording its date. Compare capacity over time, writes and reads, internet download, minimum retention, early deletion, replication, and support. A public pricing page can show the categories, but your trace is the evidence that matters.
Failure tests should include:
- an expired signature,
- an upload over the signed limit,
- a content-type mismatch,
- a lost completion callback,
- a deleted object with stale metadata,
- and concurrent deletion plus download authorization.
I use fake clocks for expiry and a disposable bucket for integration tests - unit mocks can't prove that signature fields and browser behavior agree.
How should a team choose boundaries before comparing options?
S3 compatibility reduces client work, but services can still differ operationally. I run one Python suite against every candidate endpoint and keep the results beside the decision. The catch is that direct-to-bucket uploads are not suitable when every byte must pass synchronously through application-controlled inspection; proxy the stream or quarantine it, and accept the extra bandwidth and operational load. Stick with an application-mediated download when policy requires a decision during transfer, not only when access begins.
Decision axis:
- Authorization - What I verify: Bucket is private; app checks tenant before signing. Reject when: Public URLs or object keys act as authorization.
- Upload constraints - What I verify: Signature binds key, expiry, size, and expected type. Reject when: Client can write outside its assigned prefix.
- Download behavior - What I verify: Expiry and response headers behave as tested. Reject when: Links last longer than the business action.
- Compatibility - What I verify: Exact operations and signing mode pass integration tests. Reject when: Business code needs endpoint-specific branches.
- Recovery - What I verify: Retry, reconciliation, deletion, and restore drills repeat. Reject when: Operators repair metadata by hand.
- Economics - What I verify: Measured bytes, requests, retention, and download fit forecasts. Reject when: Estimates depend on unmeasured traffic.
Content-Disposition: attachment asks a browser to download content, while filename parameters influence the suggested name; MDN documents the syntax and browser behavior. It is not authorization. Keep stored keys opaque, sanitize any displayed original filename separately, and test a safe fallback name.
An adapter has a limit. If the product depends on endpoint-specific eventing, edge authorization, object locking, or replication semantics, three generic methods can hide a requirement. Expose a capability when the application truly depends on it, and put that dependency in the eval suite. Portability is earned by tested contracts.
Turn the storage decision into evidence another engineer can rerun
Before rollout, I want a review artifact with configuration that contains no embedded credentials, a private-bucket policy check, cross-tenant denial tests, expiry tests with a controlled clock, upload-size enforcement, content verification, idempotent completion, reconciliation for abandoned rows, deletion and restore drills, redaction of URL query strings, and dashboards for bytes, operations, latency, and pending age. I also rotate signing credentials in staging and confirm that new credentials work without changing stored document records.
The application database should expose lifecycle states such as pending, ready, quarantined, and deleting. Avoid a vague uploaded Boolean. Each transition needs an owner, an idempotency rule, and an alert threshold. For AI document features, record the content hash, extraction version, and evaluation dataset version with derived data, rather than copying the original file or confusing an embedding with the user-owned source. That makes prompt-cost regressions explainable.
One final check is manual. I copy a freshly issued link into a clean browser profile, verify the download name, wait through expiry, and inspect logs to confirm the query string was not captured. Then I repeat as a second tenant. Two minutes of awkward clicking has caught configuration drift that a mocked test missed.
The result isn't a universal winner. It is a narrow, observable storage contract that can survive a vendor change, a retry storm, and an access review. For a startup app storing private user documents, that is the useful definition of simple.
Comments
No comments yet. Start the discussion.