Compact Video Metadata Serialization With Protobuf Across Services

The 40KB Problem Nobody Noticed Until Cloudflare Billed Us Every viral video we ingest at ViralVidVault crosses three service boundaries before it ever reaches a user. A PHP 8.4 ingestion worker pulls the raw metadata, hands it to a Go trend-scoring service, which in turn feeds a Python analytics pipeline that computes velocity and acceleration curves for the European feeds. For a long time these three services spoke JSON to each other, because JSON is what everyone reaches for first. It worked, it was debuggable, and nobody questioned it. Then we looked at the numbers. A single enriched VideoMetadata record - title, tags, region, published timestamp, view counts, and a nested block of trend signals - averaged just under 40KB as pretty-printed JSON, and around 28KB minified. Multiply that by the ~2.1 million records that move between services every day during a trend spike, and the internal egress alone was measurably showing up on our Cloudflare bill. Worse, the Go service was burning real CPU on encoding/json reflection, and the PHP worker spent more time in json_encode than it did doing the actual HTTP fetch. We migrated the inter-service contract to Protocol Buffers. Payloads dropped by roughly 68%, parse CPU on the Go side fell by more than half, and - the part I care about most as someone shipping under GDPR - the schema became a single enforced contract instead of a loose bag of keys. This is the write-up I wish I'd had before starting. If you want to see the end result in production, it's the discovery engine behind ViralVidVault, our GDPR-compliant European viral video tracker. Why JSON Was Costing Us More Than Bytes The byte count is the obvious problem, but it wasn't the expensive one. Three things hurt more than payload size: - No schema enforcement. A field renamed in the PHP worker would silently become null in the Go consumer. We caught these in production, not in review. - Type ambiguity. JSON numbers are all doubles. A view_count of 4,300,000,000 quietly lost precision once it crossed 2^53, which for a genuinely viral clip is not hypothetical. - Reflection cost. Both encoding/json in Go andjson_encode in PHP walk the structure at runtime. At our volumes that reflection was a real slice of CPU, and CPU on our LiteSpeed origin is finite. Protobuf addresses all three at once: a compiled schema, explicit integer widths, and generated marshalling code that doesn't reflect at runtime. The tradeoff is that the wire format is no longer human-readable, which matters less than you'd think once you have decent tooling. Defining the Schema Everything starts with the .proto file. This is the single source of truth that all three languages compile against. I keep it in a dedicated contracts/ repository that the PHP, Go, and Python services each vendor in, so nobody can drift. syntax = "proto3"; package viralvidvault.metadata.v1; message VideoMetadata { string video_id = 1; string title = 2; uint32 duration_seconds = 3; uint64 view_count = 4; Region region = 5; repeated string tags = 6; int64 published_at_unix = 7; TrendSignals signals = 8; bool gdpr_pii_stripped = 9; } enum Region { REGION_UNSPECIFIED = 0; REGION_DE = 1; REGION_FR = 2; REGION_ES = 3; REGION_IT = 4; REGION_NL = 5; REGION_PL = 6; } message TrendSignals { float velocity = 1; // views/hour, normalised float acceleration = 2; // d(velocity)/dt uint32 shares_per_hour = 3; } A few deliberate choices worth calling out, because they are the ones that bite people later: - Field numbers are permanent. The number 4 is what goes on the wire, not the nameview_count . You can rename the field freely, but never reuse or renumber a tag. If you retire a field, mark itreserved . - uint64 for view counts. This is the precision fix. Protobuf varints encode small numbers in one byte and only grow as the value grows, so you pay nothing for the wide type until you actually need it. - The region enum is versioned by thev1 package. European regions rarely change, but the enum's zero value beingREGION_UNSPECIFIED means an unset region is unambiguous rather than defaulting to a real country. - gdpr_pii_stripped is a boolean gate. Our contract says no record leaves the ingestion worker for analytics unless PII has been removed, and this flag is asserted downstream. More on that below. Enum choice matters for size too: an enum is a varint on the wire, so REGION_DE costs one byte, versus the string "DE" costing three plus framing in JSON. Encoding in PHP 8.4 Our ingestion worker is PHP. The official google/protobuf package ships a pure-PHP runtime, but for our throughput I strongly recommend installing the protobuf C extension as well - the pure-PHP path is correct but noticeably slower on hot loops. With protoc and the PHP plugin you generate classes under a namespace that mirrors the package. setVelocity((float) $row['velocity']) ->setAcceleration((float) $row['acceleration']) ->setSharesPerHour((int) $row['shares_per_hour']); $meta = (new VideoMetadata()) ->setVideoId($row['video_id']) ->setTitle($row['title']) ->setDurationSeconds((int) $row['duration_seconds']) ->setViewCount((int) $row['view_count']) ->setRegion(Region::REGION_DE) ->setPublishedAtUnix((int) $row['published_at_unix']) ->setGdprPiiStripped(true) ->setSignals($signals); // repeated fields take an array; the runtime handles the packing $meta->setTags($row['tags'] ?? []); // returns the compact binary wire format, ready for the next hop return $meta->serializeToString(); } // Decoding a payload that came back from another service: function parseMetadata(string $binary): VideoMetadata { $meta = new VideoMetadata(); $meta->mergeFromString($binary); // throws on malformed input return $meta; } Two practical notes. First, serializeToString() gives you the raw binary - you send it as the request body with Content-Type: application/x-protobuf , not as a JSON string. Second, mergeFromString() throws on genuinely malformed bytes but happily ignores unknown fields, which is exactly the forward-compatibility behaviour you want during a rolling deploy where the encoder is a version ahead of the decoder. One PHP-8.4-specific gotcha: int in PHP is a 64-bit signed integer on any 64-bit build, so a uint64 view count near the top of the range round-trips fine as a native int. If you ever run on a 32-bit SAPI, the runtime falls back to string representation for large integers - worth an assertion in your tests if you can't guarantee the build. Consuming in Go The trend-scoring service is Go, and this is where the CPU win showed up most clearly. Generated code plus google.golang.org/protobuf/proto gives you zero-reflection unmarshalling. package scoring import ( "fmt" "google.golang.org/protobuf/proto" pb "github.com/viralvidvault/contracts/gen/go/metadata/v1" ) // DecodeAndScore unmarshals a wire payload and rejects anything that // has not been through the GDPR PII strip in the ingestion worker. func DecodeAndScore(payload []byte) (float64, error) { var meta pb.VideoMetadata if err := proto.Unmarshal(payload, &meta); err != nil { return 0, fmt.Errorf("decode metadata: %w", err) } if !meta.GetGdprPiiStripped() { return 0, fmt.Errorf("refusing record %s: pii not stripped", meta.GetVideoId()) } sig := meta.GetSignals() if sig == nil { return 0, nil // no signals yet, score is zero } // A cheap composite score; the real one is a weighted model. score := float64(sig.GetVelocity())*0.6 + float64(sig.GetAcceleration())*0.3 + float64(sig.GetSharesPerHour())0.1 return score, nil } The Get accessors are the important habit here: they are nil-safe. meta.GetSignals() on a message where signals was never set returns a typed nil pointer, and sig.GetVelocity() on that nil returns the zero value rather than panicking. This is proto3's answer to the JSON null problem - there's a defined default for every scalar, so downstream code doesn't need defensive existence checks scattered everywhere. Benchmarked against our old json.Unmarshal path over a representative sample, the protobuf decode was consistently 2-3x faster and allocated far less, because there's no map construction and no reflection walk. Storing the Binary in SQLite WAL Our origin uses SQLite in WAL mode as the local cache on each LiteSpeed node. Here's a pattern that surprised people on my team: you don't have to unpack protobuf to store it. The compact binary is a perfectly good BLOB , and SQLite treats it as an opaque byte string. import sqlite3 from viralvidvault.metadata.v1 import video_metadata_pb2 con = sqlite3.connect("cache.db") con.execute("PRAGMA journal_mode=WAL") con.execute( "CREATE TABLE IF NOT EXISTS video_meta (" " video_id TEXT PRIMARY KEY," " region INTEGER," # denormalised for WHERE filters " payload BLOB NOT NULL" # the raw protobuf bytes ")" ) def store(payload: bytes) -> None: meta = video_metadata_pb2.VideoMetadata() meta.ParseFromString(payload) # validate before we trust it con.execute( "INSERT OR REPLACE INTO video_meta (video_id, region, payload) VALUES (?, ?, ?)", (meta.video_id, meta.region, payload), ) con.commit() def load(video_id: str) -> video_metadata_pb2.VideoMetadata | None: row = con.execute( "SELECT payload FROM video_meta WHERE video_id = ?", (video_id,) ).fetchone() if row is None: return None meta = video_metadata_pb2.VideoMetadata() meta.ParseFromString(row[0]) return meta The trick is denormalising the couple of fields you actually filter on - here region - into real columns while keeping the full record as an opaque blob. You get indexable queries on the hot dimensions and a compact single-blob store for everything else, and the blob is already in the exact format you'll ship to the next service. No re-encoding on read. In WAL mode these blob writes don't block concurrent readers, which keeps the analytics pipeline from stalling the ingestion path. Decoding at the Edge in a Cloudflare Worker Because our audience is European and latency-sensitive, some responses are assembled at the Cloudflare edge. Protobuf travels well here t