Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned

Architecture Deep-Dive: Building for Streaming and Scale

Streaming-First: Why Real-Time Matters

Traditional service topology systems use batch processing, aggregating data hourly or daily, then storing complete snapshots. This approach works at a modest scale but has a fundamental problem: by the time you see the data, it's already old. During a production incident at 3am, an hour-old dependency map is archaeology, not observability.

Our key architectural decision was to build streaming-first. Instead of batch jobs that process historical data, we continuously ingest flow records from multi-region Kafka streams and IPC metrics as Server-Sent Events, process them through reactive pipelines with backpressure handling, and provide near real-time topology updates, typically within tens of minutes, compared to the hours-old or day-old data that batch processing approaches provide.

This wasn't just about freshness - it was essential for our use cases. Live events can't wait for the next hourly batch. Incident response needs current data. Change validation requires seeing immediate impact. The architecture had to support continuous updates while handling massive scale without falling behind.

How Backpressure Enables Real-Time Processing

The streaming approach created new challenges, but also required solving a fundamental problem: how do you process millions of flow records per second in real-time without losing data when downstream systems slow down?

Traditional approaches fall short at our scale:

• Unbounded queues: Simple but dangerous. Keep buffering until you run out of memory, then the instance crashes.
• Drop-based flow control: Discard data when buffers fill. Fast, but now your topology is incomplete - you've lost connection information.
• Batch processing: Process everything, but hours later. By then, the incident is over (or worse, still happening with stale data).

We needed something different: the ability to slow down gracefully under load without losing data. This is where reactive streams with backpressure became essential.

Here's how it works: when Stage 3 can't write to the graph database fast enough, it signals Stage 2 to slow down. Stage 2 signals Stage 1. Stage 1 signals the Kafka consumer to pause. The data waits in Kafka until downstream capacity returns. When a downstream stage can't keep up, it signals upstream to slow down - backpressure flows in the opposite direction of the data.

Backpressure propagates naturally through the entire system. When any stage becomes overwhelmed from traffic spikes, GC pauses, or external slowdowns, the pipeline automatically slows to a sustainable rate. No data is lost in most cases, no instances crash - the system degrades gracefully.

This is what enables "real-time" at our scale. During normal operation, we process with minimal latency. During load spikes or temporary slowdowns, we slow down rather than fall over. The data still gets processed, just a few seconds or minutes later instead of immediately. For topology updates, this trade-off is acceptable: slightly delayed real-time updates are vastly better than hour-old batch data or incomplete topology from dropped records.

The cost of this approach is complexity. Reactive streams are harder to reason about compared to traditional synchronous blocking models (we'll discuss this more in the challenges section). But at Netflix scale, backpressure isn't optional - it's the mechanism that keeps the system running reliably under production load.

Multi-Layer Architecture: Physical Separation for Independent Optimization

As we covered in our first post, our multi-source approach uses three physically separate topology layers with different storage optimized for each:

• Network Layer: eBPF flow logs in graph database partition - comprehensive coverage but lacks application context
• IPC Layer: Application metrics in a different graph database isolated from the one for Network Layer - rich endpoint details but only instrumented services
• Tracing Layer: Distributed traces in columnar storage (Parquet) - actual request paths but sampled (we cover the tracing layer and its integration in our next post)

Flow logs and IPC metrics travel through two independently-optimized pipelines into separate graph stores, unified behind a single API. Physical storage isolation enables independent optimization - each layer has different throughput, query patterns, and evolution timelines. At query time, we execute parallel queries across relevant storage systems and merge results, providing unified views with sub-second latency while maintaining flexibility to evolve each layer independently.

The Three-Stage Distributed Aggregation Pipeline

The heart of the network layer ingestion is a three-stage distributed pipeline. This architecture solves a fundamental challenge with network flow logs: they only show individual network hops, not the true application-level connections we need to build a useful topology.

The Core Problem: Network Intermediaries

In cloud environments, traffic between applications rarely flows directly - it traverses intermediate network components like load balancers, NAT gateways, API gateways, and proxies. Network flow logs show individual hops: App A → Load Balancer and Load Balancer → App B appear as separate flows. But what engineers need is the logical dependency: App A → App B. Without resolving these intermediaries, our topology would be cluttered with infrastructure components rather than showing the service-to-service relationships that matter for troubleshooting.

The three-stage pipeline solves this.

Stage 1: Initial Aggregation (FlowLog Ingestion Service)

Multi-Region Kafka (4 regions) → Filter invalid flow logs → 5-minute time-window batching → Create initial aggregators per window → Distribute via consistent hashing → Stream to Stage 2 via SSE

Stage 1 consumes flow logs from multi-region Kafka, filters invalid records, batches them into 5-minute time windows, and creates initial aggregator objects. At this stage, we're still working with raw network hops, identifying which flows involve intermediaries but not yet resolving them. Aggregators stream to Stage 2 for resolution.

Stage 2: Network Intermediary Resolution Layer (Intermediate GraphEntity Ingestion Service)

Stage 1 Aggregators (via SSE streams) → Group flows by intermediary (load balancer, NAT gateway, proxy, etc.) → Identify pairs: (Source → Intermediary) + (Intermediary → Destination) → Resolve to direct edges: Source → Destination → Track which intermediaries were traversed → Aggregate metrics across both hops → Re-distribute via consistent hashing → Stream to Stage 3 via SSE

This is the key step. Stage 2 performs graph resolution:

• Collect flows by intermediary: Group aggregators where an intermediary is either source or destination, creating maps of flows going TO intermediaries (Source → Intermediary) and FROM intermediaries (Intermediary → Destination)
• Resolve direct edges: For each intermediary, join its incoming and outgoing flows to create direct application edges (App A → App B), combining metrics from both hops
• Result: Clean application-level topology showing App A → App B instead of App A → Load Balancer → App B

This resolution happens at aggregation time, not query time, with resolved edges flowing to Stage 3.

Why can't we do this in a single stage? The fundamental issue is data locality. To resolve App A → Load Balancer → App B into App A → App B, we need both flows on the same instance to perform the join. But in Stage 1, flows are scattered across instances based on Kafka's partitioning. Stage 2's critical function is to redistribute aggregators by intermediary identifier - all flows involving "Load Balancer X" route to the same instance for resolution. This is the classic map-reduce pattern: Stage 1 maps, Stage 2 shuffles and reduces by intermediary, Stage 3 performs final aggregation.

Stage 3: Final Aggregation and Enrichment (GraphEntity Ingestion Service)

Stage 2 Aggregators (via SSE streams) → Final aggregation across time windows → Enrich with external data (query key-value stores) → Convert to graph entities → Persist to graph database (throttled writes)

Stage 3 performs final aggregation of resolved edges, enriches graph nodes with external data sources (application health, ownership, metadata), converts aggregators to concrete graph entities (nodes and edges with all properties populated), and persists them to the distributed graph database with controlled throttling to respect storage system limits.

Why Three Stages, Not Two?

We initially used two stages: aggregate in Stage 1, resolve and persist in Stage 2. This worked in testing but failed at production scale - Stage 2 became overwhelmed by data concentration.

The problem: intermediary resolution requires collecting ALL flows involving an intermediary on the same instance. As a result, the instances handling flow logs for popular applications and their intermediaries became 'hot nodes' due to significant data concentration. Compounding this, data enrichment (querying external stores for health and metadata) meant the busiest instances were also doing the most I/O.

The solution: split responsibilities into three stages. Stage 2 focuses purely on resolution and redistributes. Stage 3 handles enrichment and persistence. This graduated redistribution (distribute, resolve, redistribute, persist) spreads load across multiple instances and isolates compute-heavy resolution from I/O-heavy enrichment. Even when intermediaries see 100x typical traffic, no single instance becomes a bottleneck.

Why Server-Sent Events Instead of gRPC or Message Queues?

We initially used gRPC but it became a performance bottleneck - serialization overhead, connection pool management, and memory pressure for streaming responses consumed more CPU than business logic. Message queues added infrastructure complexity without benefit for our use case.

SSE proved ideal: lightweight HTTP-based protocol with minimal serialization, natural backpressure integration with reactive streams, and simpler connection model. The lesson: industry best practices like "use gRPC for service communication" don't apply universally. For streaming large volumes of pre-aggregated data, lighter-weight alternatives may be more appropriate. Measure, don't assume.

Why IPC Doesn't Need Three Stages

The IPC pipeline mirrors the same pattern as the flow log pipeline, but needs only a single stage. The IPC layer uses single-stage aggregation because:

1. IPC metrics are already at application level - no intermediaries to resolve
2. Data is partitioned correctly from the start - each node receives all IPC metrics for its assigned applications via consistent hashing, eliminating the need for redistribution

This highlights a key principle: data partitioning strategy determines processing architecture. When data arrives with the right partitioning, you can aggregate directly; when it doesn't (like network flows requiring intermediary resolution), you need shuffle/redistribution stages.

Dynamic Load Distribution: How Hashing Works with Auto-Scaling

How do we decide which instance receives which aggregator when our Auto Scaling Groups dynamically add or remove instances? Traditional approaches assume static clusters requiring explicit rebalancing, coordination services, or manual data movement when cluster size changes.

Our Approach: Dynamic Consistent Hashing

We use consistent hashing with dynamic instance discovery from our service registry. Each instance queries the registry to get the current list of healthy ASG instances, maintains them in sorted order (ensuring all instances have the same view), and uses this list for the hash function findOwnerInstance(aggregator.primaryKey). When ASG scales up or down, the hash function naturally redistributes aggregators based on the updated instance list - no explicit coordination needed.