Service Topology: Real-Time Service Dependency Mapping

Netflix developed a real-time Service Topology map to address the challenge of understanding complex dependencies in their massive microservice ecosystem. Traditional observability tools provided fragmented views (metrics, logs, traces), but lacked a unified picture of how services interconnected at runtime. The Service Topology map aggregates data from three complementary sources: eBPF network flows (network layer), Inter-Process Communication (IPC) metrics (application layer), and end-to-end distributed tracing (request layer). This multi-layered approach provides a comprehensive view, enabling engineers to quickly answer critical questions like ‘Which services depend on each other?’, ‘What is the blast radius of a failure or maintenance?’, and ‘Where is the source of an issue?’. The system prioritizes real-time updates, fast queries, and rich context, integrating seamlessly with existing observability tools to aid in faster incident response, confident change planning, and ultimately, a more reliable member experience.

Learning state

Status: active Confidence: high

Key Takeaways

• Complex microservice architectures require a unified view of service dependencies beyond fragmented metrics, logs, and traces.
• A real-time service topology map is crucial for rapid troubleshooting, understanding blast radius, and planning changes.
• Aggregating data from multiple sources (network flows, IPC metrics, tracing) provides a more complete and accurate dependency picture.
• eBPF captures network-level communication, IPC metrics provide application-level endpoint details, and tracing shows actual request paths.
• The system must support fast queries, real-time updates, and rich context (health status, ownership) to be actionable.
• Visual and programmatic access is essential for both interactive exploration by engineers and automation by other systems.
• Understanding dependencies directly impacts member experience by enabling faster incident resolution and preventing disruptions.
• Future advancements include integrating change events and enabling automated root cause analysis.

The Observability Puzzle: Understanding Distributed Systems

In large-scale microservice environments, engineers often face the challenge of understanding intricate service dependencies, especially during critical incidents. Traditional observability tools like metrics, logs, and traces provide valuable but fragmented insights. Without a unified view of how services interact at runtime, troubleshooting becomes slow, stressful, and error-prone. Engineers frequently need to answer three fundamental questions: Which services depend on each other? What is the blast radius of a failure or planned downtime? Where is the root cause of an issue originating?

Examples

• An engineer paged at 3 am needs to quickly identify the impact of an elevated error rate in a critical service.
• Before performing maintenance on a service, teams need to know which other services or features will be affected.

Why Service Topology Matters at Scale

Netflix’s architecture, comprising thousands of microservices, enables rapid innovation but introduces significant observability challenges. Initiatives like live programming and ad-supported plans demand even faster troubleshooting and monitoring. Analysis of engineering support requests revealed a constant need for understanding upstream/downstream dependencies, pinpointing failure origins, and assessing maintenance impact. A real-time, comprehensive service map is essential to address these growing needs.

Lessons Learned from Previous Iterations

Netflix’s journey to building Service Topology involved exploring various approaches and learning key lessons:

• Real-time is critical: Stale dependency maps are ineffective in dynamic environments with frequent deployments.
• Scale is a fundamental challenge: Solutions must handle massive service counts and traffic volumes.
• Integration is key: The solution must seamlessly integrate with existing observability tools and workflows.
• Data quality is paramount: Inaccurate information leads to wrong conclusions.
• Multiple perspectives are necessary: No single data source provides a complete picture of dependencies.

Requirements for a Living Service Map

The ideal solution is a ‘living map’ that updates in real-time. Key requirements include:

• Real-time updates: Reflecting continuous deployments and traffic shifts.
• Fast queries at scale: Sub-second response times for traversing dependency graphs.
• Multiple layers: Showing network, application (API/endpoint), and request-level connections.
• Rich context: Overlaying health status, business domains, and ownership information.
• Visual and programmatic access: A UI for engineers and an API for automation.

Three Sources of Truth for a Unified View

Service Topology builds three complementary dependency graphs, each stored separately and queryable independently or merged into a unified view:

1. eBPF Network Flows (Network Layer): Captures actual network traffic, providing comprehensive coverage of all communication, regardless of application instrumentation. It shows cluster-level and app-level connections but lacks application context (e.g., specific API endpoints).
2. IPC Metrics (Application Layer): Collects metrics from instrumented services about calls made via gRPC, REST, etc. This layer offers rich application context, including endpoints, latency, and error rates, but only for instrumented services.
3. End-to-End Tracing (Request Layer): Follows individual requests through the system, showing actual runtime behavior and conditional logic. It provides valuable insights into common flows but is subject to sampling limitations.

Multi-Layer Architecture and Unified View

The power of Service Topology lies in combining these three distinct graphs. Engineers can view each layer independently or query them in parallel to generate a unified graph. This unified view leverages the strengths of each source: eBPF ensures completeness, IPC metrics provide application details, and tracing reveals actual runtime behavior. This comprehensive, accurate, and contextualized view allows exploration from multiple angles.

Examples

• A network engineer might focus on the eBPF network graph to understand inter-cluster communication.
• A service owner might use the IPC metrics graph to analyze endpoint-specific performance.
• An SRE might combine all three layers to trace a request failure from origin to impact.

Technical Architecture: From Flows to Graph

The system ingests multi-region Kafka flow logs and processes them using Apache Pekko Streams in a distributed, fault-tolerant pipeline. A three-stage aggregation process reconstructs direct application-to-application paths by resolving network intermediaries like load balancers and NAT gateways. The topology data is persisted in Netflix’s high-throughput graph database, optimized for fast multi-hop traversals. A gRPC API exposes the topology with features like filtering, pagination, and sub-second query responses.

High-level data flow:
Kafka (Flow Logs) -> Pekko Streams (Distributed Processing) -> 3-Stage Aggregation (Intermediary Resolution) -> Graph Database -> gRPC API

Engineer Capabilities Enabled by Service Topology

The Service Topology map empowers engineers with several capabilities:

• Visualize Dependencies: Explore upstream/downstream connections, filterable by tier and domain.
• Jump to Detailed Signals: Navigate directly from the topology to logs, traces, and metrics.
• Understand Blast Radius: Assess impact before maintenance or changes.
• Overlay Health Status: See real-time service health within the topology.
• Programmatic Access: Integrate topology data into automated systems via API.
• Faster Investigation: Quickly identify failure origins in incidents.
• Confident Planning: Understand the impact of architectural changes.
• Time Travel: Query historical topology states to understand changes over time.

The Living Map: Trustworthy and Dynamic

The Service Topology is a ‘living map’ that continuously updates based on actual traffic and system events. New calls appear, old ones fade, service behavior changes are reflected, and incident impacts are shown in real-time. This ensures engineers can trust the map as an accurate representation of reality, not an outdated architectural diagram.

Future Directions and Impact

Future enhancements include surfacing change events (deployments, config changes) alongside the topology and enriching it with more context. The ultimate goal is to use Service Topology as a knowledge graph foundation for automated root cause analysis, where intelligent agents can correlate failures and identify likely causes automatically. This focus on observability and rapid troubleshooting directly translates to a more seamless and reliable experience for Netflix members.

Connected Notes

• Distributed Tracing - Service Topology heavily relies on distributed tracing as one of its three core data sources.
• Unified Observability - This article discusses the need for combining different observability signals, which Service Topology exemplifies by integrating network, application, and trace data.
• Async Architecture - The underlying microservice architecture at Netflix relies heavily on asynchronous communication, which necessitates robust dependency mapping.
• Failure Modes - Understanding service dependencies is critical for analyzing and mitigating failure modes in distributed systems.
• AI Agents - The article mentions future potential for AI agents to perform automated root cause analysis, leveraging the service topology as a knowledge graph.

Questions

• How can smaller organizations implement a similar service topology system with fewer resources?
• What are the security implications of collecting and aggregating such detailed network and application communication data?
• How does the Service Topology system handle dynamic IP addresses and ephemeral services in cloud-native environments?
• What are the trade-offs between using a dedicated graph database versus other storage solutions for topology data?
• How can this service topology data be used to proactively identify potential performance bottlenecks or scalability issues?

Sources

• https://medium.com/netflix-techblog/from-silos-to-service-topology-why-netflix-built-a-real-time-service-map-0165ba13a7bc - blog, ingested 2026-06-09T03:59:57.855Z