Journey at the New York Times: Is Sidecar-Less Service Mesh Disappearing I... Lin Sun & Ahmed Bebars

Journey at the New York Times: Is Sidecar-Less Service Mesh Disappearing Into the Infrastructure?

Speakers: Lin Sun, Principal Engineer, Solo.io; Ahmed Bebars, Principal Engineer, The New York Times

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=9U3WMez9q74

Overview

This talk delves into the New York Times' extensive journey with service mesh technologies, culminating in their exploration and adoption of Istio Ambient Mesh. Presented by Lin Sun from Solo.io and Ahmed Bebars from The New York Times, the session provides a candid look at the challenges and benefits of operating a service mesh at scale within a large enterprise, specifically focusing on the transition from traditional sidecar architectures to the more efficient, sidecar-less Ambient Mesh. The speakers highlight how this new architectural paradigm addresses long-standing issues of operational overhead, resource consumption, and application transparency that have historically plagued service mesh deployments.

The core problem addressed is the inherent complexity and resource intensiveness of the sidecar model, where each application pod requires its own dedicated proxy. This design often leads to increased CPU and memory footprint, operational challenges with application restarts for proxy updates (e.g., security CVEs), and a lack of true transparency for developers. Istio Ambient Mesh, with its innovative two-layer approach, promises to alleviate these burdens by decoupling the service mesh data plane from individual application pods, making the mesh truly "disappear" into the underlying infrastructure.

For organizations running hundreds of nodes and thousands of pods, like The New York Times, the efficiency gains and simplified operations offered by a sidecar-less architecture are not merely incremental improvements but represent a significant shift in how critical infrastructure services are managed. The talk serves as a valuable case study for platform teams contemplating or currently operating a service mesh, offering insights into real-world adoption, performance benchmarks, and the practical challenges encountered during early implementation of cutting-edge service mesh technologies.

Background

▶ Watch: Introduction and New York Times' diverse offerings (0:00)

The New York Times, renowned for its journalism, also operates a diverse digital portfolio including games (like Wordle), cooking apps, product recommendations (Wirecutter), sports content (Athletic), and audio stories. To support this expansive ecosystem, their platform engineering team is tasked with providing robust, resilient, and highly available infrastructure for product engineering teams. This includes defining workflows for service creation, offering out-of-the-box CI/CD pipelines, and running applications on a shared Kubernetes cluster with integrated observability.

A critical challenge for The New York Times, typical of any large-scale microservices environment, revolved around traffic routing and inter-service communication. In a multi-region, highly available setup, services often need to communicate across clusters or within the same cluster. Without a dedicated service mesh, developers are burdened with implementing crucial reliability patterns such as retries, timeouts, and circuit breakers directly within their application code. This "application-level routing logic" adds significant overhead to development teams, diverting focus from core business logic and increasing the complexity of their services. Furthermore, advanced traffic management capabilities like canary deployments and traffic shifting in response to failures become difficult and inconsistent to implement across a heterogeneous application landscape.

To address these challenges, The New York Times identified a clear need for a service mesh. Their requirements were multifaceted:

1. Fine-grained Traffic Routing: The ability to precisely control traffic flow between services, enabling advanced deployment strategies like canary releases and intelligent traffic shifting during cluster or service failures.
2. Decoupling: Abstracting routing logic from application code, allowing developers to focus solely on business functionality.
3. Security by Default: Providing mutual TLS (mTLS) out of the box, managing certificates, and integrating with identity systems like Spiffe without requiring application teams to handle these complexities.
4. Comprehensive Observability: Offering rich, Layer 7 (L7) metrics across the entire service mesh, not just application-specific metrics, to provide deep insights into service behavior and performance.

Their journey began with Cilium for network policy, efficient networking, and isolation, primarily due to its strong capabilities at Layer 3 and Layer 4. For multi-cluster architecture across multiple VPCs and tenants, they leveraged Cilium Cluster Mesh to flatten the network. However, while Cilium provided excellent CNI capabilities, its service mesh features, particularly its reliance on Kubernetes service objects and annotations for traffic management, were found to lack the flexibility and advanced control (e.g., sophisticated retries, canary deployments, deep Kubernetes Gateway API integration) that The New York Times required. This led them to adopt Istio, which offered a feature-rich service mesh solution, but also introduced the overheads associated with its traditional sidecar architecture.

Key Findings

▶ Watch: Why NYT adopted a service mesh: routing, security, observability (3:40)

The New York Times' extensive experience with traditional sidecar-based service meshes, particularly Istio, highlighted several critical limitations that spurred their interest in Istio Ambient Mesh:

1. Sidecar Overhead: The model of injecting an Envoy proxy as a sidecar into every application pod, while powerful, comes with significant operational and resource costs. Each sidecar consumes its own CPU and memory, leading to a substantial increase in the overall cluster footprint, especially in environments with thousands of pods. This overhead directly impacts infrastructure costs and the efficiency of resource utilization.

1. Operational Complexity and Transparency Issues: Sidecars are not truly transparent to applications. Whenever there's an Envoy CVE or a configuration update to the proxy, application pods often need to be restarted. This disrupts application availability and places an operational burden on platform teams, requiring careful coordination and maintenance windows. Furthermore, webhook injection for sidecars can sometimes introduce startup delays or issues, complicating application onboarding.

1. Istio Ambient Mesh as a Game-Changer: The introduction of Istio Ambient Mesh presented a paradigm shift. By moving the service mesh data plane out of individual application pods, Ambient Mesh promised to reduce the number of proxies significantly. This architecture enables a more efficient use of resources, as a single, shared Layer 4 proxy (the Z-tunnel) can serve multiple applications on a node, and Layer 7 capabilities are provided by optional Waypoint proxies deployed per namespace or service.

1. Significant Performance Improvements: The Z-tunnel component of Ambient Mesh, written in Rust, demonstrates superior performance compared to traditional Envoy sidecars. The New York Times observed a 25% reduction in latency, with specific measurements showing a drop from 4.98 milliseconds to 3.7 milliseconds in test environments. While these numbers might seem small for a single request, they accumulate significantly across a microservices architecture where requests often traverse multiple services, making every millisecond count. Additionally, the Z-tunnel exhibits a 50% lower CPU usage compared to the sidecar proxy, further contributing to resource efficiency.

1. Simplified Operations and Faster Onboarding: With no sidecars, the complexities associated with proxy injection webhooks and proxy lifecycle management are eliminated. Applications can onboard faster into the mesh, as they no longer need to wait for a sidecar to start or deal with sidecar-related issues like 503 errors during startup. The Ambient architecture simplifies debugging, offering dedicated tooling like istioctl waypoint and istioctl ztunnel config for easier troubleshooting.

1. Decoupled Scaling: Ambient Mesh allows applications to scale independently from the routing traffic layer. This means that the scaling of the mesh components (Z-tunnels and Waypoints) can be managed separately from the scaling of application pods, providing greater flexibility and resource optimization.

1. Seamless Migration Strategy: The New York Times found that migrating from a sidecar-based Istio deployment to Ambient Mesh could be done incrementally, namespace by namespace or even service by service. The Ambient Z-tunnels were capable of seamlessly picking up traffic as needed, ensuring no interruptions during the transition. Existing application policies and configurations generally continued to work as expected, minimizing changes required from application teams.

Technical Deep Dive

▶ Watch: NYT's journey: From Cilium to Istio for advanced mesh (5:55)

The journey from a traditional sidecar-based service mesh to Istio Ambient Mesh is driven by a fundamental re-evaluation of how the data plane interacts with application workloads. The core limitations of the sidecar model, as highlighted by The New York Times, stem from its design where an Envoy proxy is co-located with every application pod. While this provides granular control and isolation, it leads to:

• Lack of True Transparency: Applications are aware of the sidecar. Updates to the sidecar (e.g., for security patches like an Envoy CVE) often necessitate restarting the application pod, violating the principle of transparency.
• Resource Inefficiency: Each sidecar is an independent process consuming CPU and memory. At scale, this leads to significant resource overhead, impacting density and cost.
• Operational Complexity: Managing the lifecycle, configuration, and debugging of potentially thousands of sidecar proxies adds considerable operational burden.

Istio Ambient Mesh addresses these challenges through an innovative multi-layer architecture that decouples the service mesh data plane from the application pod. This architecture comprises two main components:

1. Zero Trust Tunnel (Z-tunnel):

• Function: The Z-tunnel acts as a node agent, running once per Kubernetes node, similar to how kubelet manages pods. It serves all Ambient-enabled pods on that node.
• Core Capabilities:
• Identity Provisioning: Provides cryptographic identity for pods through Spiffe, enabling secure communication.
• Mutual TLS (mTLS): Upgrades all connections to mTLS, ensuring encryption in transit for Layer 4 traffic. This includes support for FIPS 140-3 compliance.
• Layer 4 Authorization: Enforces basic authorization policies at Layer 4 (TCP), controlling which services can communicate at a network level.
• Performance: The Z-tunnel is written in Rust, a language known for its performance and memory safety. This contributes to its efficiency, achieving a 25% reduction in latency and 50% lower CPU usage compared to sidecar proxies.
• Transparency: By running as a node agent, the Z-tunnel is completely transparent to application pods, eliminating the need for application restarts for Layer 4 policy or mTLS updates.

1. Waypoint Proxy:

• Function: The Waypoint proxy is an optional, Layer 7 (L7) proxy that provides advanced traffic management and policy enforcement. It is deployed separately from application pods and can be scoped per namespace, per service, or across multiple namespaces, depending on tenancy requirements.
• Core Capabilities:
• Kubernetes Gateway API Integration: A key innovation is bringing the Kubernetes Gateway API concept into the mesh. This provides a consistent experience for controlling traffic, whether it's ingress (traffic into the cluster), egress (traffic from the cluster to external services), or east-west (traffic between services within the cluster).
• Advanced Layer 7 Features: Handles sophisticated L7 capabilities such as:
• Traffic Management: Canary deployments, traffic shifting, A/B testing.
• Resilience: Retries, timeouts, circuit breakers.
• Layer 7 Observability: Generates fine-grained HTTP request metrics and traces.
• Layer 7 Security Policies: Enforces granular authorization policies based on HTTP attributes, headers, and more.
• Implementation: The Waypoint proxy is implemented using Envoy proxy, leveraging its rich feature set for L7 processing.
• Decoupling: Like the Z-tunnel, the Waypoint proxy runs outside application pods, making L7 policy enforcement transparent to applications.

Why the Two-Layer Architecture and Z-tunnel in Rust?

The decision for a two-layer architecture, particularly using a dedicated Z-tunnel for Layer 4 instead of a multi-tenant Envoy, is crucial. Envoy, while powerful, was not inherently designed for multi-tenancy at Layer 4. Using a single Envoy for all Layer 4 traffic on a node could introduce a "noisy neighbor" problem, where one busy application could impact the performance or stability of the shared proxy, potentially affecting other applications or tenants. The Z-tunnel, written in Rust, is specifically optimized for this multi-tenant, Layer 4 role, providing robust isolation and predictable performance.

Migration and Challenges:

The New York Times successfully tested a migration strategy where they could incrementally enable Ambient Mesh on a namespace-by-namespace or service-by-service basis. This allowed for a controlled rollout without disrupting existing sidecar-enabled services. However, as early adopters, they encountered some initial challenges:

• Policy Compatibility: Some existing Open Policy Agent (OPA) or authorization policies behaved differently in the Ambient layer, requiring adjustments. This was actively addressed by the Istio community.
• Observability Tweaks: The shift from per-pod sidecar proxies to shared Z-tunnels and Waypoints necessitated changes in how observability data was collected and filtered.
• Z-tunnel CPU Spikes: Early load testing revealed CPU spikes in Z-tunnels when many services were onboarded simultaneously. This performance anomaly was quickly identified and resolved in subsequent Istio releases with community support.
• Multi-cluster Support: While Ambient Mesh worked well for single clusters, robust multi-cluster support was an ongoing effort in the community during their initial testing. The community is actively working to bring similar multi-cluster capabilities to Ambient as exist for sidecar deployments, with traffic expected to flow securely through Z-tunnels, potentially via east-west gateways to Waypoints for L7 policy enforcement.

Demo / Proof of Concept

▶ Watch: Istio Ambient Mesh introduced as a game-changer (6:40)

Lin Sun attempted a live demonstration of Istio Ambient Mesh in action, showcasing its capabilities in a generative AI application context. The setup involved a Kubernetes cluster running a RAG (Retrieval Augmented Generation) application with two versions (v1 and v2) in an Ambient-enabled environment. The demonstration included:

• Ambient Mesh Components: Istio Ambient Mesh was installed, along with an Istio Ingress Gateway, Kiali (for mesh visualization), Prometheus (for metrics), and the essential Z-tunnel components running on the nodes.
• Waypoint Proxies: A Waypoint proxy was deployed in the default namespace to control internal service traffic, and an egress Waypoint proxy was set up in a dedicated egress namespace. This egress Waypoint was specifically configured to manage traffic flowing from within the cluster to an external Large Language Model (LLM) running outside the Kubernetes cluster on Lin's machine (a Llama model).
• Application Functionality: The demo aimed to illustrate two main features: mood analysis (using a phone camera connected to the application) and a chat interface interacting with the external LLM. The core idea was to show how the application, despite having no sidecars, would benefit from mTLS, L7 traffic management, and observability provided by Z-tunnels and Waypoints.

Unfortunately, the live demo encountered a common pitfall of conference presentations: technical difficulties. The primary issue stemmed from certificate rotation within Istio, which occurs every 24 hours. As the demo environment was running on a laptop that was not consistently connected, the certificates for various pods (application, Waypoints) had expired or become mismatched. This led to 503 errors and connection resets when the application attempted to communicate through the Waypoint proxies.

Despite the initial failure, Lin skillfully debugged the environment on stage by deleting and recreating the affected pods. Once the environment was stable with correctly rotated certificates, she was able to show the Kiali dashboard, illustrating the traffic flow. The dashboard successfully displayed:

• Traffic entering the cluster through the Ingress Gateway to the demo application.
• Secure communication through mutual TLS (mTLS), powered by the Z-tunnels.
• The application's interaction with the external LLM, routed and secured by the egress Waypoint proxy. This specifically highlighted how L7 policies and observability, including HTTP request metrics, were applied to external traffic without any sidecars.

The demo, though initially problematic, ultimately served as a powerful illustration of Ambient Mesh's capabilities and the robust observability it provides, even in a challenging live environment. Ahmed Bebars further promised a future demo showcasing The New York Times' production setup with actual read traffic, underscoring their commitment to Ambient Mesh.

Defensive Implications

▶ Watch: Explaining Istio Ambient Mesh: The zero trust tunnel node agent (7:40)

The adoption of Istio Ambient Mesh by The New York Times carries significant defensive implications, enhancing the security posture of their microservices architecture in several key areas:

1. Transparent Zero Trust at Layer 4: The Z-tunnel fundamentally shifts the security baseline. By providing mutual TLS (mTLS) for all Layer 4 connections between Ambient-enabled services by default, it enforces encryption in transit without any application-level changes. This eliminates the need for developers to manage certificates or implement TLS, ensuring that all internal service-to-service communication is secured. Furthermore, the Z-tunnel's capability to enforce simple Layer 4 authorization policies acts as a robust first line of defense, restricting network-level access based on identity (Spiffe) rather than just IP addresses. This is a crucial step towards a Zero Trust network model.

1. Consistent Layer 7 Policy Enforcement: The Waypoint proxy, leveraging the Kubernetes Gateway API, centralizes and standardizes Layer 7 security policy enforcement. This means that granular authorization policies, rate limiting, and other security controls can be applied consistently across ingress, egress, and east-west traffic. For instance, an egress Waypoint can strictly control which external services (like OpenAI or other third-party APIs) an application can communicate with, preventing unauthorized data exfiltration or access to malicious external endpoints. This consistent policy layer simplifies auditing and reduces the risk of misconfigurations across disparate application teams.

1. Reduced Attack Surface and Faster Patching: Moving away from per-pod sidecars significantly reduces the overall attack surface. Instead of hundreds or thousands of individual Envoy proxies that need to be secured and updated, defenders now manage fewer, shared components: Z-tunnels per node and Waypoints per scope (e.g., namespace). When an Envoy CVE is discovered, only the Waypoint proxies (which use Envoy) need to be updated, not every application pod. The Z-tunnel, being written in Rust and specifically designed for Layer 4, offers a more stable and potentially less vulnerable attack vector for its scope. This decoupled architecture allows for much faster and less disruptive security patching cycles, improving the overall agility of the security team.

1. Enhanced Observability for Threat Detection: The out-of-the-box Layer 7 observability provided by Ambient Mesh, including detailed HTTP request metrics and traces, is invaluable for security monitoring. This rich telemetry allows security teams to detect anomalous behavior, unauthorized access attempts, or policy violations more effectively. For example, sudden spikes in error rates for specific services, unusual traffic patterns to external destinations, or attempts to access unauthorized resources can be quickly identified and investigated using tools like Kiali and Prometheus. This deep visibility into service interactions is critical for proactive threat hunting and incident response.

1. Simplified Compliance and Auditing: By centralizing security policy enforcement and providing comprehensive audit trails through observability, Ambient Mesh streamlines compliance efforts. Organizations can demonstrate that security controls (like mTLS and authorization policies) are consistently applied across their microservices, meeting regulatory requirements. The ability to define and enforce policies using standard Kubernetes Gateway API resources also simplifies the auditing process, as policies are declared in a declarative manner.

In essence, Istio Ambient Mesh offers a more robust, efficient, and manageable security framework compared to traditional sidecar models, enabling defenders to implement a stronger Zero Trust posture with reduced operational overhead and improved visibility.

Key Takeaways

• Addressing Sidecar Limitations: Istio Ambient Mesh directly tackles the operational overhead, resource consumption, and application transparency issues inherent in traditional sidecar-based service mesh architectures.
• Two-Layer Architecture for Efficiency: The core innovation is a multi-layer design featuring a node-local Z-tunnel (for Layer 4 mTLS, identity, and authorization) and optional, scoped Waypoint proxies (for Layer 7 traffic management, policies, and observability), both running outside application pods.
• Significant Performance Gains: The Z-tunnel, written in Rust, provides substantial performance improvements, including a 25% reduction in latency (e.g., from 4.98ms to 3.7ms) and 50% lower CPU usage compared to sidecar proxies, leading to better resource utilization.
• Simplified Operations and Migration: Ambient Mesh eliminates sidecar injection webhooks and allows for incremental, non-disruptive migration from existing Istio deployments, making adoption easier for platform teams.
• Enhanced Security by Default: It enforces transparent mTLS at Layer 4 and provides robust Layer 7 policy enforcement via Waypoint proxies for ingress, egress, and east-west traffic, bolstering the organization's Zero Trust posture.
• Ongoing Evolution: While offering significant advantages, the technology is still evolving, particularly regarding robust multi-cluster support, which is actively being developed by the Istio community.

About the Speaker(s)

Lin Sun is a Principal Engineer at Solo.io. Based in Cary, a suburb of Raleigh, North Carolina, Lin is a key contributor to the service mesh ecosystem. Her employer, Solo.io, is recognized as the 10th largest contributor to all combined CNCF (Cloud Native Computing Foundation) projects, highlighting their significant impact on cloud-native technologies. Lin is deeply involved in the development and innovation behind Istio Ambient Mesh.

Ahmed Bebars is a Principal Engineer at The New York Times, where he plays a crucial role in infrastructure development, specializing in Kubernetes and Istio. He is instrumental in driving the adoption of advanced cloud-native solutions within the organization. Ahmed noted that this KubeCon presentation was particularly special as it was his second talk attended by his daughter, underscoring a personal connection to his work and the community.