Army of Proxies! How Netflix scales identity based zero trust...

Army of Proxies! How Netflix scales identity based zero trust...

Speakers: Grant Callaghan

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=FakCMVU3xgE

Overview

This talk, presented by Grant Callaghan, a Staff Security Software Engineer at Netflix, delves into the intricate architecture and operational strategies Netflix employs to scale identity-based zero trust across its vast and evolving ecosystem. Callaghan, a member of Netflix's Access Control and Engineering team since 2018, highlights that scaling in this context is less about requests per second (RPS) and more about scaling human operations to minimize friction on business efforts while effectively balancing risk and speed. The presentation outlines a collaborative approach involving various platform teams to adopt, deploy, and operate zero trust architectural components.

The core of the discussion revolves around a "crew" of interconnected systems, personified as characters in a high-stakes "Caper," that collectively enforce zero trust principles. These include API gateways as "Gatekeepers," identity providers as "Bouncers," service mesh components as "Bodyguards," and a sophisticated control plane as the "Mastermind." The talk emphasizes the challenges of transitioning from a legacy network model to a modern zero trust framework, particularly within an organization as diverse and dynamic as Netflix, which now encompasses streaming, studio production, gaming, animation, and live events. It provides a detailed look at how Netflix leverages both custom-built and open-source-inspired solutions to achieve granular authorization and secure inter-service communication.

Background

▶ Watch: Redefining 'scaling' for Zero Trust: human operations, not RPS. (01:40)

Netflix's journey towards its current security posture began long before Grant Callaghan's tenure, with a program called Lisa (Location Independent Security Approach). This early initiative, which predates Callaghan's time at Netflix, established identity as the perimeter, heavily relying on VPN and SSO to create a strong network boundary. While effective for a smaller, highly technical workforce and a singular streaming product, this model had a significant limitation: once inside the private corporate network, permissions were largely permissive, creating a "wide open" environment. This approach made sense initially, reducing risk to the core product while allowing rapid technical iteration. However, it rested on two critical assumptions: that a flat network would remain desirable, and that services would naturally harden over time without external catalysts.

These assumptions proved unsustainable as Netflix's business and workforce evolved dramatically. The company is no longer solely a streaming-focused engineering organization; it now includes Netflix Studios (one of the largest content producers), game and animation studios, and live production events. Each of these audiences has distinct job functions and durations, yet all require access to subsets of protected corporate resources. The traditional flat network, lacking granular segmentation and authorization, became inadequate for these accelerating business needs and the changing risk landscape. Callaghan notes that changing authorization systems is inherently difficult, citing the struggles observed at Google with BeyondCorp and BeyondProd, and Meta.

The current zero trust architecture at Netflix is heavily inspired by Google's BeyondCorp and BeyondProd white papers, focusing on concepts like service identities and identity-aware proxies. The challenge, as Callaghan explains, is not just implementing these technical components but scaling the human operations involved to avoid imposing excessive friction on business efforts. This involves leveraging various platform teams and embedding security interests within their business processes, effectively "proxying" security concerns through other teams.

Key Findings

▶ Watch: API Gateways (Kong, Zool) as central choke points for external traffic. (09:15)

The talk reveals several critical findings and contributions regarding the implementation and scaling of identity-based zero trust at Netflix:

• Human-Centric Scaling: The primary challenge in scaling zero trust is not technical throughput (RPS) but rather scaling human operations to minimize friction on business efforts. This requires a collaborative approach, embedding security into existing workflows and leveraging other teams.
• Distributed Enforcement Architecture: A comprehensive "army of proxies" architecture is essential, comprising various specialized components that work in concert. These include API gateways (Gatekeepers), identity providers (Bouncers), principal population management systems (The Hideout), service mesh components (Bodyguards), a control plane (The Mastermind), continuous deployment pipelines (The Getaway Driver), and scaffolding/component libraries (The Talent Scout).
• Identity as the New Perimeter: Building on the foundational "Lisa" program, the current architecture reinforces identity as the core perimeter, extending it from initial user authentication (VPN/SSO) to granular service-to-service communication protected by service identities and client certificate authentication.
• Automation as a Force Multiplier: Deep integration with continuous deployment pipelines (e.g., Spinnaker, Manage Delivery) is crucial. This automation allows for mining application manifests to automatically configure API gateways, OAuth clients, and even provision full end-to-end protected zero trust networks for UI developers on every pull request review, significantly reducing misconfigurations and accelerating secure deployments.
• Embedding Security into Developer Workflows: To "win the hearts and minds" of the organization, security is embedded directly into developer tools and processes. This includes providing boilerplate configurations through scaffolding tools (like Netflix's Workflow Toolkit, n) and integrating authentication logic into UI component libraries (like Netflix's Hawkins).
• Proactive Permission Management: Systems like Repo Kid and Squash SSH are used to automatically repossess unused permissions and suggest policy modifications based on observed access history. This "right-sizes" permissions over time, reducing the attack surface and adhering to the principle of least privilege.
• Centralized Observability for Incident Response: The centralized nature of API gateways and service mesh provides a singular choke point for collecting logs, data, and metrics. This observability proved invaluable during incidents like Log4j, enabling rapid detection, blocking of attack classes, and session revocation.
• Addressing Diverse Audiences: The architecture is designed to accommodate Netflix's varied audiences—streaming, studios, games, animation, live production—each with unique access needs and job durations, moving beyond a one-size-fits-all security model.

Technical Deep Dive

▶ Watch: Service Mesh with custom Envoy and OPA sidecar ('Gandalf') for BeyondProd ser... (12:10)

Netflix's zero trust architecture is a sophisticated orchestration of various components, each playing a critical role in enforcing security policies. Grant Callaghan personifies these components as a "crew" in a high-security "Caper":

1. The Gatekeepers (API Gateways and Identity-Aware Proxies): These systems form the front door of the Netflix ecosystem. Positioned at the edge, they provide a singular choke point for external traffic. Examples include Kong, native cloud services like Kubernetes Ingress controllers, or customized open-source solutions like Netflix's Zuul. Their primary functions are to inspect incoming requests for valid credentials, often exchanging external credential representations for internal ones, and then routing these requests to the appropriate internal services. Beyond application traffic, direct instance access for debugging is protected by bastions and jump hosts, leveraging an SSH authority system called Bless to centralize authorization logic and minimize key material sprawl.

1. The Bouncers (Identity Providers): If a request arrives at the Gatekeepers without a valid "ticket," it's redirected to the Bouncers. These identity providers meticulously inspect credentials, verifying their validity (e.g., username/password, passkey). They can also perform basic validation, such as checking if the requester is an employee or a contractor. Identity providers can either proxy credential validations to upstream identity providers like G Suite or perform direct API validation, allowing for the federation and aggregation of diverse user audiences. Keycloak is cited as an example of such a system.

1. The Hideout (Principal Population Management): This refers to the various processes that manage the identities of users and services. At Netflix, this involves leveraging HR business processes and application registries. For other organizations, this might include systems like Workday or ADP for organizational hierarchy, combined with SPIFFE or Spinnaker for service identities. These sources of principles are then aggregated into a shared grouping service, such as Active Directory, LDAP, or a custom-built directory (as Netflix uses internally), which authorization systems reference to check user characteristics and affiliations.

1. The Bodyguards (Service Mesh and Sidecars): Operating within intimate trust boundaries, these components provide close-at-hand security for applications. Netflix utilizes a customized version of Envoy and an authorization sidecar similar to Open Policy Agent (OPA), which they call Gandalf. These processes can be deployed as sidecars or daemons in a Kubernetes environment, or as traditional Linux processes like systemd on instances. They assist with automatic network operation metadata (e.g., request tracing, metrics), and crucially, handle client certificate authentication, OAuth, and path-based authorization for interprocess communication. This setup is instrumental in implementing a system akin to Google's BeyondProd, securing service-to-service communication with service identities and client certificates. For data use cases like ML models running in microservices, a combination of service identity and user identity is employed, ensuring that a spark job or python notebook cannot run with more permissions than the user executing it, and that the service itself also has explicit permission to access underlying data tables.

1. The Mastermind (Control Plane): To coordinate all these moving pieces—onboarding/offboarding users and permissions, managing integrations—a central control plane is indispensable. This system orchestrates the enforcement points (API Gateways, service mesh) to prevent potential security gaps. It manages workflows for user lifecycle, integrates with various platform components to reduce manual configuration errors, and crucially, observes the difference between planned and actual access. Systems like Repo Kid and Squash SSH at Netflix automatically repossess unused permissions and suggest policy modifications based on observed access history, allowing developers to initially specify a slightly wider net for permissions that can be "right-sized" over time.

1. The Getaway Driver (Continuous Deployment): Even with robust control points and a control plane, manual "click-ops" cannot scale. The "Getaway Driver" represents the tight integration with continuous deployment processes, meeting developers where they are. Netflix uses Spinnaker and a custom declarative plugin called Manage Delivery, though alternatives like Argo CD or Jenkins pipelines exist. This integration mines data from application manifests (e.g., the Open App Model by Goa) to automatically configure DNS domains for API gateways and set up OAuth clients with proper redirect URLs. This level of automation allows UI developers to get a full end-to-end protected zero trust network on every single change, tying into pull request review and enabling them to demonstrate changes in production-like systems, which is far more effective than static code review alone.

1. The Talent Scout (Scaffolding and Component Libraries): To win the hearts and minds of the organization and drive adoption, security must be embedded into developer tools. The "Talent Scout" represents efforts to find new teams and initiate them into the zero trust system. This is achieved by embedding code and tools into scaffolding tools (similar to Yeoman generator; Netflix uses its Netflix Workflow Toolkit, or n). These tools set up boilerplate configurations for API gateways and basic authentication. Additionally, security teams partner with UI teams to embed authentication information directly into UI component libraries (Netflix's internal library is called Hawkins, akin to Bootstrap or Material UI), tightly coupling deployment and identity-aware proxies with the UI and various applications.

Demo / Proof of Concept

▶ Watch: Integrating Zero Trust into Continuous Deployment (Spinnaker) for per-PR ZT n... (16:00)

While Grant Callaghan's presentation provides a comprehensive architectural overview and details the functionality of each component within Netflix's zero trust framework, it does not include a live demonstration or a specific proof-of-concept execution. The talk focuses on explaining the integrated system and its operational benefits rather than showcasing a particular technical demo. The emphasis is on the system that enables secure, automated deployments and access management.

Defensive Implications

▶ Watch: Practical impact: Log4J defense via API Gateway, 30% reduction in access-rela... (19:00)

The Netflix zero trust model offers several critical implications for defenders seeking to enhance their security posture:

• Centralized Observability and Incident Response: By routing all external and internal traffic through API gateways and service mesh components, defenders gain a centralized point for collecting logs, metrics, and tracing data. This unified observability is invaluable for incident response, as demonstrated by Netflix's ability to quickly block attack classes during the Log4j vulnerability and perform session revocation when necessary. Defenders should prioritize implementing such centralized logging and monitoring capabilities.
• Automated Access Management and Least Privilege: The use of a control plane to manage user and service lifecycles, coupled with tools like Repo Kid and Squash SSH for automatically repossessing unused permissions, is a powerful defensive strategy. This ensures that permissions are continuously "right-sized" based on observed access history, significantly reducing the attack surface and enforcing the principle of least privilege. Defenders should invest in automation for access provisioning, de-provisioning, and auditing.
• Shift to Identity-Centric Security: Moving beyond traditional network perimeters, the Netflix model underscores the importance of identity as the perimeter. Defenders should focus on strong authentication for both users and services, implementing client certificate authentication and service identities for secure service-to-service communication, especially in distributed microservice architectures.
• Embed Security into Developer Workflows: To achieve widespread adoption and reduce security misconfigurations, defenders must integrate security directly into the developer experience. This includes providing secure boilerplate configurations through scaffolding tools, embedding authentication into UI component libraries, and automating security checks and deployments within CI/CD pipelines. Making the secure path the easiest path is crucial.
• Federated Identity Management: For organizations with diverse user populations (employees, contractors, partners), leveraging identity providers that can federate and aggregate various credential types simplifies access management while maintaining strong authentication. This allows for consistent policy enforcement across different user groups.
• Device Health Integration: While further down Netflix's maturity model, the mention of device identity and tools like stethoscope and Chrome extensions to ensure device health highlights an important future defensive layer. Defenders should consider integrating device posture checks into their zero trust model, even in bring your own device (BYOD) scenarios.
• Reduce Manual Configuration Errors: The emphasis on automating configuration through continuous deployment pipelines and mining application manifests directly addresses a common source of vulnerabilities: human error in manual configuration. Defenders should push for declarative, automated configuration management for all security-critical components.

Key Takeaways

• Scaling Zero Trust is Human-Centric: Effective zero trust implementation at scale prioritizes minimizing friction for business operations and developers, rather than solely focusing on technical throughput.
• Distributed "Army of Proxies" Architecture: A comprehensive, interconnected system of API gateways, identity providers, service mesh components, and a central control plane is essential for granular, distributed security enforcement.
• Automation is Paramount for Secure Deployment: Deep integration with continuous deployment pipelines and leveraging application manifests for automated configuration significantly reduces misconfigurations and accelerates the secure rollout of zero trust components.
• Embed Security into Developer Tools: Winning organizational adoption requires embedding security directly into developer workflows through scaffolding tools and UI component libraries, making the secure path the default and easiest option.
• Centralized Observability is Critical for Response: Identity-aware proxies and service mesh provide centralized logs, metrics, and tracing, offering invaluable data for rapid incident response and proactive security operations.
• Continuous Permission Right-Sizing: Tools that automatically repossess unused permissions based on observed access history are vital for maintaining the principle of least privilege and reducing the attack surface over time.

About the Speaker(s)

Grant Callaghan is a Staff Security Software Engineer at Netflix, where he has been contributing since 2018. As a member of the Access Control and Engineering team within Netflix's information security organization, his work focuses on the development of tools and services that defend the Netflix platform. This includes the design and implementation of authentication platforms and control planes. Callaghan has also consulted on open-source and community projects such as SPIFFE and Envoy proxy. Prior to joining Netflix, his career spanned various roles at companies including PDI, Cisco, VMware, and Google, as well as several R&D startups in Silicon Valley.