EVAPorating Kubernetes Security Risk: Adopting Validating Admission P... Kaitlyn Lee & Jordan Conard

EVAPorating Kubernetes Security Risk: Adopting Validating Admission Policy at Scale

Speakers: Kaitlyn Lee, Software Engineer, DataDog; Jordan Conard, Security Engineer, DataDog

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=OJ1WoQjYAJo

Overview

In this insightful KubeCon EU talk, Kaitlyn Lee and Jordan Conard from DataDog shared their extensive experience in migrating Kubernetes admission control from OPA Gatekeeper to the native Validating Admission Policy (VAP). The presentation, titled "EVAPorating Kubernetes Security Risk," delves into DataDog's journey of transforming basic tutorial policies into robust, production-grade VAP implementations capable of operating at their immense scale. With over 100 Kubernetes clusters, 10,000 nodes, and 100,000 pods across a multi-cloud environment, DataDog's approach to securing their entirely Kubernetes-dependent infrastructure offers invaluable lessons for any organization seeking to enhance their cloud-native security posture.

The talk highlights the critical need for granular and configurable security policies to manage heterogeneous workloads effectively. Lee and Conard meticulously detail the operational, cost, and security benefits of VAP, emphasizing its in-process validation within the Kubernetes API server compared to external webhook solutions. They provide a practical, step-by-step guide to evolving a basic capabilities policy, covering advanced features like parameter resources, CEL optional field selection, and message expressions to improve flexibility, user experience, and maintainability.

This discussion is particularly relevant for Kubernetes administrators, security engineers, and platform teams grappling with the complexities of admission control. It offers a clear blueprint for adopting VAP, including robust migration strategies, essential monitoring techniques, and a forward-looking perspective on "Day 2 operations." DataDog's candid sharing of their challenges and solutions provides a compelling case for VAP as a mature and powerful tool for enforcing security best practices across large-scale Kubernetes deployments.

Background

▶ Watch: Introduction and DataDog's massive Kubernetes scale (0:00)

DataDog operates a massive and complex Kubernetes environment, underpinning its entire engineering organization. With over 2,000 employees, 100+ clusters, 10,000+ nodes, and 100,000+ pods, every single workload, from infrastructure to platforms and applications, runs on Kubernetes. This scale necessitates a highly granular and configurable set of security policies to accommodate their diverse and heterogeneous workloads.

DataDog's journey with Kubernetes admission control began around 2020, coinciding with the deprecation of Pod Security Policies (PSP). At that time, Pod Security Admission (PSA) was not deemed a suitable replacement for their needs. Consequently, OPA Gatekeeper emerged as the de facto standard in the ecosystem, and DataDog adopted it as their initial solution. OPA Gatekeeper, built on the Open Policy Agent (OPA) engine, is a CNCF graduated project designed as a general-purpose policy engine. It uses the Rego language for policy definition and wraps the OPA engine into a validating admission webhook, intercepting requests to the Kubernetes API server.

The motivation to migrate to Validating Admission Policy (VAP) stemmed from the introduction of this new feature in Kubernetes. As of Kubernetes version 1.30, VAP has graduated to stable, offering a declarative, in-process alternative to traditional validating admission webhooks. Unlike external webhooks like Gatekeeper, VAP validations are evaluated directly within the Kubernetes API server. This key architectural difference, coupled with VAP's use of the Common Expression Language (CEL) instead of Rego, presented a compelling case for DataDog to transition. A proof-of-concept demonstrated significant benefits, leading to the decision to embark on a full migration.

Key Findings

▶ Watch: Introduction to Kubernetes 1.30 Validating Admission Policy (VAP) (2:40)

DataDog's evaluation of Validating Admission Policy (VAP) identified several key advantages over their existing OPA Gatekeeper setup, which ultimately drove their migration strategy:

1. Reduced Operational Complexity and Cost: The most significant finding was VAP's in-process evaluation. By moving policy validation from an external webhook (like Gatekeeper) directly into the Kubernetes API server, DataDog anticipated and realized a substantial reduction in operational overhead. This eliminates the need to deploy, manage, and scale separate webhook deployments, reducing cloud resource consumption and associated costs. Furthermore, it improves the overall security posture by removing an external, potentially vulnerable, component from the critical admission control path.

1. Leveraging the Common Expression Language (CEL): DataDog found CEL to be a highly advantageous language for policy definition. Its increasing adoption across various Kubernetes components, such as validating fields in Custom Resource Definitions (CRDs), and its internal use in DataDog's developer tooling, made it a strategic choice. While initially requiring a paradigm shift from Rego's procedural style, CEL's declarative nature and growing ecosystem support were seen as long-term benefits.

1. Namespace-Scoped Parameters: VAP's ability to utilize namespace-scoped parameters was a crucial feature for DataDog's security engineers. This capability allows policies to be configured differently for individual namespaces, providing a clear, at-a-glance view of a namespace's security posture. This aligns well with DataDog's existing namespace ownership and Role-Based Access Control (RBAC) boundaries, enabling fine-grained control and reducing the need for broad, monolithic policies or complex exclusions. This flexibility is essential for managing the diverse security requirements of their heterogeneous workloads across a vast multi-tenant environment.

Technical Deep Dive

▶ Watch: DataDog's key motivations for adopting VAP (3:30)

DataDog's migration to Validating Admission Policy (VAP) involved a methodical process of translating existing policies and enhancing them to meet production requirements. The speakers walked through the evolution of a specific "capabilities policy," designed to restrict added capabilities within a container's security context.

Initially, a basic VAP implementation consists of a ValidatingAdmissionPolicy resource and a ValidatingAdmissionPolicyBinding resource. The policy resource specifies the API groups and kinds it validates against (e.g., pods on create and update requests). The core logic resides in the Common Expression Language (CEL) expression. For instance, an initial capabilities policy might simply check that no container's securityContext.capabilities.add field is specified. The binding resource then links the policy to namespaces, often using a namespaceSelector to apply it by default, with specific exclusions via labels. This basic setup, however, proved too rigid for DataDog's diverse needs, offering an "all or nothing" approach.

To introduce flexibility, the first enhancement involved adding a variable to the policy. A globallyAllowedCapabilities variable, defined as a list of capabilities that don't require additional security review, was introduced. This variable could then be referenced within the CEL expression, allowing pods to add specific, pre-approved capabilities without being denied. To make the CEL expression more concise and robust, DataDog adopted CEL optional field selection syntax using the ?.orValue() operator. This allowed them to safely check for the existence of nested fields like securityContext.capabilities.add and substitute an empty list if the field was absent, simplifying the logic and ensuring the validation passed correctly for pods without specified capabilities.

A significant leap in flexibility came with the introduction of Parameter Resources. VAP allows injecting information from any Kubernetes resource into a policy. DataDog opted to create their own Custom Resource Definition (CRD) to encapsulate all necessary policy variables, including allowed capabilities. This CRD-based parameter resource is referenced in the ValidatingAdmissionPolicy resource using the paramKind field (specifying the API version and kind of the CRD) and in the ValidatingAdmissionPolicyBinding resource via the paramRef field (pointing to a specific instance of the CRD). This architecture enables a single VAP policy to be configured differently across various namespaces, each with its own paramRef to a unique parameter resource instance, allowing for granular, namespace-specific security requirements. The talk also noted a crucial configuration in paramKind to define policy behavior if a parameter resource is missing in a namespace.

Recognizing that users often interact with higher-level resources, DataDog expanded the policy's scope beyond just pods to include pod-creating resources like Deployments, StatefulSets, ReplicaSets, and CronJobs. This required a mechanism to extract the podSpec from these different resource types. A podSpec variable was introduced within the CEL expression, utilizing ternary conditionals to dynamically determine the correct path to the podSpec based on the resource's kind and apiVersion. This clever approach allowed a single validation expression to apply consistently across multiple resource types, improving the user experience by providing denials directly on the resources users are more familiar with.

Finally, to enhance usability and aid troubleshooting, DataDog implemented message expressions. These are also CEL expressions, attached to the main validation expression, and are rendered when a request is denied. Message expressions can dynamically generate helpful error messages, including details like the specific container names that violated the policy and a hyperlink to internal documentation. This significantly improves the developer experience by providing actionable feedback directly within the admission denial message.

Through these enhancements—variables, optional field selection, parameter resources, multi-resource validation with podSpec variables, and message expressions—DataDog transformed a basic VAP policy into a production-grade solution, balancing stringent security requirements with necessary operational flexibility and user-friendliness.

Demo / Proof of Concept

▶ Watch: Adding flexibility to VAP with policy variables (7:00)

While the talk did not feature a live coding demonstration or a traditional "demo" of a running system, the speakers effectively walked through a comprehensive policy evolution lifecycle, which served as a detailed proof of concept for VAP's capabilities and DataDog's approach. This "demo" focused on illustrating how a barebones capabilities policy could be incrementally enhanced to meet complex production demands.

The journey began with a basic Validating Admission Policy and Binding that enforced a strict "no added capabilities" rule. This initial state highlighted the inflexibility of a naive implementation, where legitimate use cases for certain capabilities would either be outright denied or require wholesale exclusion of an entire namespace, creating security gaps.

The first step in the evolution was demonstrating the addition of a globallyAllowedCapabilities variable. This showed how to introduce a list of pre-approved capabilities, moving the policy from an "all or nothing" stance to one with controlled exceptions. This was further refined by showcasing CEL optional field selection (?.orValue()), illustrating how to write more concise and robust CEL expressions that gracefully handle missing fields, preventing unnecessary errors and improving readability.

The core of the "proof of concept" for advanced flexibility was the introduction of Parameter Resources. DataDog demonstrated their custom CRD for defining namespace-specific allowed capabilities. This illustrated how a single VAP could be configured differently across multiple namespaces by referencing distinct parameter resource instances, providing granular control without policy duplication. This concept is crucial for large organizations with diverse application needs.

Next, the speakers demonstrated how to expand the policy's scope to higher-level pod-creating resources (like Deployments and CronJobs) using a podSpec variable and ternary conditionals. This showed how VAP could provide a consistent user experience by validating resources that developers directly interact with, rather than just the underlying pods.

Finally, the talk showcased the power of message expressions to provide immediate, actionable feedback to users. By dynamically generating error messages that included violating container names and links to documentation, DataDog demonstrated how VAP could become a helpful guardrail rather than just a blocking mechanism.

In essence, the entire technical deep dive served as a comprehensive "proof of concept" for how VAP can be adopted and scaled, addressing real-world operational and security challenges through iterative policy refinement and leveraging VAP's advanced features. The speakers also mentioned using a CEL playground tool for testing and validating their CEL expressions, including observing the associated validation cost, which is a critical aspect for ensuring API server reliability. This playground allowed them to check the validity and behavior of their policies with test request objects and understand the performance implications before deployment.

Defensive Implications

▶ Watch: Using CEL's optional field selection for concise policies (7:50)

Adopting Validating Admission Policy (VAP) at scale has significant defensive implications, particularly concerning safe migration, continuous monitoring, and future operational strategies. DataDog meticulously planned these aspects to ensure a smooth transition and maintain API server stability.

Safe Migration Strategy:

DataDog employed a two-phase migration strategy to transition from OPA Gatekeeper to VAP:

1. Audit Mode First: Initially, VAP policies were deployed with the audit validation action. In this mode, VAP logs all policy violations to a specified audit log but does not deny requests. This crucial step allowed DataDog to validate the behavior of their new VAP policies against their existing Gatekeeper policies. Since VAP is evaluated before validating admission webhooks (where Gatekeeper runs), they could observe VAP logging violations while Gatekeeper still enforced denials. The goal was to ensure that every pod admission denial from Gatekeeper had a corresponding policy violation in the VAP audit log, confirming behavioral parity.
2. Deny Mode and Decommissioning: Once confidence was established through audit logs and comprehensive unit/end-to-end testing, the deny validation action was added to the VAP binding resources. At this point, VAP began actively denying requests before they reached Gatekeeper. DataDog confirmed the success of this phase by observing a complete absence of admission denials from Gatekeeper. With VAP fully in the critical path and operating correctly, Gatekeeper could then be safely decommissioned.

Robust Testing:

To ensure the correctness and matching behavior of migrated policies, DataDog developed a suite of unit and end-to-end tests. These tests involved defining various pod resources—some expected to pass, others to fail—and applying the VAP policies within a local kind cluster using kubectl dry-run. This allowed them to verify that the VAP policies behaved identically to their corresponding OPA Gatekeeper policies. For Rego, unit testing is more straightforward, but for CEL, the speakers noted that the CEL playground was invaluable for initial development and validation. They also specifically mentioned using the E2E framework for their end-to-end tests, ensuring local development accurately reflects cluster behavior.

Monitoring and API Server Health:

With VAP running directly within the API server, monitoring its impact on API server health is paramount. Kubernetes provides several metrics for this:

• total_policy_checks and policy_check_duration: These metrics track the overall number of policy evaluations and their duration, providing a high-level view of VAP's performance.
• CEL Compilation and Evaluation Duration: These aggregate metrics (available in the API server) offer insights into how CEL expressions, including those in VAP and CRD validations, affect API server performance. While aggregate, they provide a general sense of any potential issues.

Troubleshooting and Performance Guardrails:

To prevent runaway or inefficient CEL expressions from degrading API server performance, Kubernetes implements a validation cost budget. Each CEL expression has an associated validation cost, and the API server protects itself with a static estimated cost limit hardcoded to 10 million. The CEL playground tool, used by DataDog, displays the validation cost for each variable and expression, as well as a total policy cost. This allows engineers to proactively identify and optimize complex or nested CEL expressions that could exponentially increase costs, ensuring the reliability of the API server.

Day 2 Operations and Future Enhancements:

DataDog outlined several "Day 2" operational considerations and next steps to further enhance their VAP implementation:

• Expanded Scope: Extending policies to cover ephemeral and init containers in addition to main containers, as these can also add capabilities to their security contexts.
• Automated Sidecar Exclusions: Implementing a parameter or variable that maps container image names to an allowed list of capabilities. This automatically grants admission to specific sidecars (e.g., DataDog's own injected sidecars) without manual toil for individual exclusions.
• Self-Service Exclusion Platform: Developing an API-driven internal platform to allow end-users to request policy exclusions (modifying parameter resources) with an integrated security review process.
• Pruning Unneeded Exclusions: Creating mechanisms to identify and remove outdated or unnecessary policy exclusions over time. This proactive approach prevents "security debt" from abandoned namespaces or changed workloads, which could otherwise introduce latent security risks.

These defensive strategies collectively ensure that VAP is not only adopted securely but also maintained efficiently and proactively evolved to meet changing operational and security demands.

Key Takeaways

• Validating Admission Policy (VAP) offers significant operational, cost, and security benefits over external admission webhooks like OPA Gatekeeper by integrating policy evaluation directly into the Kubernetes API server.
• Common Expression Language (CEL) is a powerful and increasingly adopted language for defining Kubernetes policies, providing a declarative approach to admission control, though it requires a paradigm shift from procedural languages like Rego.
• Parameter Resources are crucial for creating flexible and scalable VAP implementations, enabling a single policy to be configured differently across namespaces using custom CRDs, aligning with granular security requirements and ownership models.
• Comprehensive migration strategies, starting with audit mode before deny, are essential for safely transitioning to VAP, coupled with robust unit and end-to-end testing to ensure behavioral parity with existing admission controllers.
• Proactive monitoring of Kubernetes API server metrics (policy checks, CEL compilation/evaluation duration) and awareness of the CEL validation cost budget are vital for maintaining API server health and preventing performance degradation from complex policies.
• Advanced VAP features like optional field selection, podSpec variables for multi-resource validation, and message expressions significantly improve policy conciseness, maintainability, and user experience, providing clear error messages and reducing troubleshooting time for developers.

About the Speaker(s)

Jordan Conard is a Security Engineer at DataDog, where he works on the compute team. His primary focus is on securing DataDog's Kubernetes control planes, leveraging his expertise in cloud-native security to manage and implement robust admission policies across their extensive Kubernetes infrastructure.

Kaitlyn Lee is a Software Engineer, also on the compute team at DataDog. She began her tenure on the control plane team, gaining deep insights into Kubernetes internals, and now specializes in workload autoscaling. Her background in control plane engineering provides a strong foundation for understanding the practical implications and engineering challenges of adopting new Kubernetes features like Validating Admission Policy. Together, they bring a blend of security and software engineering perspectives to the challenges of scaling Kubernetes security.