Trust No One: Secure Storage With Confidential Containers - Aurélien Bombo, Microsoft

Trust No One: Secure Storage With Confidential Containers

Speakers: Aurélien Bombo, Microsoft

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=etCmLttqJsQ

Overview

In his KubeCon EU talk, Aurélien Bombo from Microsoft presented a comprehensive overview of securing storage within Confidential Containers (often referred to as CoCo), a critical advancement in the realm of confidential computing. As a core contributor to the Confidential Containers project and a member of the Kata Containers architecture committee, Bombo highlighted the project's belief that confidential computing represents the future, demanding protection for data not only at rest and in transit but crucially, also in use. Traditionally, confidential computing has heavily focused on protecting the compute aspect, often overlooking the equally vital components of networking and storage. This talk directly addresses that gap, detailing the ongoing work within the Confidential Containers community to enable secure storage for containerized workloads.

The presentation delved into both the theoretical underpinnings and practical implementations for securing both ephemeral and persistent storage within a confidential computing environment. Bombo emphasized that while much of this work is still in progress, the presented designs showcase a robust approach to integrating secure storage with Kubernetes at scale. The core challenge lies in maintaining confidentiality and integrity when dealing with untrusted host environments, a problem CoCo tackles by leveraging Trusted Execution Environments (TEEs), remote attestation, and a novel container security policy. This initiative is particularly significant for end-users seeking to protect sensitive data from their cloud providers and for cloud providers aiming to enable secure multi-tenancy and maximize resource utilization.

Background

▶ Watch: Speaker introduction and confidential computing problem (0:00)

The evolution of container isolation has seen several stages, each offering varying degrees of security. Initially, traditional container runtimes like Docker and runc share the host kernel with the containers. While efficient, this model presents a relatively loose isolation, making containers vulnerable to kernel exploits that could lead to a container breakout, allowing an attacker to compromise the host or other containers.

To enhance isolation, projects like Kata Containers introduced virtualization, placing each container within its own lightweight virtual machine (VM). This provides a stronger boundary, but crucially, it does not protect against a potentially malicious host or cloud provider. For users with stringent security requirements—such as those operating in highly regulated industries or handling extremely sensitive data—protection from the underlying infrastructure itself becomes paramount.

This is where Confidential Containers (CoCo) comes into play. Building upon the foundation of Kata Containers, CoCo leverages Trusted Execution Environments (TEEs). TEEs are hardware-backed secure areas within a processor that guarantee the confidentiality and integrity of code and data loaded inside them. For CoCo, this means the entire VM's memory can be encrypted, making its contents inaccessible even to the host operating system or hypervisor. Furthermore, CoCo utilizes remote attestation, a process that allows a remote party (or the workload itself) to cryptographically verify the integrity and configuration of the TEE and the software running within it. This provides a crucial proof of the VM's trustworthiness and the contents of the deployed workload.

A key innovation developed by Microsoft's team within the CoCo project is the container security policy. This policy is designed to secure the interface between the trusted, confidential VM and all external, untrusted components. Given that orchestrators like Kubernetes are essential for deploying containers at scale, seamless integration with Kubernetes is a fundamental requirement for CoCo's adoption. Kubernetes handles critical aspects such as scaling, load balancing, fault tolerance, and service discovery, and notably, storage management. Therefore, extending CoCo's protection to storage within the Kubernetes ecosystem is not just an enhancement but a necessity for comprehensive data protection.

Key Findings

▶ Watch: Explaining Confidential Containers (CoCo) and TEEs (2:58)

Bombo's presentation highlighted several key findings and architectural contributions towards enabling secure storage in Confidential Containers:

1. Comprehensive Data Protection: Confidential Containers are being extended to protect data in use, specifically for both ephemeral and persistent storage, addressing a critical gap in traditional confidential computing which often prioritizes compute.
2. Ephemeral Storage Security: Ephemeral storage within confidential VMs can be secured by generating encryption keys inside the VM and using Linux kernel modules like DM-Crypt and DM-Integrity to encrypt block devices. This ensures that even temporary data is inaccessible to the untrusted host.
3. Kubernetes Integration via CSI: The Container Storage Interface (CSI) driver model in Kubernetes is leveraged to provision and manage confidential storage. This allows CoCo to integrate seamlessly with existing Kubernetes storage paradigms, making the transition smoother for users.
4. Security Policy as a Trust Anchor: A robust security policy, written in the Rego policy language and enforced by a Rust implementation called regor-rust, is crucial. This policy ties the high-level Kubernetes container specification to the low-level JSON specification executed by the kata agent, preventing malicious modification by untrusted host components and guarding against unauthorized device injection.
5. Remote Attestation for Policy and Workload Trust: The trustworthiness of the security policy itself, and by extension the container and its attached devices, is established through a remote attestation process. An external, trusted Remote Attestation Service (RAS) verifies the observed policy against a known, trusted reference policy.
6. Persistent Storage Leveraging Key Broker Service: For persistent storage, the design introduces a Key Broker Service (KBS). This service, in conjunction with the RAS, securely delivers decryption keys to the confidential VM after successful attestation, allowing the VM to decrypt pre-encrypted remote storage.
7. User Transparency: The encryption layer for storage is designed to be transparent to the end-user application running inside the confidential container, simplifying adoption and minimizing application-level changes.

Technical Deep Dive

▶ Watch: Why CoCo needs good Kubernetes integration (5:30)

The technical architecture for securing storage within Confidential Containers is intricate, designed to maintain trust in an inherently untrusted host environment. Bombo detailed the lifecycle of a CoCo container in Kubernetes, emphasizing the security boundaries.

When a user deploys a container specification (a YAML file) to a Kubernetes cluster, it first reaches the kubelet on a host node. The kubelet then forwards this spec to the kata runtime, which, in turn, triggers a Virtual Machine Manager (VMM) to create the confidential VM. Inside this VM, a kata agent (written in Rust) interfaces with the kata runtime to instantiate the container. Critically, all components outside the confidential VM boundary—including the kubelet, kata runtime, and VMM—are considered untrusted. This fundamental assumption drives the need for robust in-VM security mechanisms.

Ephemeral Storage Implementation

For ephemeral storage, the design focuses on ensuring confidentiality for data that does not outlive the container but is too large for memory.

1. CSI Driver Integration: To integrate with Kubernetes, a custom CSI driver is implemented. When a user defines a volume in their container spec referencing this CSI driver (e.g., Coco local CSI), the kubelet triggers the driver.
2. Block Device Creation: The CSI driver creates an empty block storage device on the host.
3. VM Ingestion: This host-side block device is then passed into the confidential VM using Virtio-blk, a standard transport for virtualized devices in Linux.
4. In-VM Encryption: Inside the VM, the Confidential Data Hub (CDH) component takes over. The CDH generates a random encryption key entirely within the confidential VM. This key never leaves the VM, and because the VM's memory is encrypted by the TEE, the key's confidentiality is guaranteed.
5. DM-Crypt and DM-Integrity: The CDH then uses this key with Linux kernel modules DM-Crypt and DM-Integrity to transform the raw block device from the host into an encrypted device. DM-Crypt handles the encryption, while DM-Integrity adds protection against tampering. The kernel then exposes a plaintext device to the container, making the encryption transparent to the application.

Securing the Container Specification and Devices with Policy

A significant challenge arises from the untrusted path of the container specification. The kubelet transforms the high-level YAML spec into a lower-level JSON spec before it reaches the kata agent. Malicious host components could potentially tamper with this JSON spec.

1. Policy Generation: A Policy Generator tool takes the original YAML spec and generates a corresponding security policy in the Rego language. Rego, originating from the Open Policy Agent (OPA) project, allows for expressive policy definitions.
2. Policy Injection: This generated policy is injected back into the YAML spec (typically as a base64-encoded annotation) and sent to the kata agent along with the JSON spec.
3. In-VM Enforcement: Inside the confidential VM, the kata agent uses regor-rust (a Rust implementation of Rego developed at Microsoft) to enforce that the received JSON spec precisely matches the injected security policy. This prevents unauthorized modifications to the container's configuration or runtime environment.
4. Device Protection: Crucially, the security policy also includes metadata about any devices injected into the VM, such as the ephemeral storage block device. This ensures that only authorized devices, matching the policy, can be mounted, mitigating a potential attack vector.

Establishing Trust in the Policy via Remote Attestation

While the security policy enforces integrity inside the VM, the policy itself originates from outside the VM and is thus initially untrusted.

1. Reference Policy: A trusted, remote Remote Attestation Service (RAS) stores the "ground truth" or reference security policy for a given workload.
2. Attestation Process: After the kata agent has enforced the policy against the JSON spec, an attestation agent (within the VM) fetches the "observed policy" (the one injected into the VM). This observed policy is signed by the TEE, providing cryptographic proof of its origin from within the confidential environment.
3. Verification: The attestation agent sends the TE-signed observed policy to the RAS. The RAS then compares this observed policy against its stored reference policy. If they match, the policy is deemed trusted. This completes the loop, establishing trust in the policy, and consequently, in the container and its attached devices.

Persistent Storage Design (Work in Progress)

For persistent storage, the problem is more complex as data must survive container restarts and be accessible across different confidential VMs. This design is currently in the conceptual phase:

1. Pre-encrypted Remote Storage: The CSI driver for persistent storage would provision already encrypted storage from a cloud service.
2. Key Broker Service (KBS): When the confidential VM needs to access this storage, the CDH will request the decryption key from a Key Broker Service (KBS).
3. Attestation-driven Key Release: The KBS, like the RAS, is considered a trusted component. Before releasing the key, the KBS will interact with the attestation agent and the RAS. The attestation agent sends the TE-signed security policy to the KBS, which forwards it to the RAS for verification against the reference policy.
4. Decryption and Mounting: If the attestation succeeds, the RAS signals the KBS to release the decryption key to the attestation agent, which then passes it to the CDH. The CDH uses this key to decrypt the remote storage and mount it into the container.

Bombo noted that this design places the attestation protocol control within the VM image, making it difficult for users to bring their own attestation protocols. Future designs are exploring moving the CDH and attestation agent into the container itself, enabling greater flexibility but introducing challenges like new API design and bootstrapping problems.

Demo / Proof of Concept

▶ Watch: Simplified lifecycle diagram of a CoCo container in Kubernetes (6:20)

Aurélien Bombo provided a live demonstration of the ephemeral storage solution, based on the code in his active pull request. The demo aimed to showcase the end-user experience and the transparency of the encryption layer.

The demonstration involved deploying a container to a Kubernetes cluster using a YAML specification. Key elements of the YAML included:

• A container named my-app.
• A base64-encoded representation of the security policy embedded as an annotation within the container spec. This policy, though truncated in the display, is a significant payload confirming the detailed rules governing the container's execution.
• A reference to the custom Coco local CSI driver.
• A claim for 10 gigabytes of storage against this driver.
• The volume mounted inside the container at the path /mnt/encrypted.

After successfully deploying the container using kubectl apply, Bombo opened a debug shell into the running confidential container. Inside the container, he executed df -h to inspect the mounted file systems. The output clearly showed a file system mounted on /mnt/encrypted with approximately 10 GB of available space, formatted as ext4. Crucially, the device backing this filesystem was displayed as a /dev/mapper device. This dev/mapper prefix is a strong indicator that the device is managed by the Linux device mapper subsystem, specifically confirming the use of DM-Crypt (and implicitly DM-Integrity) to encrypt the underlying host-provided block device.

To further illustrate the transparency, Bombo navigated into the /mnt/encrypted directory. Initially, the directory was empty. He then used echo "very sensitive data" > confidential.txt to write a confidential payload into a file within this encrypted mount point. Listing the directory (ls) showed the new file, and cat confidential.txt successfully displayed its contents. This confirmed that from the perspective of the application or end-user inside the confidential container, the storage operates like any standard filesystem, with the encryption and decryption processes handled entirely by the kernel in the background, without requiring any specific setup or configuration from the user. This transparency is vital for broad adoption and ease of use for developers.

Defensive Implications

▶ Watch: Untrusted components and introduction to storage types (8:00)

The work presented by Aurélien Bombo on secure storage with Confidential Containers offers several critical implications for security defenders:

1. Adopt Confidential Computing for Sensitive Workloads: Organizations handling highly sensitive data, particularly in multi-tenant cloud environments or where trust in the cloud provider is limited, should strongly consider migrating eligible workloads to Confidential Containers. This extends data protection beyond at-rest and in-transit to data in use, closing a significant security gap.
2. Leverage Kubernetes CSI for Secure Storage: Defenders should integrate and utilize CSI drivers specifically designed for confidential storage. This allows secure ephemeral and persistent storage to be provisioned and managed directly through Kubernetes, aligning with modern cloud-native deployment practices while enforcing strong security properties.
3. Implement and Enforce Robust Security Policies: The container security policy is a cornerstone of CoCo's security model. Defenders must understand how to define, generate, and manage these policies (e.g., using Rego). Strict policies should dictate allowed container configurations, injected devices, and other runtime parameters to prevent malicious modifications by an untrusted host.
4. Establish and Protect Remote Attestation Services: The trustworthiness of the entire confidential computing environment hinges on the Remote Attestation Service (RAS). Defenders must ensure that the RAS is itself highly secure, properly configured with trusted reference policies, and protected from tampering. The RAS acts as a critical arbiter of trust.
5. Understand Key Management for Persistent Storage: For persistent confidential storage, the Key Broker Service (KBS) becomes paramount. Defenders need to understand its role in key lifecycle management, its reliance on attestation, and ensure its secure deployment and operation. Secure key management is non-negotiable for data confidentiality.
6. Be Aware of Performance Overheads: While security is enhanced, it comes with a cost. Defenders should anticipate and measure potential performance overheads related to container creation time (due to setup work for storage and attestation, measured in seconds) and I/O operations (due to on-the-fly encryption/decryption by DM-Crypt). These factors must be considered during planning and resource allocation for production deployments.
7. Monitor and Audit: As with any critical security infrastructure, comprehensive monitoring and auditing of CoCo environments, especially interactions with CSI drivers, attestation services, and key brokers, are essential to detect anomalies and potential breaches.
8. Stay Informed on Evolving Designs: The field of confidential computing, particularly for storage, is rapidly evolving. Defenders should stay updated on new design patterns (e.g., moving attestation components into containers) and their associated security implications and benefits.

Key Takeaways

• Confidential Containers (CoCo) extend data protection to in-use storage, addressing a critical gap in traditional confidential computing.
• Both ephemeral and persistent storage can be secured within confidential VMs by leveraging in-VM key generation (for ephemeral) or Key Broker Services (KBS) and Remote Attestation Services (RAS) (for persistent).
• Kubernetes Container Storage Interface (CSI) drivers are fundamental for seamlessly integrating confidential storage into cloud-native deployments.
• A robust security policy (written in Rego) and verified by remote attestation is crucial for ensuring the integrity of container specifications and guarding against untrusted host components.
• The encryption layer for confidential storage is transparent to the end-user application, simplifying adoption and reducing developer overhead.
• Implementing confidential storage introduces performance overheads, particularly during container creation and I/O operations, which require careful consideration and measurement.

About the Speaker(s)

Aurélien Bombo is a prominent figure in the confidential computing and container communities, currently working at Microsoft. He is actively involved with the Confidential Containers project, contributing to various aspects, notably continuous integration (CI) and, more recently, secure storage. Additionally, Bombo serves on the architecture committee for the Kata Containers project, a group responsible for steering the project's direction and making final decisions. Within Microsoft, his primary assignment is with the Linux confidential platform as part of the Azure Linux OS team. Originally from Belgium, Aurélien is now based in Chicago and presented at KubeCon EU, marking his first attendance at the European conference.