The Road to Trust: Building Enclaves within Confidential VMs

The Road to Trust: Building Enclaves within Confidential VMs

Speakers: Wenhao Wang (representing Institute of Information Engineering Chinese Academy of Sciences, Ant Group, and Indiana University)

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=IHvQy6qPFe8

Overview

This talk, "The Road to Trust: Building Enclaves within Confidential VMs," presented by Wenhao Wang, introduces Nested SGX, a novel architecture designed to enhance the security of confidential computing environments. The core problem Nested SGX addresses is a critical vulnerability in existing VM-based Trusted Execution Environments (TEEs): what happens if the guest operating system (OS) itself is compromised? While VM-based TEEs like AMD SEV encrypt the entire virtual machine, they traditionally trust the guest OS to manage applications. Nested SGX challenges this assumption by creating isolated enclaves within confidential VMs, ensuring that sensitive applications remain secure even if the underlying guest OS is compromised.

The significance of Nested SGX lies in its hybrid approach, which merges the robust process-based isolation of Intel SGX with the comprehensive VM-level encryption of AMD SEV. By minimizing the Trusted Computing Base (TCB) to only the enclave and a slim security monitor, it drastically reduces the attack surface compared to prior attempts like VSGX, which treated the entire VM as trusted. This work is a crucial step towards making confidential computing more resilient and trustworthy, particularly for cloud environments where guest OS compromise is a persistent concern.

This research, a collaborative effort between the Institute of Information Engineering Chinese Academy of Sciences, Ant Group, and Indiana University, demonstrates a practical and efficient solution. Nested SGX not only achieves strong security guarantees but also maintains compatibility with existing Intel SGX applications, thereby facilitating broader adoption. Its implementation on commercial hardware and thorough evaluation against Intel SGX benchmarks affirm its efficiency and readiness for real-world deployment, paving the way for more secure cloud-native confidential workloads.

Background

▶ Watch: Identifying the core security challenge in VM-based TEEs (1:40)

Trusted Execution Environments (TEEs) are a fundamental technology for protecting sensitive code and data from unauthorized access, even from privileged software on the same system, such as the operating system or hypervisor. TEEs ensure security by restricting access to sensitive computations. Historically, TEEs have branched into two primary families: process-based and VM-based.

Process-based TEEs, exemplified by Intel SGX, operate by isolating specific application code and data into secure enclaves within a normal execution environment. These enclaves are cryptographically protected, ensuring that even a compromised OS or hypervisor cannot access their contents. The workflow for creating and managing these enclaves is highly structured, ensuring sensitive data remains protected. However, Intel SGX requires specific hardware and extensive modifications to applications to utilize its features.

VM-based TEEs, such as AMD SEV (Secure Encrypted Virtualization), Intel TDX (Trust Domain Extensions), and ARM CCA (Confidential Compute Architecture), take a different approach. They encrypt the entire virtual machine's memory, protecting it from the hypervisor. This provides a broader scope of protection, as the entire guest OS and its applications operate within an encrypted domain. VM-based TEEs have become more prevalent due to their ease of deployment for existing VM workloads, as they require fewer application-level changes than process-based TEEs.

However, VM-based TEEs introduce a new security concern: what if the guest OS itself is compromised? While the VM's memory is encrypted from the hypervisor, the guest OS within the VM still has full control over the applications running inside it. If malicious actors gain control of the guest OS, they can potentially compromise applications and sensitive data, undermining the trust model of the confidential VM. This critical challenge forms the core motivation for Nested SGX. Prior research, such as VSGX, attempted to address this by running applications in a separate, untrusted VM, isolating them from an enclave, but it treated the entire VM as part of the TCB (Trusted Computing Base), leading to a massive TCB and significant overhead for virtual machine switching. The need for a solution that minimizes the TCB while maintaining efficiency and compatibility became paramount.

Key Findings

▶ Watch: Summary of Nested SGX's three main contributions (4:10)

Nested SGX presents several pivotal contributions to the field of confidential computing, directly addressing the limitations of prior approaches and the inherent challenges of VM-based TEEs.

Firstly, a core finding is that Nested SGX successfully minimizes reliance on the guest OS by establishing the enclave as the only trusted component within the confidential VM. This significantly shrinks the TCB, thereby reducing the attack surface. In contrast to VM-based TEEs that implicitly trust the guest OS, Nested SGX explicitly assumes the guest OS is untrusted, building a robust isolation layer above it.

Secondly, the architecture demonstrates remarkable compatibility with existing Intel SGX applications. By designing Nested SGX with a management model for enclaves that closely mirrors that of Intel SGX, the effort required to port existing SGX-enabled applications is substantially reduced. This compatibility is crucial for accelerating the adoption of enhanced confidential computing solutions.

Thirdly, the research validates the efficiency and availability of Nested SGX through its implementation on commercial hardware. Rigorous evaluation shows that the overhead introduced by Nested SGX is comparable to that of native Intel SGX hardware, affirming its practical viability for real-world scenarios. This finding is critical, as high overhead has historically been a barrier to the widespread adoption of advanced security mechanisms.

Finally, Nested SGX innovatively utilizes AMD SEV-SNP's Virtual Machine Privilege Levels (VMPL) feature to achieve fine-grained privilege separation. This hardware-assisted isolation mechanism allows for a clear logical and physical separation between trusted components (security monitor and enclaves) and untrusted components (guest OS and applications), even within the same confidential VM. This leverages the underlying hardware capabilities to build a strong security foundation, which is a key differentiator of this work.

Technical Deep Dive

▶ Watch: How Nested SGX uses AMD SEV-SNP VMPL for privilege separation (5:05)

Nested SGX leverages the advanced features of modern VM-based TEEs, specifically AMD SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging), to construct a highly isolated environment for enclaves within a confidential VM. The architecture's robustness hinges on three key features: a dedicated security monitor, comprehensive memory isolation, and a verifiable measurement and integrity tracking mechanism.

The cornerstone of Nested SGX's security model is the strategic use of Virtual Machine Privilege Levels (VMPL) provided by AMD SEV-SNP. VMPL allows for the separation of permissions and privileges across different levels within a VM, ranging from VMPL0 (highest privilege) to VMPL3 (lowest privilege). In Nested SGX, the trusted components – the security monitor and the enclaves themselves – reside in VMPL0. Conversely, the untrusted guest OS and its applications operate at VMPL1. This clear privilege separation ensures that even if the VMPL1 guest OS is compromised, it cannot compromise the VMPL0 trusted components.

Security Monitor and Enclave Life Cycle Management

The security monitor is a critical component, running in the kernel mode of VMPL0. Its primary responsibility is to manage the entire life cycle of enclaves, from creation to destruction. To achieve compatibility with Intel SGX, the security monitor includes an SGX instruction emulation layer, allowing it to intercept and handle SGX-specific instructions and requests.

The process for an application to interact with an enclave is meticulously controlled:

1. Request Initiation: An application in VMPL1 sends a request via a control sys_call to a dedicated driver within the guest OS.
2. VMPL Switching: The guest OS driver, recognizing an enclave-related request, triggers a VMPL switch to transfer control to the VMPL0 security monitor.
3. Enclave Initialization/Measurement: The security monitor then handles the initialization and cryptographic measurement of the enclave, establishing its integrity.
4. Synchronized Entry/Exit: For an application to enter an enclave, the control flow first traps into the security monitor. The monitor then uses a sys_return instruction to direct execution to the correct entry point within the enclave. Once the enclave's computation is complete, it uses a syscall to return control to the monitor. The monitor then performs another VMPL switch to return execution to the guest OS driver in VMPL1.
5. Asynchronous Exit (AEX) Handling: In cases of an asynchronous exit (e.g., an interrupt or exception from within the enclave), the exception is handled by the VMPL0 security monitor. The monitor stores the enclave's context, switches back to VMPL1, and allows the guest OS to handle the initial exception. When a subsequent u_resume instruction (or similar) is issued, the security monitor restores the enclave's context and uses an I_return instruction to resume the enclave's execution.
6. Enclave Deletion: Upon receiving a request to delete an enclave, the security monitor securely clears and recycles the associated enclave pages, preventing data leakage.

Memory Isolation

Memory isolation is fundamental to Nested SGX's security. This isolation is established early in the boot process and maintained rigorously:

• Boot-time Control: During the system boot, the VMPL0 security monitor initializes the guest OS and configures its page tables. It then hands over control to the guest OS BIOS. Critically, the pages accessible to the guest OS are entirely under the control of the security monitor.
• EPC Management: The Enclave Page Cache (EPC), where enclave code and data reside, is exclusively allocated and managed by the VMPL0 security monitor. The monitor maintains an EPCM (Enclave Page Cache Map) data structure to track the state of EPC pages and handles any page faults originating from enclaves.
• Shadow Page Tables: The security monitor provides shadow page tables for both enclaves and applications. While the guest OS manages the application's page tables, the security monitor directly manages the enclave's page tables. This dual-page table approach is crucial.
• Data Sharing: To enable secure data sharing between enclaves and applications, the security monitor can maintain the same Guest Physical Address (GPA) to Host Physical Address (HPA) mapping. However, it explicitly de-privileges the VMPL1 guest OS from executing the content of these shared pages, ensuring that shared data can be accessed but not maliciously executed by a compromised guest OS.
• VMPL-Guaranteed Isolation: The VMPL feature inherently guarantees isolation. VMPL1 (guest OS) simply cannot access the secure memory regions allocated to VMPL0 (security monitor and enclaves). Furthermore, the security monitor actively ensures that the page tables of enclaves do not map to any GPAs belonging to the security monitor or the guest OS, reinforcing isolation between all components.

Measurement and Integrity Tracking

Establishing a root of trust and maintaining integrity is vital for any TEE. In Nested SGX:

• Hardware Root of Trust: The chain of trust begins at the AMD Root CA, which guarantees the integrity and trustworthiness of the Confidential VM (CVM) itself.
• Attestation Identity Key (AIK): The VMPL0 security monitor generates its own Attestation Identity Key (AIK).
• Customized Attestation Reports: The ultimate attestation report generated by Nested SGX includes not only standard CVM attestation data but also custom privacy-generated information specific to the Nested SGX environment. This allows remote parties to cryptographically verify the integrity and configuration of both the underlying CVM and the Nested SGX security monitor and enclaves.

Security Analysis

The design of Nested SGX is predicated on a robust security analysis:

• Untrusted Guest OS: The untrusted application and guest OS (including the Nested SGX driver) run at VMPL1. Their inability to harm the VMPL0 security monitor or enclaves is guaranteed by the separate page tables and the VMPL-enforced isolation.
• Untrusted Host VMM: An untrusted Virtual Machine Monitor (VMM) running on the host machine can only observe patterns of ECall, OCall, and AEX (Asynchronous Exit), which are inherent to any SGX-like environment. Crucially, Nested SGX introduces no additional attack vectors compared to native Intel SGX in this regard.
• Untrusted Enclaves: Even if an enclave itself were somehow untrusted, its actions are constrained. The security monitor controls its page table, and all syscall-related steps taken by the enclave are mediated by and cannot bypass the security monitor. This ensures that a rogue enclave cannot gain unauthorized privileges or access.

Demo / Proof of Concept

▶ Watch: Explaining Nested SGX's robust memory isolation techniques (8:20)

While no live demonstration was explicitly detailed, the presentation thoroughly covered the implementation and evaluation of Nested SGX, serving as its proof of concept.

The Nested SGX implementation comprises approximately 7,500 Lines of Code (LoC). It is built upon the Open Source Linux L Secure VM Service Module Framework, indicating a commitment to an open and extensible base. The system was deployed and evaluated on an AMD server, specifically leveraging the AMD SEV-SNP hardware capabilities.

For evaluation, the researchers conducted a comprehensive comparison of Nested SGX against both Intel SGX hardware and the simulation mode of Intel SGX on the same AMD server. This direct comparison provides a strong baseline for assessing performance. The evaluation employed a mix of microbenchmarks and real-world applications:

• Microbenchmarks: These tests assessed fundamental operations critical to enclave performance.
• Real-world Applications: The team evaluated various applications, including:
• rust-crypto: A Rust implementation of cryptographic primitives, assessing computation-intensive tasks within enclaves.
• endbench: A general benchmarking tool.
• SSL IO: Measuring secure I/O performance.
• SQLite: A popular embedded database, testing data-centric operations.
• TLS servers: Evaluating the performance of secure communication protocols.

The results of this evaluation were highly encouraging. Nested SGX was found to be significantly quicker than previous approaches like VSGX, demonstrating its efficiency in reducing overhead. While the Rust implementation of cryptographic operations (rust-crypto) was noted to be "a little bit slower" than native Intel SGX hardware, the overall overhead of Nested SGX was reported as comparable to that of Intel SGX. This affirms its efficiency and practical viability, making it a compelling alternative for deploying secure enclaves within confidential VMs without prohibitive performance penalties.

The authors also acknowledged certain limitations of the current work: not all SGX model features are fully supported, the scheduling of enclave threads is currently lacking, and applications not written with an SDK might not be directly supported. These points highlight areas for future development but do not diminish the significance of the core architectural contributions.

Defensive Implications

▶ Watch: Establishing the root of trust and attestation in Nested SGX (10:20)

Nested SGX presents significant defensive implications for organizations and individuals concerned with data security in cloud and virtualized environments, particularly those utilizing confidential VMs.

Firstly, for users and organizations deploying sensitive workloads in confidential VMs, Nested SGX offers a crucial layer of defense against a compromised guest OS. While AMD SEV-SNP protects the VM from the hypervisor, Nested SGX extends this trust boundary within the VM, ensuring that critical applications and their data remain isolated and secure even if the guest OS itself falls victim to an attack. This drastically reduces the TCB, providing a more robust security posture than traditional CVM deployments. Defenders can now have higher confidence that their application's secrets are protected, even from privileged software they control but cannot fully trust.

Secondly, for developers and architects of confidential applications, Nested SGX provides a path to port existing Intel SGX applications to CVMs with enhanced security guarantees. The compatibility with SGX's enclave management model means that the effort to migrate or adapt applications is minimized. This allows developers to leverage the benefits of VM-based TEEs (such as easier deployment of existing VM images) without sacrificing the strong process-level isolation offered by SGX. It offers a "best of both worlds" scenario, enabling the deployment of highly sensitive components within a broader, encrypted VM environment.

Thirdly, the explicit leveraging of AMD SEV-SNP's VMPL feature demonstrates how hardware-assisted security mechanisms can be effectively utilized to build robust software architectures. Defenders should look for platforms that offer such granular privilege separation capabilities, as they are fundamental to constructing highly isolated computing environments. This work provides a blueprint for how to design systems that assume components are untrusted and use hardware features to enforce strict boundaries.

Finally, by explicitly addressing the guest OS compromise scenario, Nested SGX encourages a defense-in-depth strategy. It acknowledges that no single security layer is perfect and provides a critical secondary layer of isolation for the most sensitive application components. This approach is vital in complex cloud environments where the attack surface is vast and the integrity of every component cannot be absolutely guaranteed. Organizations should consider Nested SGX as a means to achieve a higher level of assurance for their most critical data processing.

Key Takeaways

• Addresses Guest OS Compromise: Nested SGX provides a solution to the critical security vulnerability where the guest OS in VM-based TEEs (like AMD SEV) is compromised, ensuring application security even in such scenarios.
• Hybrid TEE Architecture: It combines the process-based isolation benefits of Intel SGX with the VM-level encryption of AMD SEV-SNP, creating a robust hybrid model that minimizes the Trusted Computing Base (TCB).
• Leverages AMD SEV-SNP VMPL: Nested SGX utilizes AMD SEV-SNP's Virtual Machine Privilege Levels (VMPL) to achieve fine-grained privilege separation, running trusted components (security monitor, enclaves) at VMPL0 and the untrusted guest OS at VMPL1.
• High Compatibility and Efficiency: The architecture is compatible with existing Intel SGX applications, reducing porting effort, and demonstrates comparable performance overhead to native Intel SGX hardware on commercial AMD servers.
• Security Monitor for Enclave Management: A dedicated security monitor in VMPL0 manages the entire enclave life cycle, including initialization, measurement, synchronized/asynchronous entry/exit, and memory isolation, through an SGX instruction emulation layer.
• Enhanced Data Protection: Nested SGX offers stronger memory isolation for enclaves and ensures the integrity of the computing environment through a hardware root of trust and customized attestation reports.

About the Speaker(s)

The talk "The Road to Trust: Building Enclaves within Confidential VMs" was presented by Wenhao Wang at the NDSS Symposium. This research is a collaborative effort, representing joint work from the Institute of Information Engineering Chinese Academy of Sciences, Ant Group, and Indiana University. While specific titles for Wenhao Wang were not provided in the transcript, his presentation of this detailed technical work highlights his expertise and contributions in the field of confidential computing and trusted execution environments, particularly in addressing complex security challenges within virtualized environments. The collaboration across academic and industry institutions underscores the practical relevance and rigorous research behind Nested SGX.