HYPERPILL: Fuzzing for Hypervisor-bugs by Leveraging the Hardware Virtualization Interface

HYPERPILL: Fuzzing for Hypervisor-bugs by Leveraging the Hardware Virtualization Interface

Speakers: Alexander Bulekov, Qiang Liu, Manuel Egele, Mathias Payer

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=bcwS54EHaso

Overview

In the realm of modern computing, hypervisors serve as the foundational layer enabling the efficient and secure execution of multiple virtual machines (VMs) on a single physical host. Their ubiquity spans critical infrastructure, from vast cloud environments and personal computing to specialized applications in automotive systems and vulnerability research. The integrity and security of the hypervisor are paramount; a compromise at this level can lead to a complete breakdown of isolation, allowing a rogue VM to seize control of the underlying host and impact all neighboring virtual machines. This talk, "HYPERPILL: Fuzzing for Hypervisor-bugs by Leveraging the Hardware Virtualization Interface," presented by Alexander Bulekov and co-authored by Qiang Liu, Manuel Egele, and Mathias Payer at USENIX Security '24, addresses the critical challenge of systematically identifying vulnerabilities in these indispensable components.

The core contribution of HYPERPILL is a novel, generic fuzzing architecture designed to uncover bugs in arbitrary hypervisors—whether open-source or closed-source—on a specific hardware architecture. It achieves this by ingeniously leveraging the standardized hardware virtualization interface (HVI) exposed by the CPU itself. By focusing on this architectural commonality, HYPERPILL bypasses many of the traditional hurdles associated with hypervisor fuzzing, such as the diversity of implementations and the complexity of their internal structures. The research demonstrates a significant leap forward in automated bug finding for hypervisors, securing a layer of software that is increasingly vital to our digital infrastructure.

The talk highlights the formidable challenges in applying state-of-the-art fuzzing techniques to hypervisors, ranging from their intricate input spaces to their low-level system nature. HYPERPILL's innovative approach, centered on snapshotting, inspection, and intelligent fuzzing of the CPU's virtualization interface, proves remarkably effective. It not only achieves superior code coverage compared to previous methods but also led to the discovery of 26 previously unknown bugs across leading hypervisor platforms, reinforcing its significance in enhancing the resilience of virtualized environments.

Background

▶ Watch: Introduction to HyperPill and hypervisor security. (0:00)

Fuzzing, a widely recognized and effective automated bug-finding technique, faces substantial obstacles when applied to hypervisors. The speaker elucidated several key challenges:

1. Complex Input Space: Hypervisors expose a highly intricate virtualization interface to VMs, comprising numerous virtual devices. This interface is composed of complex system code, often written in C or C++, making it difficult to even harness a single virtual device for fuzzing, let alone an entire hypervisor. Meaningful interactions often require sequences of operations with intricate dependencies (e.g., a Port I/O operation creating a Memory-Mapped I/O region, which then triggers a DMA access).
2. Diverse Implementations: Hypervisors vary wildly in their design. Some run as kernel modules within a general-purpose operating system (e.g., KVM), while others might rely on microkernels or even full VMs for emulation (e.g., Hyper-V's architecture). This architectural diversity makes it exceptionally difficult to develop a generic fuzzing solution that can adapt to different internal structures and APIs.
3. Low-Level System Software: Hypervisors are large, low-level system components. This precludes the use of common fuzzing optimizations like in-process fuzzing or fork servers, which are typically tailored for smaller user-space libraries or parsers. Fuzzing a hypervisor often requires rebooting or resetting a complex system state, leading to significant performance overhead.
4. Reliance on Sophisticated CPU Features: To deliver near-native performance, hypervisors extensively utilize hardware virtualization extensions provided by modern CPUs (e.g., Intel VT-x, AMD-V). This deep hardware interaction makes them challenging targets for traditional software-level fuzzing.

Despite these challenges, the authors identified a critical architectural commonality: regardless of implementation, every hypervisor seeking to provide near-native performance on a given architecture must interact with the CPU through an identical hardware virtualization interface. On Intel x86, this interface is primarily managed through the Virtual Machine Control Structure (VMCS). The VMCS is a memory-resident data structure that the hypervisor configures to define the guest's virtual CPU state, allocated memory, MMIO regions, and other virtualization parameters. The CPU, in turn, uses the VMCS to communicate information about VM exits—events where control transfers from the guest back to the hypervisor (e.g., due to a privileged instruction, I/O request, or memory access). This standardized CPU interface became the lynchpin for HYPERPILL's generic approach.

The attack surface of a hypervisor, from the perspective of a VM, encompasses several key communication channels:

• Port I/O (PIO): Allows the VM to perform I/O operations through specific CPU instructions (eIN/OUT on x86) that trigger a VM exit into the hypervisor for emulation.
• Model Specific Registers (MSRs): Special CPU registers that can be read or written by privileged instructions (RDMSR/WRMSR), often triggering VM exits for hypervisor intervention.
• Memory-Mapped I/O (MMIO): Regions of physical memory that, when accessed by the guest, are configured by the hypervisor to trigger a VM exit, allowing the hypervisor to emulate device behavior.
• Hypercalls: A rapid, VM-specific communication mechanism where the guest issues a special instruction to directly invoke a hypervisor service. Each hypervisor can define its own calling conventions and argument passing through registers.
• Direct Memory Access (DMA): Unlike the others, DMA is not directly associated with a VM exit initiated by the guest. Instead, it involves the hypervisor accessing guest memory while handling other requests (e.g., reading data from a buffer provided by the guest during a network packet emulation). This is crucial for high-performance applications like storage and networking.

The sheer volume and semantic complexity of these combined input spaces—spanning gigabytes of guest memory, numerous Port I/O addresses, and MMIO regions—underscore the difficulty of effective hypervisor fuzzing. HYPERPILL's innovation lies in its ability to systematically navigate and fuzz these diverse interfaces by treating the CPU's HVI as the universal point of interaction.

Key Findings

▶ Watch: Core insight: All hypervisors use identical CPU interface. (4:00)

HYPERPILL represents a significant advancement in the field of hypervisor security, delivering several key findings and contributions:

• Generic Hypervisor Fuzzing: The most significant finding is the feasibility and effectiveness of fuzzing any hypervisor on a given architecture by exclusively leveraging the hardware virtualization interface (e.g., VMCS on x86). This generic approach overcomes the implementation diversity challenge, making it applicable to both open-source and closed-source hypervisors.
• Discovery of 26 Bugs: Across three major hypervisor platforms—QEMU/KVM (Linux), Hyper-V (Windows), and Apple Virtualization Framework (macOS)—HYPERPILL successfully identified a total of 26 unique bugs. These vulnerabilities ranged in severity and impact from Denial of Service (DoS) conditions to critical memory corruptions, highlighting the practical security implications of the research.
• Superior Code Coverage: In comparative benchmarks against previous state-of-the-art fuzzing approaches on QEMU, HYPERPILL demonstrated higher code coverage for 10 out of 12 virtual devices. This improvement was particularly notable given HYPERPILL's inherent performance penalty due to its full-system snapshot emulation mode. The superior coverage is attributed to HYPERPILL's more granular and precise approach to DMA fuzzing, which operates at the instruction level.
• Precise DMA Fuzzing: HYPERPILL's method for fuzzing Direct Memory Access (DMA) is more precise than prior techniques that rely on source-level hooking. By monitoring individual instruction accesses to guest memory within a full system emulator, HYPERPILL can accurately inspect and manipulate byte-level DMA interactions, which is crucial for uncovering subtle memory corruption bugs that coarser-grained methods might miss.
• Comprehensive Input Surface Coverage: The fuzzer effectively covers all major input surfaces exposed by hypervisors to VMs, including Port I/O, MMIO, MSRs, Hypercalls, and crucially, DMA, providing a holistic fuzzing solution.

These findings collectively demonstrate that focusing on the CPU's standardized virtualization interface offers a powerful and efficient paradigm for automated bug discovery in hypervisors, significantly bolstering the security posture of virtualized environments.

Technical Deep Dive

▶ Watch: Understanding hypervisor input spaces and attack surface. (4:15)

HYPERPILL's architecture is meticulously designed around a three-stage process: snapshot collection, snapshot inspection, and fuzzing. This methodology strategically leverages the CPU's hardware virtualization interface to achieve generic and comprehensive hypervisor fuzzing.

Stage 1: Snapshot Collection

The first critical step involves capturing a precise snapshot of the target hypervisor's state. The timing of this snapshot is crucial: it must be taken just as the hypervisor is about to begin handling an input request from a VM. This moment is typically signified by a VM exit.

To achieve this, HYPERPILL employs a technique called nested virtualization. A small, custom hypervisor, dubbed HYPERPILL-SNAP, is introduced as an outer layer. HYPERPILL-SNAP runs the target hypervisor (the one being fuzzed), which in turn runs the guest VM. HYPERPILL-SNAP is designed to be very simple, forwarding most VM exits directly to the target hypervisor as if it were running natively.

However, HYPERPILL-SNAP intercepts one specific type of event: a specially crafted hypercall injected by the guest VM. When this "special hypercall" is detected, HYPERPILL-SNAP pauses both the target hypervisor and its guest VM. At this precise moment, a full-system snapshot is collected. This snapshot captures the entire memory state, all register states (including CPU registers and control registers), and crucially, the contents of the VMCS (Virtual Machine Control Structure) that governs the interaction between the CPU and the target hypervisor. The significance of this timing is that the next instruction the target hypervisor would execute after the snapshot is taken is the very beginning of its handler for the VM exit caused by our injected hypercall. This provides a clean, consistent entry point for fuzzing.

Stage 2: Snapshot Inspection and Enumeration

With a comprehensive snapshot in hand, the next stage involves inspecting its contents to enumerate the hypervisor's exposed input surfaces. The snapshot contains the hypervisor's memory and register state, as well as the active VMCS.

By analyzing the VMCS, HYPERPILL can extract vital information configured by the target hypervisor for the guest. This includes:

• Memory-Mapped I/O (MMIO) regions: The VMCS specifies which guest physical memory regions are configured as MMIO, triggering VM exits upon access.
• Port I/O (PIO) regions: The VMCS can indicate which I/O ports are sensitive and should cause VM exits.
• Guest memory allocation: By observing the memory mappings and structures within the snapshot, HYPERPILL can identify the contiguous regions of memory that the hypervisor has allocated for the guest VM's use. This is crucial for later DMA fuzzing.

In addition to VMCS analysis, the system can perform some basic probing within the snapshot to confirm or refine these identified regions. The goal of this stage is to create a map of the attack surface that the hypervisor presents to the VM, ready for manipulation.

Stage 3: Fuzzing the Snapshot

The final and most intricate stage is the actual fuzzing process, which uses the collected and inspected snapshot. The fundamental idea is to replay arbitrary sequences of I/O operations into the hypervisor by modifying the VMCS within the snapshot, then resuming its execution in a full-system emulator.

1. Input Generation: The fuzzer generates a sequence of I/O operations as its input. This sequence can include Port I/O, MMIO, MSRs, and Hypercalls.
2. Snapshot Emulation and VMCS Manipulation: The collected snapshot is loaded into a full-system emulator. For each I/O operation in the fuzzer's input sequence:

• The emulator first modifies the VMCS state within the snapshot to reflect the desired type of I/O operation. For example, if the fuzzer wants to inject an MMIO write, the VMCS is updated to indicate an MMIO VM exit, specifying the address and data. Similarly, for Port I/O, the VMCS is modified to simulate a Port I/O VM exit, including the port number and data.
• Once the VMCS is configured for the current operation, the emulator "resumes" the snapshot. This effectively triggers a VM exit directly into the hypervisor's VM exit handler, as if the guest VM had just performed that specific I/O operation.
• The hypervisor then processes this emulated I/O request. If the operation is handled successfully, the hypervisor will typically attempt to "re-enter" the VM (i.e., return control to the guest).
• Instead of allowing the guest VM code to run, the emulator immediately intercepts this re-entry. It then takes the next I/O operation from the fuzzer's input, modifies the VMCS again, and triggers another VM exit directly into the hypervisor. This cycle continues until all operations in the fuzzer's sequence have been injected.

(Note: The talk mentions that injecting hypercalls has "more detail" and refers to the paper for specifics, implying a slightly different VMCS manipulation or invocation mechanism for these specific calls.)

1. Direct Memory Access (DMA) Fuzzing: DMA is a unique challenge because it doesn't directly correspond to a VM exit initiated by the guest. Instead, the hypervisor initiates DMA by accessing guest memory while handling another request (e.g., a Port I/O or MMIO operation).

• To fuzz DMA, HYPERPILL leverages the guest memory regions identified in Stage 2.
• While the snapshot is running in the full-system emulator, the emulator is configured to monitor every single instruction executed by the hypervisor.
• If an instruction is observed reading data from a guest memory region (which indicates a potential DMA operation), the emulator intervenes. It populates that specific region of guest memory with fuzzer-provided data. This data is typically part of the fuzzer's overall input, associated with the I/O operation that triggered the hypervisor's access to guest memory.
• This instruction-level monitoring allows for highly precise manipulation of data read by the hypervisor via DMA, enabling the discovery of bugs related to incorrect handling of guest-provided data buffers.

By combining VMCS manipulation for VM exit-driven I/O with instruction-level memory monitoring for DMA, HYPERPILL achieves comprehensive fuzzing across all major hypervisor input spaces. This detailed control over the CPU's interaction with the hypervisor is what makes HYPERPILL a powerful and generic tool for discovering vulnerabilities.

Demo / Proof of Concept

▶ Watch: Virtualization mechanics: VMCS, VM entry/exit. (6:00)

While the talk did not feature a live, step-by-step demonstration of a specific exploit or vulnerability, the effectiveness of HYPERPILL was thoroughly validated through its bug-finding capabilities. The research team successfully identified 26 distinct bugs across three leading hypervisor implementations: QEMU/KVM on Linux, Hyper-V on Windows, and the Apple Virtualization Framework on macOS. These findings serve as concrete proof of concept for HYPERPILL's design and efficacy. The identified vulnerabilities encompassed a range of security issues, from Denial of Service (DoS) conditions, which could disrupt virtual machine operation, to more critical memory corruptions, which often open pathways for privilege escalation or arbitrary code execution within the hypervisor itself. The discovery of these real-world vulnerabilities across diverse, widely-used hypervisors underscores the practical impact of HYPERPILL in enhancing the security posture of modern virtualized environments.

Defensive Implications

▶ Watch: HyperPill's three-stage fuzzing methodology introduced. (8:00)

The HYPERPILL research offers critical insights and actionable guidance for hypervisor developers, security researchers, and users alike:

1. Proactive Fuzzing Integration: Hypervisor developers should adopt and integrate similar hardware-interface-level fuzzing techniques into their continuous integration and testing pipelines. The ability of HYPERPILL to fuzz arbitrary hypervisors on a given architecture means that even organizations developing closed-source hypervisors can benefit from such an approach without requiring extensive source code instrumentation. Early detection of bugs, especially those leading to memory corruption or DoS, is paramount.
2. Hardening the Virtualization Interface: The research highlights the entire virtualization interface (Port I/O, MMIO, MSRs, Hypercalls, and particularly DMA) as a primary attack surface. Developers should pay meticulous attention to input validation, bounds checking, and state management within the handlers for these interfaces. The precision of HYPERPILL's DMA fuzzing, which operates at the instruction level, emphasizes the need for careful handling of guest-provided data buffers.
3. Architectural Approach to Security: The success of HYPERPILL underscores the value of an architectural perspective in security. By focusing on the standardized CPU-hypervisor interface (VMCS), the researchers found a common ground for attacking diverse implementations. This suggests that security audits and design reviews should place significant emphasis on the correctness and robustness of interactions with hardware virtualization extensions.
4. Timely Patching and Updates: For end-users and cloud providers, the discovery of 26 bugs across major hypervisors reinforces the critical importance of applying security patches and updates promptly. Hypervisor vulnerabilities can have catastrophic consequences, compromising entire host systems and all hosted VMs.
5. Complementary Security Measures: While fuzzing is highly effective, it should be complemented by other security measures such as static analysis, formal verification (for critical components), and thorough code reviews. The complexity of hypervisors necessitates a multi-faceted approach to security.
6. Security Research Tool: HYPERPILL itself, or its underlying methodology, serves as a powerful new tool for security researchers. Its generic nature allows for independent security assessments of hypervisors without requiring deep knowledge of their internal, often proprietary, implementation details. This facilitates broader and more efficient vulnerability discovery by the security community.

In essence, HYPERPILL serves as a stark reminder of the continuous need for rigorous security testing in the low-level systems software that forms the backbone of modern computing.

Key Takeaways

• Generic Hypervisor Fuzzing: HYPERPILL introduces a novel, architecture-specific fuzzing paradigm capable of targeting any hypervisor (open-source or closed-source) by leveraging the standardized hardware virtualization interface (e.g., VMCS on x86).
• Overcoming Fuzzing Challenges: It successfully addresses the long-standing difficulties in hypervisor fuzzing, including diverse implementations, complex input spaces, and the low-level nature of hypervisor software.
• Three-Stage Methodology: The fuzzer operates in three distinct stages: collecting full-system snapshots at critical VM exit points, inspecting these snapshots to enumerate input regions, and then intelligently fuzzing the hypervisor by manipulating the VMCS within an emulator to inject sequences of I/O operations.
• Comprehensive Input Coverage: HYPERPILL effectively fuzzes all major hypervisor-VM communication channels, including Port I/O, Memory-Mapped I/O (MMIO), Model Specific Registers (MSRs), Hypercalls, and crucially, Direct Memory Access (DMA).
• Instruction-Level DMA Fuzzing: Its innovative approach to DMA fuzzing, which monitors guest memory accesses at the instruction level within an emulator, provides superior precision compared to previous source-level hooking methods.
• Demonstrated Effectiveness: The research led to the discovery of 26 previously unknown bugs, ranging from Denial of Service to memory corruptions, across QEMU/KVM, Hyper-V, and the Apple Virtualization Framework, proving its real-world impact on hypervisor security.

About the Speaker(s)

The primary presenter of "HYPERPILL: Fuzzing for Hypervisor-bugs by Leveraging the Hardware Virtualization Interface" was Alexander Bulekov. He was joined in the research by co-authors Qiang Liu, Manuel Egele, and Mathias Payer. Based on the technical depth and innovative nature of their work presented at USENIX Security '24, the speakers are clearly accomplished researchers deeply involved in the fields of system security, virtualization, and automated bug finding. Their expertise lies in understanding the intricate interactions between operating systems, hypervisors, and underlying hardware to uncover vulnerabilities in critical system software.