Project Lightning Talk: Multi-Arch KubeVirt and CDI - C. A. Fillekes, WG Chair

Project Lightning Talk: Multi-Arch KubeVirt and CDI

Speakers: C. A. Fillekes, WG Chair

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Je6GIoagHvU

Overview

In a concise yet impactful lightning talk at KubeCon EU, C. A. Fillekes, a WG Chair and Red Hat partner engineer at IBM Systems, presented a critical advancement in cloud-native virtualization: the successful implementation of multi-architecture (multi-arch) support for KubeVirt and the Containerized Data Importer (CDI). This initiative extends the ability to run KVM virtual machines within Kubernetes containers beyond the ubiquitous x86 architecture to include ARM64 and the IBM mainframe architecture, S390X. The talk highlights a significant step towards truly heterogeneous and flexible cloud environments, addressing long-standing limitations in virtual machine portability and management across diverse hardware.

The core problem tackled by this project is the historical architectural lock-in, particularly concerning the import and conversion of virtual machines originating from x86-centric virtualization platforms like VMware. By enabling KubeVirt and CDI to operate natively on ARM64 and S390X, the project not only facilitates the deployment of VMs on these power-efficient and specialized architectures but also provides a streamlined, declarative mechanism to convert and manage VMs across this architectural spectrum. This capability empowers organizations to leverage existing virtualized assets in new, optimized environments, paving the way for more efficient data centers and edge deployments.

This work is particularly relevant for enterprises seeking to modernize their infrastructure, consolidate workloads, and optimize resource utilization across a varied hardware landscape. It offers a transparent and reproducible method for VM lifecycle management, eliminating manual, error-prone conversion processes. For anyone involved in cloud infrastructure, virtualization, or hybrid cloud strategies, understanding the implications of multi-arch KubeVirt and CDI is crucial for designing future-proof and resilient systems.

Background

▶ Watch: Introduction to Multi-Arch KubeVirt and CDI (0:00)

The evolution of cloud-native computing has largely centered around containers and Kubernetes, offering unparalleled agility and scalability for application deployment. However, a significant portion of enterprise workloads still reside within traditional virtual machines. KubeVirt emerged as a pivotal project to bridge this gap, enabling the management and execution of KVM virtual machines directly within a Kubernetes cluster. This allows operators to define, start, stop, and reconfigure VMs using standard Kubernetes primitives, such as YAML files, effectively bringing the benefits of container orchestration to virtualized workloads. KubeVirt provides features like VM templating and lifecycle management, integrating VMs seamlessly into a cloud-native ecosystem.

Complementing KubeVirt is the Containerized Data Importer (CDI), an essential component for managing storage for these virtual machines. CDI facilitates the use of persistent volume claims (PVCs) for KubeVirt VMs by way of data volumes. Its core functionalities include importing, cloning, and uploading attached storage for VMs. This is crucial for migrating existing VM images or creating new ones efficiently within the Kubernetes environment. However, a critical limitation has historically existed: CDI’s capabilities for importing, cloning, and uploading VMs from external sources, particularly those from VMware installations, relied on the Virtual Disk Development Kit (VDDDK). This toolkit, as Fillekes pointed out, is supported only on x86. This created a significant barrier for organizations looking to leverage their existing VMware-based virtual machine libraries on non-x86 architectures.

The context of "different architectures" is central to understanding the problem and the solution.

• x86: This is the dominant architecture in data centers globally, known for its widespread use in servers running Linux distributions like Red Hat CoreOS. It benefits from a vast ecosystem of VM templates, container registries, and robust Kubernetes support. Its instruction set is Complex Instruction Set Computing (CISC), and it uses Little Endian encoding for data representation.
• ARM64: Characterized by its Reduced Instruction Set Computing (RISC) architecture and significantly lower power consumption per CPU cycle (approximately half that of x86). ARM64 processors are prevalent in mobile devices, Raspberry Pis, edge devices, and increasingly in large server rooms. Like x86, it uses Little Endian encoding. The move towards ARM64 in data centers is driven by energy efficiency and specific workload optimizations.
• S390X: This refers to the IBM mainframe architecture. Despite common misconceptions about its relevance, S390X mainframes remain critical infrastructure, processing an astonishing 97% of global credit card transactions. They also feature a RISC instruction set and offer low power consumption. A key distinction of S390X is its use of Big Endian encoding, which presents unique challenges for porting software originally designed for Little Endian systems. Beyond basic porting, the S390X architecture offers many specialized features that, when enabled, allow for highly optimized and secure virtual machines.

The problem, therefore, was twofold: first, the lack of native KubeVirt and CDI support on ARM64 and S390X prevented the direct deployment and management of containerized VMs on these architectures. Second, even if KubeVirt could run, the x86-only nature of the VDDDK meant that the rich ecosystem of x86-based VMware VMs could not be easily imported or converted for use on ARM64 or S390X machines, creating isolated virtualization silos. This project aimed to break down these silos, enabling a truly multi-arch Kubernetes environment where VMs could be fluidly managed and transitioned across x86, ARM64, and S390X nodes.

Key Findings

▶ Watch: Overview of x86, ARM64, and S390X architectures (1:30)

The central achievement of this project is the successful extension of KubeVirt and Containerized Data Importer (CDI) capabilities to a multi-architecture landscape, specifically encompassing x86, ARM64, and S390X. This represents a significant leap forward in enterprise virtualization and cloud-native infrastructure, breaking down long-standing architectural barriers.

The key findings and contributions include:

1. Native KubeVirt and CDI Support on ARM64 and S390X: The core components of KubeVirt and CDI have been successfully ported and optimized to run natively on both ARM64 and S390X architectures. This means that organizations can now deploy and manage KVM virtual machines within Kubernetes clusters composed of these diverse hardware types, leveraging the full power and efficiency of each architecture.
2. Enabling Heterogeneous Kubernetes Clusters: The project facilitates the creation of Kubernetes clusters where different worker nodes can operate on different architectures. For instance, a control plane might run on x86, while worker nodes could be a mix of x86, ARM64, and S390X machines. This architectural flexibility unlocks new possibilities for workload placement, allowing specific VMs to run on the most suitable hardware, whether for performance, power efficiency, or specialized features.
3. Cross-Architecture VM Conversion and Templating: Perhaps the most impactful finding is the ability to import VMware virtual machines (which are typically x86-based) into an x86 KubeVirt node, and then, through a managed process, convert and template them for deployment as S390X or ARM64 virtual machines. This capability is explicitly highlighted as a feature not inherently offered by VMware itself, providing a unique and powerful bridge between traditional x86 virtualization and modern multi-arch cloud-native environments. It enables the reuse of existing VM assets across fundamentally different CPU architectures.
4. Transparent, Reproducible, and Declarative VM Management: The project delivers a robust management platform that makes the entire multi-arch VM lifecycle transparent and reproducible. By utilizing YAML files and Kubernetes’ declarative model, VM conversions, deployments, and configurations are codified, ensuring consistency and reducing manual intervention. This eliminates scenarios where "somebody off in a room somewhere doing a VM conversion" and instead integrates the process directly into the automated, GitOps-friendly workflows of Kubernetes.
5. Addressing VDDDK Limitations: By providing a pathway for cross-architecture VM conversion, the project effectively mitigates the limitation of the Virtual Disk Development Kit (VDDDK) being x86-only. While VDDDK may still be used for initial x86 imports, the subsequent conversion and deployment to other architectures are handled by the newly enabled multi-arch KubeVirt and CDI capabilities, bypassing the architectural restriction for the final target environment.

These findings collectively represent a significant step towards creating truly unified and flexible hybrid cloud environments, allowing enterprises to maximize their hardware investments and optimize workloads across the full spectrum of modern computing architectures.

Technical Deep Dive

▶ Watch: Defining Multi-Arch and heterogeneous Kubernetes clusters (2:50)

The technical foundation of this multi-architecture initiative rests on extending the capabilities of KubeVirt and Containerized Data Importer (CDI) to operate seamlessly across x86, ARM64, and S390X architectures. This involves addressing fundamental differences in instruction sets, data representation, and leveraging Kubernetes' inherent flexibility.

At its core, KubeVirt enables the execution of KVM virtual machines as pods within a Kubernetes cluster. This is achieved by packaging a lightweight virtual machine manager (VMM) and the VM's disk images within containers, allowing Kubernetes to manage their lifecycle alongside traditional containerized applications. Users define their VMs using YAML manifests, specifying CPU, memory, network, and storage configurations. KubeVirt then orchestrates the underlying KVM hypervisor on the Kubernetes worker nodes to provision and run these VMs.

CDI acts as the crucial storage orchestrator for KubeVirt VMs. It leverages persistent volume claims (PVCs) and introduces its own concept of data volumes to manage VM disk images. CDI's functionalities include:

• Import: Pulling VM disk images from various sources (HTTP, registry, S3, and crucially, VMware via VDDDK).
• Clone: Creating copies of existing data volumes.
• Upload: Allowing users to upload disk images directly.

The critical technical challenge addressed by this project was that the VMware import functionality, which depends on the Virtual Disk Development Kit (VDDDK), was exclusively available on x86. This meant that while KubeVirt could theoretically run on other architectures, the ability to easily ingest existing x86-based VM images from VMware for use on ARM64 or S390X was severely constrained.

The multi-arch enablement required a systematic approach to overcome architectural disparities:

1. Instruction Set Porting: x86 employs a Complex Instruction Set Computing (CISC) architecture, while ARM64 and S390X utilize Reduced Instruction Set Computing (RISC). This necessitates recompiling KubeVirt and CDI binaries and their dependencies for each target architecture. Docker and Kubernetes' support for multi-arch images (using manifest lists) is crucial here, allowing a single image reference to point to different architecture-specific image layers, ensuring the correct binary is pulled by the respective worker node.
2. Endianness Differences: A significant hurdle, explicitly mentioned by Fillekes, is the difference in endianness. x86 and ARM64 are Little Endian, meaning the least significant byte of a multi-byte data type is stored at the smallest memory address. S390X, however, is Big Endian, storing the most significant byte at the smallest address. This difference impacts how data (especially multi-byte values like integers or pointers) is represented and interpreted in memory. Porting efforts had to meticulously address these "reporting issues" to ensure data integrity and correct program execution across architectures. This often involves byte-swapping routines or careful data structure alignment in the underlying codebases.
3. Specialized S390X Features: Beyond basic functionality, the S390X architecture offers unique hardware features and optimizations. The project aimed not just for bare functionality but also to enable these "other wonderful features" specific to the mainframe. This implies deeper integration and optimization work to fully leverage the S390X platform's capabilities for specialized VM workloads, ensuring performance and stability akin to its native virtualization environments.
4. The Cross-Architecture VM Conversion Workflow: The most innovative technical aspect lies in the ability to convert x86-based VMware VMs to ARM64 or S390X. While the talk doesn't detail the exact mechanism, the conceptual flow is clear:

• An x86-based VMware VM disk image is first imported into an x86 KubeVirt node using CDI (potentially still leveraging VDDDK for the initial ingest).
• Once represented as a data volume within the x86 Kubernetes cluster, this VM image is then subjected to a conversion or re-templating process. This likely involves:
• Disk Image Conversion: Transforming the disk image format (e.g., VMDK to QCOW2) and potentially adjusting its internal structure to be compatible with the target architecture's virtualization stack.
• Guest OS Adaptation: The guest operating system within the VM image (e.g., Linux kernel) might require specific drivers or bootloader configurations to run on ARM64 or S390X. This could involve using tools like virt-customize or virt-sysprep to inject necessary components or modify configurations post-import.
• CPU Architecture Emulation/Translation (Less Likely for Production): While QEMU can emulate different architectures, the emphasis here is on native execution. The conversion process aims to create a natively bootable VM image for the target architecture, not an emulated one, for optimal performance.
• Finally, the converted VM image is then provisioned as a new KubeVirt VM on an ARM64 or S390X worker node, managed through standard Kubernetes YAML manifests.

This workflow essentially decouples the source architecture (x86 VMware) from the target architecture (ARM64/S390X KubeVirt), providing a flexible and declarative pipeline for VM portability. The use of Kubernetes as the central management plane ensures that these complex operations are transparent, reproducible, and can be automated as part of a larger infrastructure-as-code strategy. The mention of related projects like ISTIO (for service mesh capabilities) and Node Feature Discovery (for identifying node capabilities like architecture) further hints at the sophisticated ecosystem supporting this multi-arch vision, allowing for intelligent VM scheduling and networking in heterogeneous clusters.

Demo / Proof of Concept

▶ Watch: Benefits: Transparent, reproducible, declarative VM management (3:50)

While the lightning talk format, with its brief duration, did not allow for a live, in-depth demonstration of the multi-architecture capabilities, the speaker clearly articulated the conceptual proof of concept that underpins this project. The core demonstration, if performed, would showcase the seamless migration and conversion of virtual machines across disparate hardware architectures, a feat previously challenging or impossible without significant manual effort.

Such a demonstration would logically involve the following steps:

1. Initial Import on x86: An existing VMware virtual machine image (an x86 artifact) would be imported into an x86-based Kubernetes cluster running KubeVirt and CDI. This step would utilize CDI’s import functionality, potentially leveraging the Virtual Disk Development Kit (VDDDK), creating a data volume representing the VM’s disk on the x86 node.
2. Cross-Architecture Conversion: The imported x86 data volume would then be subjected to a conversion process. This would involve a declarative YAML definition that instructs KubeVirt/CDI to transform the VM image into a format compatible with, for example, the ARM64 or S390X architecture. This conversion would address instruction set and endianness differences, potentially involving re-packaging the guest operating system for the new architecture.
3. Deployment on Target Architecture: The newly converted ARM64 or S390X VM image would then be deployed as a KubeVirt virtual machine onto a worker node of the corresponding architecture within the same or a federated Kubernetes cluster. The demonstration would show this VM booting up and running natively on the ARM64 or S390X hardware, proving the success of the cross-architecture conversion.
4. Lifecycle Management: Further aspects could demonstrate standard KubeVirt operations—stopping, starting, or reconfiguring the converted VM using Kubernetes YAML files, highlighting the consistent management experience across all supported architectures.

The very existence of the ported KubeVirt and CDI components for ARM64 and S390X, as affirmed by the speaker, serves as the fundamental proof of concept. The "interesting possibilities" described, particularly the ability to "take them apart, put them back together, template them and convert them to S390X or ARM virtual machines," are the tangible results of this complex engineering effort. The emphasis on a "transparent," "reproducible," and "declarative" process managed via YAML files underscores that the proof of concept extends beyond mere technical feasibility to encompass operational efficiency and reliability in a multi-arch environment.

Defensive Implications

▶ Watch: Project contributors and invitation to open-source community (4:15)

The advent of multi-architecture KubeVirt and CDI introduces powerful new capabilities for managing virtual machines across heterogeneous hardware, but it also presents a new set of considerations for security defenders. As the attack surface expands to include ARM64 and S390X, security teams must adapt their strategies to ensure robust protection.

1. Expanded Attack Surface and Architecture-Specific Vulnerabilities: Running KVM virtual machines on ARM64 and S390X means that potential vulnerabilities specific to these architectures must be considered. While x86 has been thoroughly scrutinized for decades, ARM64 and S390X, while generally robust, may have unique hardware or firmware vulnerabilities, or less mature security tooling compared to x86. Defenders need to stay informed about security advisories for each architecture and ensure that hypervisors, guest operating systems, and KubeVirt/CDI components are patched promptly for all supported platforms.
2. Supply Chain Security for Multi-Arch Images: The project relies on multi-arch images for KubeVirt and CDI. This necessitates a stringent supply chain security strategy. Defenders must ensure that all architecture-specific image variants (x86, ARM64, S390X) are built from trusted sources, undergo comprehensive vulnerability scanning (e.g., using tools like Clair, Trivy, or Snyk), and are signed. Any compromise in one architecture's image could potentially affect the entire multi-arch deployment.
3. Secure VM Conversion and Migration: The ability to convert VMware virtual machines from x86 to ARM64 or S390X nodes introduces a critical security checkpoint. The conversion process itself must be secure, ensuring data integrity and confidentiality. Defenders need to verify that:

• The conversion tools are hardened and free from known vulnerabilities.
• Sensitive data within the VM disk images is not exposed during conversion.
• The converted VM images are scanned for malware or configuration weaknesses before deployment on the target architecture.
• Any temporary storage used during conversion is properly secured and purged.

1. Configuration Management and Policy Enforcement: The declarative nature of KubeVirt using YAML files is a double-edged sword. While it promotes consistency and allows for security baselines to be enforced via GitOps, misconfigurations can have widespread impact across multiple architectures. Security policies, such as network segmentation, resource quotas, and access controls, must be defined and applied uniformly or specifically tailored for each architecture as needed. Tools for policy enforcement, like OPA Gatekeeper, become even more critical in a multi-arch environment.
2. Robust Isolation and Hypervisor Security: The security of the KVM hypervisor and the container runtime (e.g., runC, Kata Containers) is paramount for ensuring strong isolation between VMs and the host, as well as between different VMs. Defenders must ensure that hypervisor hardening guides are followed for all architectures and that the underlying operating systems (like Red Hat CoreOS) are kept up-to-date. Architectural differences might introduce subtle variations in how isolation mechanisms function, requiring careful validation.
3. Monitoring and Logging in Heterogeneous Environments: Centralized logging and monitoring solutions are essential for detecting anomalies and security incidents. In a multi-arch cluster, these systems must be capable of ingesting logs and metrics from x86, ARM64, and S390X nodes, potentially from different hypervisor implementations or operating system variants. Correlation of events across these diverse platforms becomes crucial for identifying sophisticated attacks.
4. Access Control and Identity Management: With more diverse nodes and the ability to manage VMs across architectures, robust Role-Based Access Control (RBAC) within Kubernetes is more critical than ever. Granular permissions must be defined to control who can deploy, modify, or convert VMs on specific architectural nodes, preventing unauthorized access or privilege escalation.

By proactively addressing these defensive implications, organizations can fully leverage the benefits of multi-architecture KubeVirt and CDI while maintaining a strong security posture across their hybrid cloud infrastructure.

Key Takeaways

• Multi-Architecture Support: KubeVirt and Containerized Data Importer (CDI) now natively support x86, ARM64, and IBM mainframe (S390X) architectures, enabling broader deployment options for KVM virtual machines in Kubernetes.
• Heterogeneous Kubernetes Clusters: The project allows for Kubernetes clusters with mixed worker nodes (e.g., x86 control plane with ARM64 and S390X worker nodes), optimizing workload placement based on architectural strengths like power efficiency or specialized features.
• Cross-Architecture VM Conversion: A significant capability is the ability to import x86-based VMware virtual machines, convert them, and then deploy them as native ARM64 or S390X KubeVirt VMs, a feature not typically offered by VMware itself.
• Declarative and Reproducible Management: The entire multi-arch VM lifecycle, including conversion and deployment, is managed through declarative YAML files, ensuring transparency, reproducibility, and integration with GitOps workflows.
• Addressing VDDDK Limitations: The project bypasses the x86-only restriction of the Virtual Disk Development Kit (VDDDK) for VMware imports by providing a managed conversion path to other architectures.
• Open Source Collaboration: This initiative is an open-source project, benefiting from contributions from Red Hat, IBM (particularly the Burlan lab), Nvidia, ARM, and the broader community, encouraging future enhancements.

About the Speaker(s)

C. A. Fillekes is a WG Chair and a highly experienced Red Hat partner engineer at IBM Systems, where they have dedicated approximately five years. Fillekes has been deeply involved with the multi-architecture KubeVirt and CDI project for about two years, indicating a significant personal investment and expertise in this area. The speaker mentioned upcoming travels to the IBM Burlan lab, suggesting a close working relationship with the development teams there. Fillekes also emphasized the collaborative nature of this open-source endeavor, acknowledging contributions from Red Hat, the IBM Burlan lab, Ryan Hollley from Nvidia, and ARM, along with the wider open-source community.