Tutorial: Mind Your Pod's Business: Netwo... Surya Seetharaman, Miguel Duarte Barroso & Keith Burdis

Tutorial: Mind Your Pod's Business: Network Segmentation with OVN Kubernetes & KubeVirt

Speakers: Surya Seetharaman, Miguel Duarte Barroso, Keith Burdis

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=bFKls7IvzNE

Overview

In modern cloud-native environments, robust network segmentation is paramount for security, compliance, and multi-tenancy. The KubeCon EU tutorial "Mind Your Pod's Business: Network Segmentation with OVN Kubernetes & KubeVirt" by Surya Seetharaman, Miguel Duarte Barroso, and Keith Burdis tackles this critical challenge head-on. The talk introduces and demonstrates the concept of User Defined Networks (UDNs) in Kubernetes, an advanced networking capability built on OVN Kubernetes that provides strong, default-deny isolation for both containerized workloads (pods) and virtual machines (VMs) managed by KubeVirt.

The speakers elucidate how UDNs empower administrators to create truly isolated network segments within a single Kubernetes cluster, addressing limitations of traditional NetworkPolicies and providing a more secure foundation for diverse workloads. Through live demonstrations involving cross-network communication attempts between pods and a compelling live migration of a KubeVirt VM, the tutorial vividly illustrates the practical benefits and underlying mechanics of UDNs. This approach significantly enhances the security posture of Kubernetes clusters, making it a vital tool for organizations operating in regulated industries or managing complex multi-tenant platforms.

Background

▶ Watch: Overlapping IP ranges with default network assignment (0:40)

Kubernetes, by default, provides a relatively flat network model where pods within a cluster can typically communicate with each other. While Kubernetes NetworkPolicies offer a mechanism to control traffic flow between pods and namespaces, their implementation can be complex, often requiring meticulous configuration to achieve true isolation. In multi-tenant environments, or those with strict compliance requirements (e.g., PCI-DSS, HIPAA), relying solely on NetworkPolicies can lead to security gaps, configuration drift, and operational overhead. The default allow-all nature of Kubernetes networking means that NetworkPolicies function as an allow-list on a fundamentally open network, requiring explicit rules to restrict communication.

The problem intensifies when dealing with overlapping IP addresses across namespaces, a common scenario if users are allowed to define their own network CIDRs without proper cluster-wide coordination. For instance, if multiple namespaces are assigned the default 10.244.0.0/16 CIDR, pods in different namespaces could end up with the same IP addresses, leading to unexpected connectivity or routing issues if they ever need to interact. Furthermore, the integration of virtual machines into Kubernetes environments via projects like KubeVirt introduces additional networking complexities, particularly concerning live migration and persistent IP addresses, which are fundamental expectations for traditional VM workloads. The need for a more opinionated, default-deny approach to network segmentation that is both robust and easily manageable across heterogeneous workloads (pods and VMs) drove the development and adoption of UDNs in OVN Kubernetes.

Key Findings

▶ Watch: Network policies with user-defined network segmentation (4:00)

The tutorial highlights several key findings and capabilities of User Defined Networks (UDNs):

• Default-Deny Network Segmentation: UDNs establish a strong, default-deny posture for network communication between different network segments. Pods or VMs assigned to one UDN are completely isolated from those in another UDN by default, preventing any cross-network IP connectivity unless explicitly configured otherwise. For example, a pod in the "blue" network (CIDR 10.3.103.0/24) cannot communicate with a pod in the "green" network (CIDR 203.203.203.0/24).
• Scoped NetworkPolicies: While UDNs provide a foundational layer of isolation, Kubernetes NetworkPolicies remain effective. However, they are strictly scoped to their respective UDNs. An allow rule in a NetworkPolicy within the "blue" network will only apply to pods within that "blue" network. Crucially, NetworkPolicies do not override the cross-UDN segmentation; the network segmentation always takes precedence (as confirmed by audience polling, with the majority agreeing).
• Service Isolation: The isolation provided by UDNs extends to Kubernetes Services. Even though all UDNs within a cluster share the same cluster-wide Service CIDR (e.g., 10.96.0.0/12), a pod in one UDN cannot access a ClusterIP service in a different UDN. This ensures end-to-end network isolation for applications.
• Dual Network Interfaces for Pods: Pods running in a UDN environment are equipped with two logical network interfaces:
• eth0 (Default Kubernetes Network): This interface, often using the 10.244.x.x range, is primarily used for kubelet probes (health checks), access to the Kubernetes API server, and DNS services running in the infrastructure network. All other traffic on this interface is blocked.
• UDN Interface (e.g., udn0, udn1): This is the primary interface for all application-specific traffic within the UDN. Its IP address (e.g., 10.3.103.2.5 for a blue pod) is the one applications use for communication.
• VM IP Persistence During Live Migration: For KubeVirt VMs, UDNs enable a critical feature: maintaining the same IP and MAC address during a live migration between nodes. This is achieved by tying the IP address's lifecycle to the VM's lifecycle, not the underlying pod's, through a dedicated IPAM claim CRD. This ensures that established TCP connections are not broken during migration, a fundamental requirement for enterprise virtualization workloads.
• Roadmap for Interconnection: While default isolation is key, the speakers acknowledge the need for controlled communication between UDNs. The roadmap includes features like interconnecting UDNs via a Custom Resource Definition (CRD) to allow explicit exposure of services or partial connectivity, as well as integration with BGP for advertising pod IPs and potentially eliminating overlay tunnels for East-West traffic using EVPN and VRF-lite for advanced telco use cases.

Technical Deep Dive

▶ Watch: IP connectivity between workloads in different UDNs (roadmap) (5:00)

The core of the network segmentation presented in this tutorial revolves around User Defined Networks (UDNs), implemented using OVN Kubernetes as the Container Network Interface (CNI). OVN (Open Virtual Network) is a network virtualization solution that enables flexible, programmable network services for cloud-native environments. When integrated with Kubernetes, it provides advanced capabilities beyond standard NetworkPolicies.

User Defined Networks (UDNs)

UDNs are custom network segments defined within a Kubernetes cluster. They are managed through Custom Resource Definitions (CRDs), allowing administrators to declare network configurations in a Kubernetes-native way. Each UDN is typically associated with a specific namespace or a group of namespaces, providing logical isolation. When a UDN is defined for a namespace, all pods and KubeVirt VMs subsequently deployed in that namespace will automatically be attached to the UDN's network, receiving an IP from its allocated CIDR range.

A crucial aspect of UDNs is the concept of primary and secondary networks. A pod or VM can be connected to multiple networks (e.g., one primary UDN and several secondary UDNs for specific management or signaling traffic). However, only one network can be designated as primary, through which all default outbound traffic will flow. Secondary networks provide additional interfaces for specialized communication, each with its own routes and connectivity.

Pod Networking with UDNs

When a pod is launched within a UDN-enabled namespace, it receives a dual-interface configuration:

1. eth0 (Default Kubernetes Network): This interface is a remnant of the cluster's default CNI setup. It typically uses an IP from the cluster's default pod CIDR (e.g., 10.244.1.9). The kubelet still requires this interface for performing health probes and for the pod to reach the Kubernetes API server and DNS services (which often reside in an infrastructure network). Critically, all application traffic on this interface is blocked, ensuring that the pod's actual workload communication occurs over its designated UDN interface. This design acknowledges the current limitations of core Kubernetes regarding multi-network awareness for fundamental services like probes and DNS.
2. UDN Interface (e.g., udn0, udn1): This is the interface provisioned by OVN Kubernetes for the specific UDN. It receives an IP address from the UDN's allocated CIDR (e.g., 10.3.103.2.5 for a pod in the "blue" UDN). All application data traffic, including communication with Kubernetes Services (which might use a 10.96.x.x ClusterIP), is routed through this UDN interface. The routes inside the pod are configured to direct UDN-specific traffic through this interface, while the cluster service CIDR (10.96.x.x) is also reachable via the UDN interface.

This dual-interface approach allows UDNs to provide strong application-level isolation while maintaining compatibility with essential Kubernetes control plane functions.

Service Networking and Isolation

Kubernetes Services, specifically ClusterIP services, are also subject to UDN isolation. While the talk clarifies that all UDNs in a cluster share the same cluster-wide Service CIDR (e.g., 10.96.0.0/12), the isolation mechanisms ensure that a pod in one UDN cannot access a ClusterIP service in another UDN. For instance, a "blue" pod attempting to reach a "green" service via its ClusterIP will fail, reinforcing the segmentation. This is achieved by leveraging OVN's capabilities to filter traffic at the network layer, even for shared service IPs. This design decision simplifies the integration with core Kubernetes service controllers while still enforcing the desired isolation.

KubeVirt and VM Networking

The integration of UDNs with KubeVirt is a significant highlight, addressing common challenges in virtual machine networking within Kubernetes. KubeVirt allows running traditional VMs alongside containers, and UDNs extend the same robust network segmentation to these workloads.

For KubeVirt VMs, the networking setup differs slightly: the VM itself typically receives only one network interface, directly connected to the UDN. The default Kubernetes eth0 interface that pods typically have is effectively "discarded" from the VM's perspective, and the UDN attachment is directly "plumbed" into the VM.

A critical feature for VMs is IP persistence during live migration. When a VM migrates from one node to another, it is effectively restarted within a new pod on the destination node. In a standard Kubernetes IPAM scenario, this new pod would typically receive a new IP address, breaking any established TCP connections. To prevent this, UDNs for KubeVirt leverage a dedicated IPAM claim CRD. This CRD persists the VM's IP address and MAC address independently of the underlying pod's lifecycle. When a VM is created, an IPAM claim CR is provisioned. OVN Kubernetes assigns an IP to the VM's network interface and updates this CR with the assigned IP. During a live migration, the new pod on the destination node retrieves the same IP and MAC address from the IPAM claim, ensuring that the VM's network identity remains constant and established connections survive the migration. This is particularly relevant for UDNs configured with a Layer 2 topology, which act like a "big switch" across the cluster, allowing VMs to maintain their network identity regardless of their physical node.

Overlapping CIDRs

The speakers address the scenario of overlapping CIDRs. If users don't specify a CIDR for a UDN, a default CIDR (e.g., 10.244.0.0/16) might be assigned. If multiple UDNs default to the same CIDR, this results in overlapping IP spaces. While this is acceptable as long as workloads in these UDNs never need to communicate, it highlights the importance of proper CIDR planning. The strong default-deny segmentation of UDNs ensures that even with overlapping CIDRs, cross-network communication is inherently blocked, preventing accidental exposure or routing conflicts.

Future Roadmap

The roadmap for UDNs includes several advanced features:

• Interconnecting UDNs: To allow controlled communication between isolated networks, a CRD is planned that would explicitly request exposure of services or define routing between UDNs. This would move beyond strict isolation to configurable, secure inter-network communication.
• BGP Integration: Advertising pod IPs using BGP (Border Gateway Protocol) would allow external networks to directly route to UDN pods, potentially eliminating overlay tunnels for East-West traffic.
• EVPN (Ethernet VPN) and VRF-lite: These technologies are being considered to extend segmentation for traditional telco networking virtualization use cases, providing even more sophisticated network slicing and isolation capabilities.

Demo / Proof of Concept

▶ Watch: Interconnecting UDNs: virtual router functionality (roadmap) (7:00)

The tutorial included comprehensive live demonstrations showcasing the capabilities of User Defined Networks for both pods and KubeVirt virtual machines.

Pod Network Segmentation Demo (Surya Seetharaman)

1. Network Setup: The demonstration began with a Kubernetes cluster configured with OVN Kubernetes. Several UDNs were created, including a "blue" network (CIDR 10.3.103.0/24), a "green" network (CIDR 203.203.203.0/24), and a "colored enterprise" network (CIDR 192.168.0.0/16) which included "red" and "yellow" segments that could communicate within themselves.
2. Pod Creation: Pods were deployed into these respective networks. For example, app-blue-0 and app-blue-1 in the "blue" namespace/network, and similar pods in the "green" network.
3. Interface Inspection: Inside a "blue" pod, the speaker showed ip a output, confirming three interfaces:

• lo (localhost)
• eth0: Assigned an IP from the default Kubernetes range (10.244.1.9). This IP, as explained, is only for kubelet probes, DNS, and Kubernetes API server access.
• udn0: The actual UDN interface, with the pod's application IP (10.3.103.2.5). All default traffic flows through this interface.

The ip route output further confirmed that the cluster service CIDR (10.96.0.0/12) was also routed via the udn0 interface.

1. Intra-Network Communication: A curl command from app-blue-0 to app-blue-1 (using their UDN IPs, e.g., 10.3.103.1.5) successfully demonstrated communication within the "blue" network. Attempts to ping the 10.244.x.x (eth0) IP of another pod failed, validating its restricted use. Similarly, green pods could communicate with other green pods, and red pods could communicate with yellow pods (as they were part of the same 192.168.0.0/16 UDN).
2. Cross-Network Communication: A crucial test involved curl from a "blue" pod to a "green" pod's UDN IP (203.203.203.2.5). This attempt failed, explicitly demonstrating the default-deny isolation between different UDNs.
3. Service Isolation: Using an interactive poll, the audience was asked if a blue pod could talk to a green service (ClusterIP). The speaker confirmed that, by design, cross-UDN service communication is also blocked, reinforcing the comprehensive isolation.
4. Network Policy Precedence: Another poll clarified that NetworkPolicies are network-scoped and apply within a UDN. They do not override the fundamental cross-UDN segmentation, meaning network segmentation takes precedence over NetworkPolicy allow rules for cross-network traffic.

KubeVirt VM Live Migration Demo (Miguel Duarte Barroso)

1. VM Setup: Two KubeVirt VMs were created, one in the "blue" namespace and one in the "red" namespace, both attached to a UDN with a 192.168.0.0/16 CIDR and a Layer 2 topology.
2. VM IP Assignment: After the VMs were running, virtctl console was used to log into each VM. The blue VM had IP 192.168.0.6 and the red VM had 192.168.0.5. It was highlighted that VMs typically only have one network interface, directly plumbed to the UDN, unlike pods which retain eth0.
3. Egress and East-West Connectivity:

• From within one VM, a curl to the internet (www.google.com) successfully demonstrated egress connectivity.
• A ping between the blue VM (192.168.0.6) and the red VM (192.168.0.5) succeeded, proving East-West connectivity within the same UDN.

1. Live Migration with Persistent IP/MAC:

• An iperf3 server was started on the blue VM (port 9000).
• An iperf3 client was started on the red VM, connecting to the blue VM's IP, establishing a continuous TCP connection.
• The virtctl migrate command was executed to migrate the blue VM (blue/blue) to a different node.
• During the migration, the iperf3 client briefly reported packet loss (a "hiccup") but the connection survived, and the iperf3 session continued.
• An interactive poll before the demo revealed that many expected the VM's IP to change during migration, but the demo proved it stays the same. This persistence is critical for maintaining established connections and is achieved through the IPAM claim CRD, which ties the IP/MAC lifecycle to the VM, not the pod. The MAC address also remains the same during migration.

The demos effectively validated that UDNs provide strong isolation for both pods and VMs, and crucially, enable enterprise-grade features like live migration with IP persistence for KubeVirt workloads.

Defensive Implications

▶ Watch: Understanding pod IP discrepancies with multiple networks (7:59)

The implementation of User Defined Networks (UDNs) with OVN Kubernetes and KubeVirt offers significant defensive advantages for securing cloud-native environments, particularly in multi-tenant or highly regulated contexts:

1. Strong Default-Deny Security Posture: UDNs fundamentally shift the networking paradigm from an allow-all default to a deny-all default between network segments. This means that unless explicitly allowed via UDN interconnection mechanisms (which are on the roadmap), pods and VMs in different UDNs cannot communicate. This "zero-trust" approach significantly reduces the attack surface by preventing lateral movement between isolated environments by default.
2. Enhanced Compliance: For industries with stringent compliance requirements (e.g., PCI-DSS, HIPAA, GDPR), robust network segmentation is often a mandatory control. UDNs provide a clear, auditable mechanism to enforce complete network isolation between different applications, tenants, or data classifications within a single Kubernetes cluster. This simplifies the process of demonstrating compliance and reduces the risk of data breaches.
3. Simplified Network Security Management: While Kubernetes NetworkPolicies are powerful, their granular nature can lead to complex, error-prone configurations, especially in large clusters with many namespaces. UDNs provide a higher-level abstraction for segmentation. By creating distinct network zones, administrators can rely on the inherent isolation of UDNs, reducing the number and complexity of NetworkPolicies needed, as NetworkPolicies then only need to govern traffic within a UDN.
4. Mitigation of Overlapping IP Issues: The UDN architecture, with its default isolation, gracefully handles scenarios where different UDNs might accidentally or intentionally use overlapping IP CIDRs. While not ideal for inter-network communication, the strict segmentation prevents routing conflicts or unintended connectivity, offering a layer of resilience against misconfigurations.
5. Secure KubeVirt Workloads: For environments integrating virtual machines via KubeVirt, UDNs ensure that traditional VM workloads benefit from the same strong network isolation as pods. The ability to maintain persistent IP and MAC addresses during live migration is crucial for operational stability and security, preventing connection resets that could be exploited or lead to service disruptions.
6. Clear Separation of Infrastructure and Application Traffic: The design choice to keep the default eth0 interface for kubelet probes and API access, while routing all application traffic through the UDN interface, provides a clear separation of concerns. This means that even if an attacker compromises an application pod, their ability to leverage the eth0 interface for broader network access is severely limited, as only specific control plane traffic is permitted.
7. Future-Proofing with Advanced Networking: The roadmap for UDNs, including BGP integration, EVPN, and VRF-lite, indicates a commitment to supporting highly sophisticated and secure network architectures. This allows organizations to evolve their security posture to meet future demands, such as extending UDN isolation to external networks or integrating with telco-grade virtualization.

In summary, UDNs provide a powerful, architectural solution for network segmentation in Kubernetes, moving beyond reactive NetworkPolicies to a proactive, default-deny security model that is easier to manage and more robust against common attack vectors.

Key Takeaways

• Robust Default-Deny Segmentation: User Defined Networks (UDNs) provide complete network isolation between different segments for pods and KubeVirt VMs by default, enhancing security and compliance.
• Dual Network Interfaces for Pods: Pods in UDNs have a restricted eth0 for kubelet probes and API access, while all application traffic flows through the dedicated UDN interface (e.g., udn0).
• Comprehensive Isolation for Services: Kubernetes ClusterIP services are also isolated by UDNs; a pod cannot access a service in a different UDN, even with a shared cluster-wide Service CIDR.
• Network Policies are UDN-Scoped: Kubernetes NetworkPolicies apply within a UDN and do not override the fundamental cross-UDN segmentation, which takes precedence.
• VM IP Persistence during Live Migration: KubeVirt VMs using UDNs maintain their IP and MAC addresses during live migration across nodes, ensuring established TCP connections are preserved via an IPAM claim CRD.
• Evolving Capabilities: The UDN roadmap includes features like explicit UDN interconnection, BGP integration for advertising pod IPs, and advanced EVPN/VRF-lite support for complex networking scenarios.

About the Speaker(s)

The tutorial was presented by Surya Seetharaman, Miguel Duarte Barroso, and Keith Burdis. Based on the technical content and the projects discussed (OVN Kubernetes, KubeVirt), it can be inferred that they are experts and contributors in the fields of Kubernetes networking, virtual machine management within Kubernetes, and open-source network virtualization technologies like OVN. Their roles likely involve developing, implementing, and advocating for advanced networking solutions in cloud-native environments, particularly focusing on robust segmentation and the integration of diverse workloads.