Tutorial: Exploring Multi-Tenant Kubernetes APIs and Controllers With Kcp

Tutorial: Exploring Multi-Tenant Kubernetes APIs and Controllers With Kcp

Speakers: MJ, Staff Engineer, Castai & KCP Maintainer; Robert, Engineer, Clyo; Mark Mudrinich, Senior Software Engineer, Kubermatic & KCP Contributor; Navarun, Kubernetes Contributor & SIG Contrib Chair; Vasha, Software Engineer, Red Hat

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Fb_3dWJdY9I

Overview

This tutorial, presented at KubeCon EU, delves into KCP, an open-source, horizontally scalable Kubernetes-like control plane designed for building and managing custom APIs across multi-tenant and multi-cluster environments. The session, structured as a hands-on workshop, aimed to demystify KCP's core concepts by guiding attendees through practical exercises. Unlike typical conference talks that present high-level overviews, this tutorial focused on getting attendees' "hands dirty" with live setup and exploration of KCP's unique features, addressing previous feedback that while KCP looked promising on slides, its practical application was often unclear.

The talk highlighted KCP's role as an API gateway that extends existing Kubernetes multi-tenancy in a multiplexing way, enabling platform teams to build sophisticated Software-as-a-Service (SaaS) offerings. By demonstrating how to convert an existing PostgreSQL operator into a database-as-a-service, the speakers showcased KCP's power in abstracting underlying infrastructure complexities and providing isolated, self-service experiences for consumers. The tutorial underlined KCP's significance in the evolving landscape of cloud-native development, where managing diverse APIs and workloads across distributed environments is a growing challenge.

The speakers, a diverse group of engineers and KCP maintainers from various companies, emphasized the session's technical depth, positioning it as a deep dive into hardcore tech rather than a marketing pitch. Their collective expertise underscored KCP's broad industry relevance and its potential to revolutionize how developers interact with and extend Kubernetes capabilities beyond single-cluster limitations.

Background

▶ Watch: Workshop introduction, prerequisites and interactive format. (0:00)

The evolution of cloud-native applications has led to increasingly complex deployments, often spanning multiple Kubernetes clusters and requiring sophisticated multi-tenancy solutions. While Kubernetes provides robust orchestration capabilities, its inherent single-cluster design presents challenges when attempting to achieve strong isolation and API extensibility for multiple teams or customers. Each cloud provider offers its own control plane for managing Kubernetes, but an open-source, vendor-agnostic alternative capable of building custom APIs and lifecycle management has been a persistent need.

KCP emerges as a solution to these challenges. It is not Kubernetes itself, but rather a control plane that "loves Kubernetes," adopting its declarative API style and core principles while extending them to a multi-tenant, multi-cluster context. KCP introduces the concept of workspaces, which are essentially virtual Kubernetes clusters. These workspaces provide a unit of tenancy, offering users an isolated, stripped-down API server where they can interact with their own set of APIs and resources, much like having a personal Kubernetes cluster without the underlying infrastructure overhead.

Prior to KCP, managing multi-tenancy often involved complex solutions like namespace-based isolation (which offers weaker isolation guarantees) or deploying dedicated clusters per tenant (which is resource-intensive and operationally burdensome). KCP addresses these limitations by providing a horizontally scalable control plane that allows for a hierarchical organization of workspaces, enabling platform teams to define and share custom APIs across an organization while maintaining strict isolation between consumers. This foundation sets the stage for building advanced platform-as-a-service (PaaS) and SaaS offerings on top of Kubernetes, where users can consume specialized services without needing direct access to the underlying infrastructure or a deep understanding of its intricacies.

Key Findings

▶ Watch: High-level workshop plan and content warning. (3:28)

The tutorial highlighted several key findings and architectural contributions that define KCP's approach to multi-tenancy and API extensibility:

• Workspaces as Isolated Virtual Clusters: KCP's fundamental unit of tenancy, workspaces, provide strong isolation. Each workspace acts as a virtual Kubernetes cluster with its own set of APIs and resources, ensuring that objects created in one workspace are not visible in another. This isolation is crucial for multi-tenant environments, preventing accidental or malicious interference between different users or teams. Workspaces can also be organized hierarchically, allowing for complex organizational structures.
• API Exports and API Bindings for Controlled API Sharing: KCP introduces a novel mechanism for sharing APIs between workspaces: API Exports and API Bindings. A provider workspace can define and "export" a custom API (via an API Resource Schema), specifying the resources it offers and any permission claims required from consumers. Consumer workspaces can then create an API Binding to consume these exported APIs, explicitly accepting the permission claims. This mechanism allows API providers to curate and expose specific functionalities without requiring consumers to install CRDs directly into their workspaces, promoting controlled and secure API consumption.
• The API Sync Agent for Bridging KCP and Physical Clusters: To execute workloads defined by KCP's logical APIs on actual Kubernetes clusters, the API Sync Agent plays a critical role. This component acts as a bridge, synchronizing resources created in KCP workspaces down to designated "provider clusters" (physical Kubernetes clusters, e.g., a Kind cluster). It ensures that the abstract API objects in KCP are materialized as concrete resources in a Kubernetes cluster where they can be acted upon by standard Kubernetes controllers, effectively enabling KCP to manage workloads across multiple physical clusters.
• Immutable API Resource Schemas for API Evolution: KCP's API Resource Schemas are immutable once created. This design choice is a key finding for managing API evolution, ensuring stability and predictability for API consumers. Changes to an API require creating a new schema, which helps in versioning and managing breaking changes more explicitly.
• Multicluster Controller Runtime (Introduced): While not fully demonstrated due to time constraints, the tutorial introduced the concept of Multicluster Controller Runtime. This is a new component designed to enable controllers to be aware of and operate across multiple clusters managed by KCP, further enhancing KCP's capabilities for building distributed applications and services.

These findings collectively demonstrate KCP's comprehensive approach to delivering a scalable, secure, and flexible control plane for the next generation of cloud-native platforms, abstracting the complexities of multi-cluster and multi-tenant management.

Technical Deep Dive

▶ Watch: Deep dive into KCP Workspaces and hierarchy. (6:00)

KCP's architecture is built on a foundation of familiar Kubernetes concepts, re-imagined for multi-tenancy and API extensibility. At its core, KCP runs as a single binary process, functioning as a Kubernetes-like control plane that combines the functionalities of an API server and a controller manager. It strips out many Kubernetes-specific components, focusing on API management and resource orchestration across logical boundaries.

The primary interaction point for users with KCP is through the kubectl kcp plugin, which extends standard kubectl commands to manage KCP-specific resources. This includes operations like creating and navigating workspaces. Each workspace is a logical cluster, accessible via a unique URL, providing an isolated environment. When a user runs kubectl get apiresources within a workspace, they see only the APIs available to that specific workspace, demonstrating KCP's strong isolation model. Objects created within one workspace (e.g., ConfigMaps) are entirely invisible from another, even if they share the same parent in the hierarchical workspace structure.

Custom APIs in KCP are defined using API Resource Schemas. These schemas are analogous to Kubernetes Custom Resource Definitions (CRDs) but with a crucial distinction: they are immutable once created. This immutability ensures API stability and facilitates controlled evolution by requiring new schemas for significant changes. An API Resource Schema specifies the API group, version, kind, and structural schema of the custom resource, along with its scope (namespace-scoped or cluster-scoped).

The sharing of these custom APIs is facilitated by API Exports and API Bindings:

• An API Export is created by a provider workspace to publish a specific API Resource Schema. Crucially, an API Export includes permission claims, which are declarative statements about the permissions the provider expects from any consumer workspace that binds to its API. For example, a database-as-a-service provider might claim permissions to create Secrets or Namespaces in the consumer's workspace to manage database credentials or dedicated environments. The talk used an analogy of mobile app permissions (e.g., WhatsApp asking for storage access) to illustrate this concept. Each API Export also exposes a unique virtual workspace URL, which the provider can use to observe all resources of that exported API created by its various consumers.
• An API Binding is created by a consumer workspace to subscribe to an API Export. When creating a binding, the consumer explicitly accepts the permission claims made by the provider. This action makes the exported API available within the consumer's workspace without installing a CRD directly into its etcd. Instead, KCP dynamically routes API requests for these bound resources to the appropriate provider's API server, providing a seamless user experience.

To bridge the logical APIs in KCP with actual workloads on physical Kubernetes clusters, the API Sync Agent is deployed. This controller is configured with a kubeconfig that grants it access to both the KCP provider workspace and the target "provider cluster" (a standard Kubernetes cluster, such as a Kind cluster). The API Sync Agent monitors Published Resources CRDs, which specify:

1. The KCP API Export it's responsible for.
2. The corresponding Kubernetes API (e.g., postgresql.cnpg.io/v1/Cluster) in the provider cluster.
3. Synchronization rules and related resources (e.g., Secrets) that need to be synced between KCP consumer workspaces and the provider cluster.

When a consumer creates an instance of a bound API resource in their KCP workspace (e.g., a Cluster object for a database), the API Sync Agent intercepts this, translates it, and creates the corresponding Kubernetes resource in the designated provider cluster. It also synchronizes related objects, such as database credentials stored in Secrets, from the consumer's workspace down to the provider cluster. This enables platform teams to manage a fleet of services (like databases) from a central KCP instance, while individual teams consume these services through isolated, KCP-managed workspaces, without direct exposure to the underlying physical infrastructure. The use of a Multicluster Controller Runtime was also mentioned as a more advanced way to build custom controllers that are natively aware of KCP's multi-cluster capabilities, offering more fine-grained control than the opinionated API Sync Agent.

Demo / Proof of Concept

▶ Watch: Starting hands-on part one: environment setup. (7:13)

The tutorial featured a comprehensive hands-on demonstration structured into three main parts, each building upon the previous one to illustrate KCP's capabilities:

Part 1: Basic Workspace Exploration

The initial part focused on setting up the environment using GitHub Code Spaces for consistency, installing prerequisites like git, KCP, API Sync Agent, Kind, kubectl, and Crew (a kubectl plugin manager). Attendees then started the KCP binary locally, which spun up the KCP control plane.

The core of this part involved demonstrating workspaces. Users created multiple workspaces (e.g., workspace1, workspace2, potato) and navigated through them using kubectl kcp workspace use. The key takeaway was the strong isolation: a ConfigMap created in workspace1 was confirmed to be invisible when querying ConfigMaps in workspace2, even if both were children of the same root workspace. This visually reinforced KCP's tenancy model, where each workspace behaves like an independent virtual Kubernetes cluster. The hierarchical nature of workspaces was also shown by creating nested workspaces (e.g., root/providers/cowboy).

Part 2: API Sharing with the "Cowboy" API

This section introduced API Exports and API Bindings.

1. Provider Setup: A "provider" workspace (root/providers/cowboy) was created. Within this workspace, a custom API called Cowboy was defined using an API Resource Schema. This schema specified the Cowboy resource's structure, including its namespace scope.
2. API Export: The Cowboy API was then exported using an API Export. This export included a crucial permission claim: it required any consumer to grant access to ConfigMaps in their workspace. This illustrated how providers can declare necessary permissions for their services. The API Export also exposed a virtual workspace URL for the provider to monitor consumer resources.
3. Consumer Binding and Usage: A "consumer" workspace (root/consumers/wild-west) was created. The consumer then used kubectl kcp bind to create an API Binding to the Cowboy API Export from the provider workspace, explicitly accepting the ConfigMap permission claim. After binding, the Cowboy API became available in the consumer's workspace. The consumer proceeded to create a Cowboy resource (e.g., buckaroo-bill). A key demonstration was showing that while kubectl get cowboy worked, kubectl get crd did not show a CRD for Cowboy in the consumer's workspace, emphasizing that the API is virtualized and routed by KCP, not locally installed.
4. Provider Observation: Finally, the provider used its virtual workspace URL to list Cowboy resources, successfully observing the buckaroo-bill object created by the consumer in the wild-west workspace. This closed the loop, demonstrating how a provider can manage and gain visibility into the usage of its exported APIs across multiple consumers.

Part 3: Dynamic Providers - Database-as-a-Service

The most complex part showcased KCP's ability to provision services on actual Kubernetes clusters.

1. Service Owner (Physical Cluster) Setup: A Kind cluster was created, acting as the "service owner" or physical Kubernetes cluster where actual workloads would run. The Cloud Native PG operator was deployed onto this Kind cluster to manage PostgreSQL databases.
2. KCP Service Provider Setup: A KCP provider workspace (root/providers/database) was established, and an empty API Export for PostgreSQL clusters was created. A kubeconfig was prepared for the API Sync Agent to allow it to communicate with both KCP and the Kind cluster.
3. API Sync Agent Deployment: The PublishedResources CRD, provided by the API Sync Agent, was deployed to the Kind cluster. This CRD specified that the Cluster resource from the postgresql.cnpg.io API group (provided by the Cloud Native PG operator) should be published via the KCP API Export. It also defined synchronization rules for related resources, specifically Secrets, to carry database credentials. The API Sync Agent binary was then started, connecting the KCP provider workspace to the Kind cluster.
4. KCP Service Consumer Usage: A KCP consumer workspace (root/consumers/pg) was created. This consumer workspace then created an API Binding to the PostgreSQL API Export, accepting permission claims for Secrets and Namespaces. The consumer proceeded to create a Cluster resource (representing a PostgreSQL database) in their KCP workspace.
5. Verification: The API Sync Agent observed the Cluster resource in the KCP consumer workspace, translated it, and created a corresponding PostgreSQL Cluster resource in the Kind cluster. kubectl get cluster run against the Kind cluster confirmed that the database instance was being provisioned. The demonstration also showed that a Secret containing credentials was automatically synced from the KCP consumer workspace to the Kind cluster's namespace dedicated to that database.

Due to time constraints and some live debugging, the final steps of directly connecting to the provisioned database were cut short, and the "Multicluster Controller Runtime" exercise was deferred for self-study. However, the core workflow of KCP abstracting a physical service for multi-tenant consumption was clearly demonstrated.

Defensive Implications

▶ Watch: Step-by-step guide to GitHub Codespaces setup. (9:55)

KCP introduces several significant defensive implications that enhance security and operational efficiency in multi-tenant and multi-cluster Kubernetes environments:

1. Strong Workload and API Isolation: The fundamental concept of workspaces provides a robust isolation boundary. Each workspace acts as a virtual Kubernetes cluster, ensuring that API objects and data created by one tenant are strictly isolated from others. This prevents cross-tenant data leakage or accidental interference, a common security challenge in shared environments. Platform teams can confidently onboard multiple users or applications without fear of their workloads impacting or observing each other.

1. Controlled API Exposure via API Exports and Bindings: KCP's API Exports and API Bindings mechanism enables platform operators to expose curated sets of APIs to consumers. Instead of granting full Kubernetes API access or requiring direct CRD installations, consumers bind to specific, well-defined APIs. This limits the attack surface by ensuring tenants only interact with the services they need, reducing the blast radius of potential misconfigurations or vulnerabilities.

1. Explicit Permission Claims for Transparency: The inclusion of permission claims within API Exports is a powerful security feature. Consumers are explicitly informed about the permissions (e.g., access to Secrets or Namespaces) that a provider service requires in their workspace. This transparency allows consumers to make informed decisions about binding to an API, akin to granting permissions to a mobile application. It enforces a clear contract between provider and consumer, enhancing trust and auditability.

1. Abstraction of Underlying Infrastructure: KCP abstracts away the complexities of the underlying physical Kubernetes clusters. Consumers interact only with their KCP workspace, never directly with the "provider clusters" where the actual workloads run. This significantly reduces the need to grant direct kubeconfig access to tenants, minimizing the risk of unauthorized access or manipulation of shared infrastructure. The API Sync Agent acts as a secure intermediary, translating KCP API calls into physical Kubernetes operations.

1. Centralized API Governance: By centralizing API definition and sharing within KCP, platform teams gain better governance over the entire API landscape. They can enforce standards, manage API versions, and revoke access more effectively. The immutability of API Resource Schemas further contributes to stable API contracts, reducing the likelihood of unexpected behavior or security regressions due to unmanaged API changes.

1. Reduced Operational Burden for Multi-Tenancy: From a defensive operations perspective, KCP simplifies multi-tenancy management. Instead of deploying and securing separate Kubernetes clusters for each tenant, which can be resource-intensive and complex, KCP allows for logical separation on a shared, horizontally scalable control plane. This consolidates security monitoring and management efforts, making it easier to maintain a secure posture across a large number of tenants.

To leverage these defensive benefits fully, it is crucial to secure the KCP control plane itself, properly configure role-based access control (RBAC) within KCP workspaces, and ensure that API Sync Agents operate with the principle of least privilege when interacting with both KCP and the target Kubernetes clusters.

Key Takeaways

• KCP is a Multi-Tenant, Kubernetes-like Control Plane: It extends Kubernetes API concepts to enable horizontally scalable, open-source control planes for managing custom APIs across multiple tenants and clusters, without being Kubernetes itself.
• Workspaces Provide Strong Isolation and Hierarchy: KCP workspaces act as isolated, virtual Kubernetes clusters, offering a unit of tenancy where resources are invisible across boundaries, and can be organized into a hierarchical structure.
• API Exports and Bindings Enable Controlled API Sharing: API Exports allow providers to publish custom APIs with explicit permission claims, while API Bindings enable consumers to subscribe to these APIs, accepting the required permissions, without direct CRD installation.
• API Sync Agent Bridges KCP to Physical Workloads: The API Sync Agent is crucial for connecting KCP's logical APIs with actual Kubernetes clusters, synchronizing resources and related objects (like Secrets) to execute workloads defined in KCP workspaces on underlying physical infrastructure.
• Enables SaaS-like Platform Building: KCP facilitates the creation of sophisticated Software-as-a-Service (SaaS) and Platform-as-a-Service (PaaS) offerings by abstracting complex infrastructure, allowing platform teams to expose curated services (e.g., Database-as-a-Service) to consumers in an isolated and self-service manner.
• Enhances Security and Governance: KCP's design, including strong workspace isolation, explicit permission claims, and centralized API management, significantly improves the security posture and governance capabilities for multi-tenant cloud-native environments.

About the Speaker(s)

The tutorial was presented by a team of experienced engineers and KCP contributors:

• MJ is a Staff Engineer at Castai and a dedicated KCP maintainer, having been involved with the project for the last four to five years.
• Robert works for Clyo, a company focused on software-defined storage and Kubernetes, and has recently started investing time into the KCP project.
• Mark Mudrinich is a Senior Software Engineer at Kubermatic and a recent contributor to KCP.
• Navarun has been a significant contributor to Kubernetes for six years, maintaining several areas in the project and serving as a chair for SIG Contrib. He also contributes to KCP and builds products around it.
• Vasha is a Software Engineer at Red Hat and was part of the original KCP team when the project was first introduced and designed.