Crossplane Intro and Deep Dive - The Cloud Native Control Plane Framework - Jared Watts & Nic Cope

Crossplane Intro and Deep Dive - The Cloud Native Control Plane Framework - Jared Watts & Nic Cope

Speakers: Jared Watts, Leadership, Crossplane Project; Nic Cope, Leadership, Crossplane Project

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=6MrXcbcxnN4

Overview

This talk provides a comprehensive overview and deep dive into Crossplane, an open-source cloud-native control plane framework. Presented by Jared Watts and Nic Cope, key leaders in the Crossplane project, the session explores how Crossplane extends Kubernetes to manage and compose external infrastructure, SaaS offerings, and on-premises resources. The speakers highlight Crossplane's role in enabling platform teams to build their own internal developer platforms, offering self-service capabilities and abstracting away underlying infrastructure complexity.

The presentation is structured to cater to both new users and seasoned Crossplane professionals. Watts begins with an introduction to Crossplane's core concepts, including managed resources and composition functions, while Cope delves into the exciting advancements and future direction with the Crossplane v2 preview. This talk is crucial for anyone looking to leverage Kubernetes as a universal control plane, streamline infrastructure provisioning, or build robust internal developer platforms that span diverse environments and services.

The significance of Crossplane lies in its ability to unify disparate cloud and on-premises resources under a single, declarative Kubernetes API. By transforming external services into Kubernetes objects, Crossplane empowers developers to provision and manage their required resources using familiar Kubernetes tooling and workflows. The introduction of Crossplane v2, with its focus on namespaced resources and broader composition capabilities, marks a pivotal evolution, aiming to make Crossplane even more versatile and intuitive for managing not just infrastructure, but also applications and microservices.

Background

▶ Watch: Talk Introduction and Overview of Crossplane (0:00)

The concept of a control plane is not new; cloud providers have long operated sophisticated control planes to manage their vast array of services. Similarly, Kubernetes itself serves as an incredibly effective control plane for orchestrating containers. However, the challenge arises when attempting to manage the myriad of external resources—such as cloud databases, message queues, SaaS subscriptions, or on-premises virtual machines—using a consistent, declarative approach from a single interface. This is precisely the problem Crossplane aims to solve.

Crossplane extends the Kubernetes API and control plane, teaching it how to manage "everything else" beyond just containers. In its initial iterations (Crossplane v1), the project introduced the fundamental building blocks: managed resources and compositions. Managed resources represent real-world external services as Kubernetes API objects, allowing users to declare their desired state. Compositions then enable platform engineers to combine these granular managed resources into higher-level abstractions, which can be exposed to developers through custom APIs.

While Crossplane v1 successfully established this paradigm, certain design choices, primarily inspired by Kubernetes' PersistentVolume and PersistentVolumeClaim model, led to some complexities. Specifically, composite resources (XRs) and managed resources were primarily cluster-scoped. This necessitated a concept called "claims," which were namespaced proxies that mirrored a user's request to a cluster-scoped XR. This vertical integration, where Crossplane strongly encouraged creating managed resources only via composite resources and limited composition to only Crossplane resources, sometimes made it less flexible for managing non-infrastructure Kubernetes objects like deployments or services directly via composition, or for integrating with other tools that might provision managed resources independently. These foundational aspects, while effective, laid the groundwork for the more intuitive and flexible architecture introduced in Crossplane v2.

Key Findings

▶ Watch: Managed Resources: S3 Bucket Practical Example (1:49)

The talk highlights several key findings and advancements within the Crossplane ecosystem, particularly emphasizing the transformative changes introduced in Crossplane v2.

1. Composition as a Core Paradigm: Crossplane's fundamental contribution is the concept of composition, which allows platform engineers to define higher-level, self-service abstractions by assembling granular managed resources. This enables developers to provision complex infrastructure (e.g., a GKE cluster with networking) with simplified API definitions, adhering to "golden paths" established by platform teams.
2. Flexible Composition Logic via Functions: A significant evolution in Crossplane's composition mechanism is the use of a pipeline of functions. These functions, which can be written in various languages (e.g., Python, KCL, Go, Q, Helm-style YAML), allow platform engineers to codify unique platform logic, enforce configuration, and define how resources are composed. The robust ecosystem of reusable functions provides flexibility for different comfort levels, from no-code declarative to full-code programmatic approaches.
3. Crossplane v1.19 and 1.20 Maturity: Recent releases have focused on maturing core Crossplane APIs and features. Crossplane v1.19 saw the Usage and Claim Server-Side Apply features promoted to beta, alongside improvements in safe API promotion and support for host network scenarios and private repositories. The upcoming v1.20 release will introduce the crucial change logs feature across all providers, offering an audit trail of Crossplane's actions on resources, and further enhancements in observability and metrics.
4. Crossplane v2: A More Intuitive and Flexible Control Plane: The most significant finding is the preview release of Crossplane v2, aimed at making Crossplane "more useful for more things," more intuitive, and less opinionated. The three major changes are:

• Namespaced Composite Resources (XRs): Previously cluster-scoped, XRs can now be namespaced, eliminating the need for the "claims" concept and simplifying the user experience.
• Namespaced Managed Resources: Managed resources are also becoming namespaced, enabling better tenancy isolation and granular RBAC within Kubernetes.
• Composition of Any Kubernetes Resource: Crossplane v2 allows composite resources to compose any Kubernetes resource (e.g., Deployments, Services) directly, not just Crossplane managed resources. This broadens Crossplane's applicability beyond just infrastructure to include application and microservice abstractions.

1. Backward Compatibility and Deprecation: Crossplane v2 is designed to be backward compatible with v1, allowing for a smoother upgrade path. It introduces a legacy-cluster-scoped option for XRDs and retains support for v1-style cluster-scoped managed resources. However, it also takes the opportunity to remove long-deprecated features like ControllerConfig (never left alpha) and native patch and transform (the precursor to composition functions), prompting users to migrate to the modern function-based composition.

Technical Deep Dive

▶ Watch: The Power of Composition for Higher-Level Abstractions (3:00)

Crossplane's architecture hinges on extending the Kubernetes control plane to manage external resources. At its core are managed resources, which are Kubernetes API objects representing external services like an AWS S3 bucket or a GCP Cloud SQL instance. These objects have a spec for declarative configuration (desired state) and a status reflecting the observed state of the real-world resource, along with conditions and events for lifecycle tracking. Crossplane controllers continuously reconcile these managed resources with their external counterparts using provider-specific APIs (e.g., Amazon API for AWS).

The true power of Crossplane, however, lies in composition. Platform engineers use this mechanism to combine these granular managed resources into higher-level abstractions. This involves two key components:

1. Composite Resource Definitions (XRDs): An XRD defines the schema and API group for a new custom API that platform teams expose to their developers. For example, an XRD might define an App or a Postgres database abstraction, specifying the configuration knobs (spec fields) available to developers. In Crossplane v2, XRDs gain a scope field, allowing platform engineers to define whether the resulting Composite Resources (XRs) are namespaced or cluster-scoped. This is a crucial departure from v1, where XRs were exclusively cluster-scoped.
2. Compositions: A Composition resource specifies the logic for how an XR is fulfilled. It dictates which underlying managed resources (or any Kubernetes resources in v2) are created and how their properties are derived from the XR's spec. The mechanism for defining this logic is a pipeline of functions.

Functions are central to Crossplane's composition model. Unlike earlier versions that relied on native patch and transform (now deprecated in v2), functions provide a highly flexible and extensible way to define composition logic. They are essentially pluggable configuration languages that enable platform teams to codify their unique "golden paths" and configuration standards. Crossplane supports a wide range of languages and approaches:

• Helm-style templated YAML: For simple, declarative transformations.
• KCL (KCL Configuration Language): A new CNCF configuration language, offering more programmatic control within a configuration context.
• Python: For imperative logic, allowing complex transformations and conditional resource generation.
• Q: Another configuration language.
• Go: For full-code implementations, enabling the use of native language tools like unit tests, linting, and IDE features.

The key benefit of functions is that platform engineers can choose the language they are most comfortable with and even mix and match different functions within a single composition pipeline. Crossplane handles the lifecycle management, garbage collection, and reconciliation, allowing the function developer to focus solely on the unique logic of their platform.

Crossplane v2 Architectural Changes:

Crossplane v2 introduces fundamental shifts that address limitations and enhance flexibility:

• Namespaced XRs: In v1, XRs were cluster-scoped, requiring a "claim" (a namespaced proxy) for developers to interact with them in their namespaces. V2 eliminates claims by making XRs directly namespaced. This simplifies the API model and aligns better with Kubernetes' tenancy concepts. A platform engineer can still define cluster-scoped XRs using the scope field in the XRD, which is useful for composing cluster-wide resources like Argo CD instances or operators that manage resources across multiple namespaces.
• Namespaced Managed Resources: Similar to XRs, managed resources in v1 were generally cluster-scoped. In v2, managed resources are also becoming namespaced. This provides finer-grained tenancy isolation and enables more robust Role-Based Access Control (RBAC), allowing teams to manage their infrastructure within specific namespaces. While AWS providers already support this in the preview, other providers will be updated in the coming months.
• Decoupling Composition: Crossplane v1 had an opinionated, "vertically integrated" view, suggesting that managed resources should primarily be created via composite resources and that composite resources should largely compose Crossplane managed resources. Crossplane v2 sheds this opinion. It explicitly allows:
• Managed resources to be created by other tools (e.g., Helm charts) if desired, though composition remains the recommended "best" way.
• Composite resources to compose any Kubernetes resource, not just Crossplane managed resources. This means a single App XR can directly orchestrate a Deployment, a Service, and an AWS RDS instance, blending application and infrastructure concerns seamlessly.
• Improved User Experience: Crossplane machinery fields (like compositionRef) are moved under spec.crossplane within an XR. This clearly distinguishes internal Crossplane configuration from the actual workload-specific configuration (e.g., image for an App XR), making the API more intuitive for end-users.

Backward Compatibility: Crossplane v2 ensures a smooth upgrade path by supporting v1-style cluster-scoped XRs and managed resources, albeit marking them as "legacy features" to encourage migration. A legacy-cluster-scoped option for XRDs maintains v1 behavior. However, certain deprecated features have been removed:

• ControllerConfig: This alpha feature for configuring providers, deprecated for 11 releases, is now removed.
• Native patch and transform: The precursor to composition functions, deprecated for 2-3 releases, is also removed. Users relying on this must migrate to composition functions before upgrading to v2.

Demo / Proof of Concept

▶ Watch: Defining Your Platform API using CRDs (5:58)

Nic Cope presented a concise and impactful live demonstration of Crossplane v2, showcasing its new namespaced capabilities and the flexibility of composition functions. The demo's objective was to create a simple App composite resource that, when instantiated by a developer, would automatically provision a standard Kubernetes Deployment and Service in the same namespace, without involving any external cloud managed resources.

The demonstration followed three key steps:

1. Defining the App XR with an XRD:

• First, an XRD (Composite Resource Definition) was applied. This YAML manifest defined the App custom resource, specifying its API group (apps.example.org), kind (App), and schema. For this demo, the App's spec only contained a single field: image, indicating the container image to be deployed. Crucially, the XRD for this App was configured to create a namespaced XR, a cornerstone feature of v2.

1. Installing a Composition Function:

• The next step involved installing a composition function that would define the logic for composing the Deployment and Service. The speaker offered a choice between Helm-style templated YAML, Python, and KCL (KCL Configuration Language). The audience voted for KCL.
• (Note: Due to potential conference Wi-Fi issues, the speaker pre-installed all four function types, but the focus was on KCL.) This function, once installed, provides Crossplane with the executable logic for a given composition.

1. Configuring the Composition:

• Finally, a Composition resource was applied. This resource linked the App XRD to the installed KCL function, instructing Crossplane that whenever an App XR is created, it should invoke the KCL function to generate the necessary Kubernetes resources. The KCL function's output, as defined in the Composition, would be a Kubernetes Deployment and Service. The replicas for the Deployment would be copied from the App's status, and the address in the App's status would be copied from the Service's status.

Execution and Results:

Once these foundational components were in place, the speaker applied a simple App YAML manifest:

Upon applying this App resource in a specific namespace, Crossplane immediately reconciled it:

• It identified the App XR.
• It located the corresponding Composition (the KCL one).
• It called the KCL function, passing the App XR's spec as input.
• The KCL function generated the YAML for a Kubernetes Deployment and Service based on the image specified in the App XR.
• Crossplane then created these Deployment and Service resources within the same namespace as the App XR.

The demo concluded by using the kubectl tree plugin, which visually confirmed that the my-webapp App XR was indeed composed of a Deployment and a Service, all residing within the same namespace. The speaker emphasized this as the "big change" in Crossplane v2: the seamless, namespaced orchestration of arbitrary Kubernetes resources directly from a custom XR. The App XR quickly showed synced: true and ready: true, indicating successful provisioning.

Defensive Implications

▶ Watch: Functions: The Core of Crossplane Composition Logic (6:40)

The advancements in Crossplane, particularly with the v2 preview, offer significant advantages for security and operational posture, empowering platform teams to build more secure, compliant, and observable systems.

1. Enforced Golden Paths and Policy-as-Code: Crossplane's composition mechanism allows platform engineers to define and enforce "golden paths" for infrastructure provisioning. By encapsulating best practices, security configurations, and compliance requirements within XRDs and Compositions, developers are abstracted from the underlying complexity and inherently provision secure-by-default resources. This shifts policy enforcement left, ensuring that infrastructure aligns with organizational standards from the moment it's requested.
2. Enhanced Tenancy and RBAC with Namespaced Resources: The introduction of namespaced XRs and managed resources in Crossplane v2 is a game-changer for multi-tenant environments. Previously, cluster-scoped resources complicated fine-grained access control. Now, platform teams can grant developers granular RBAC permissions to create App XRs or even direct managed resources only within their designated namespaces. This provides stronger isolation between teams and workloads, reducing the blast radius of misconfigurations or compromised credentials.
3. Improved Auditability with Change Logs: The upcoming change logs feature in Crossplane v1.20 across all providers is a critical defensive capability. It will provide a detailed audit trail of "everything Crossplane is doing to all of your resources, why it's doing it, when it's doing it, how it's doing it, where it's doing it." This level of visibility is invaluable for incident response, compliance audits, and understanding the complete lifecycle and evolution of provisioned infrastructure.
4. Controlled Resource Reconciliation and Rate Limiting: Crossplane provides mechanisms to control the reconciliation loop interval for providers, allowing platform teams to slow down how often resources are synced with the real world. This directly addresses concerns about rate limiting by cloud providers, preventing service disruptions due to excessive API calls. Furthermore, the ability to apply a pause annotation to resources or composite resources allows operators to temporarily halt all reconciliation, providing a critical circuit breaker during maintenance windows or when investigating issues.
5. Reduced Attack Surface through Abstraction: By exposing high-level abstractions to developers, Crossplane reduces the need for developers to have direct access to low-level cloud provider APIs or credentials. The platform team manages the underlying Crossplane providers and their credentials, effectively centralizing and securing access to sensitive infrastructure operations.
6. Future-Proofing with Real-time Compositions: While still in alpha, the concept of real-time compositions aims to move away from constant polling at the composition layer towards watch-based or pub/sub mechanisms. This will lead to fewer unnecessary reconciles, reducing the overall load on cloud APIs, improving efficiency, and potentially lowering the risk of unintended or frequent state changes.

Key Takeaways

• Crossplane as a Universal Control Plane: Crossplane extends Kubernetes to manage any external resource—cloud, SaaS, or on-premises—treating them as Kubernetes API objects for declarative provisioning and management.
• Composition and Functions are Core: The ability to compose granular resources into higher-level abstractions, defined by flexible, language-agnostic functions, is fundamental to building powerful, self-service developer platforms.
• Crossplane v2 is a Major Evolution: The upcoming Crossplane v2 preview significantly enhances usability and flexibility by introducing namespaced XRs and managed resources, eliminating the "claims" concept, and allowing composition of any Kubernetes resource, not just Crossplane-specific ones.
• Enhanced Security and Compliance: Features like enforced "golden paths" via composition, granular RBAC with namespaced resources, and the forthcoming audit logs (change logs in v1.20) provide robust defensive capabilities for platform teams.
• Community-Driven Development: The Crossplane project actively seeks community feedback, especially for the v2 preview, and welcomes new contributors to shape its future direction.
• Maturity and Observability: Ongoing development focuses on maturing core APIs (e.g., Usage, Claim Server-Side Apply in v1.19) and improving observability and metrics to provide deeper insights into Crossplane's operations.

About the Speaker(s)

Jared Watts and Nic Cope are integral members of the Crossplane project leadership. They have dedicated a significant portion of their careers to the development and advancement of Crossplane, demonstrating deep passion and expertise in the cloud-native control plane space. Nic Cope specifically led the efforts for the Crossplane v2 preview, highlighting his role in shaping the future architectural direction of the project. Their combined experience and commitment are pivotal to Crossplane's ongoing success and innovation.