Choose Your Own Adventure: The Dignified Pursuit of a Developer Platf... Whitney Lee & Viktor Farcic

Choose Your Own Adventure: The Dignified Pursuit of a Developer Platform

Speakers: Whitney Lee, Developer Advocate, VMware Tanzu; Viktor Farcic, Principal Developer Advocate, Upbound

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=hnmtjCkO8FE

Overview

In the sprawling landscape of cloud-native technologies, developers often find themselves overwhelmed by the sheer number of choices and the complexity of integrating them. This talk, "Choose Your Own Adventure: The Dignified Pursuit of a Developer Platform," presented by Whitney Lee and Viktor Farcic, tackles this challenge head-on by demonstrating how to construct an Internal Developer Platform (IDP). The core premise revolves around a metaphorical "Hero," representing application source code, aspiring to run in production. The speakers illustrate how an IDP, built on Kubernetes APIs, can streamline this journey, abstracting away intricate infrastructure details and empowering developers with self-service capabilities.

The presentation is highly interactive, simulating a live "You Choose" show where the audience votes on specific technology choices for an IDP component, which Viktor then live-demos. This engaging format highlights the practical considerations and trade-offs involved in platform engineering. The talk underscores the critical role of platform engineers in bridging the gap between specialized experts and application developers, ensuring that services are delivered in a compliant and secure manner without requiring deep infrastructure knowledge from end-users.

Ultimately, this talk provides a compelling blueprint for organizations looking to simplify their development workflows and accelerate their path to production. By advocating for Kubernetes APIs as the foundational layer, Lee and Farcic showcase how an IDP can harness the power of the Kubernetes ecosystem to manage diverse resources, enforce policies, automate pipelines, and provide a unified developer experience, even for resources residing outside the Kubernetes cluster itself.

Background

▶ Watch: Introduction and the overwhelming CNCF landscape (0:00)

The journey to production for an application, as personified by "Hero" in this talk, is fraught with decisions, particularly within the cloud-native ecosystem. The Cloud Native Computing Foundation (CNCF) alone hosts hundreds of projects, each offering solutions for various aspects of application deployment, management, and security. For developers, navigating this labyrinth of choices, integrating disparate tools, and maintaining a secure, compliant posture can be a significant burden, diverting focus from core application development.

This immense complexity has given rise to platform engineering, a discipline focused on building and maintaining internal developer platforms. An IDP is defined as a set of integrated capabilities that empower developers in their day-to-day work. Whitney Lee and Viktor Farcic categorize the key stakeholders in platform engineering into three groups:

1. Experts: Deeply knowledgeable in one specific domain (e.g., infrastructure, security).
2. Developers: Primarily focused on their application, desiring to consume services without understanding underlying complexities.
3. Platform Engineers: The crucial intermediary, enabling developers to self-service their needs from experts, ensuring the resulting services are naturally compliant and secure.

The speakers' approach to exploring these choices is rooted in their YouTube show, "You Choose," where they evaluate different CNCF technologies for specific system design problems. Prior to this KubeCon presentation, their show had already made several foundational IDP choices, including Buildpacks for container images, Harbor for a registry, Carvel for application configuration, Cert-Manager for certificates, Crossplane for declaratively defining databases, SchemaHero for database schema management, DevSpace for Kubernetes development, Argo CD for GitOps, Contour for ingress, KubeArmor for runtime security, External Secrets Operator for secrets, Cilium for network policy, KubeScape for Kubernetes scanning, Notary Project for signing, Spiffe/Spire for workload identity, KCLO for authentication, OpenFGA for authorization, Headlamp for Kubernetes API visualization, Thanos and Prometheus for metrics, OpenTelemetry and Jaeger for tracing, Istio for service mesh, and OpenCost for cost management. This extensive list vividly illustrates the scale of integration challenges an IDP aims to solve.

Key Findings

▶ Watch: Live demo plan: building a self-service developer platform (3:00)

The central finding and contribution of this talk is the demonstration of how to construct a functional, self-service Internal Developer Platform (IDP) using Kubernetes APIs as the universal control plane, even for resources that don't natively run within Kubernetes. The speakers effectively argue that platform engineering is not just about choosing tools, but about creating an integrated, compliant, and secure environment that empowers developers.

The key discoveries and results presented are:

• Kubernetes as the Foundational API Layer: The talk firmly establishes Kubernetes APIs as the industry standard and the ideal backbone for an IDP. This allows leveraging the Kubernetes synchronization loop to declaratively define desired states and utilize the vast Kubernetes ecosystem for managing diverse resources.
• Abstraction and Self-Service: By building custom Kubernetes APIs (like SQLClaim or AppClaim), platform engineers can abstract complex infrastructure details (e.g., subnets, VPCs, security groups) from developers. This enables developers to provision resources like databases or applications with simplified YAML manifests, fostering a true self-service model.
• Policy-Driven Compliance and Security: Implementing robust admission controller policies, exemplified by Kyverno, is crucial. This allows platform engineers to define and enforce rules (e.g., resource limits, required configurations) at the Kube API server level, preventing misconfigurations, accidental resource bloat, or security vulnerabilities before they are even applied to the cluster.
• Automated Workflows and Pipelines: The IDP can integrate tools like Argo Workflows to automate one-time tasks such as testing, building container images, pushing to registries, and updating Git repositories. This provides a structured and repeatable way to manage CI/CD pipelines as part of the platform.
• Enhanced Developer Experience with GUIs: While not strictly necessary, a graphical user interface like Backstage significantly improves developer experience by providing discoverability, visualization, and simplified interaction with platform capabilities. It acts as a "single pane of glass" for both developers and platform engineers to understand the state of provisioned resources.
• Bridging On-Cluster and Off-Cluster Resources: Tools like Crossplane are highlighted as essential for extending the Kubernetes control plane to manage external cloud resources (e.g., AWS databases) as if they were native Kubernetes objects, unifying management across hybrid environments.

These findings collectively illustrate a practical and opinionated approach to building an IDP that addresses the challenges of cloud-native complexity, promotes developer autonomy, and upholds organizational standards for security and compliance.

Technical Deep Dive

▶ Watch: Why Kubernetes APIs are essential for your platform (4:00)

The technical core of this talk revolves around the strategic use of Kubernetes APIs as the unifying interface for an Internal Developer Platform (IDP). The speakers emphasize three primary reasons for this choice:

1. Industry Standard: Kubernetes is widely adopted, meaning developers are increasingly familiar with its declarative model.
2. Synchronization Loop: The declarative nature of Kubernetes allows platform engineers to define desired states, and Kubernetes continuously works to achieve and maintain that state. This principle extends to managing resources outside the Kubernetes cluster itself.
3. Ecosystem Leverage: The vast and mature Kubernetes ecosystem provides a wealth of tools and patterns that can be adapted for IDP functionalities.

The talk then dives into specific system design choices, with the audience voting on preferred technologies for each component.

API Management and State Management

The first choice focused on how to manage APIs and state, particularly for provisioning infrastructure.

• Crossplane: This tool allows platform engineers to take any API (inside or outside Kubernetes, including SaaS) and manage it as a Kubernetes API. It provides the Kubernetes synchronization loop for external resources, enables composition of multiple resources, and offers a simplified interface to developers. The drawback is its inherent complexity dueard to its power and flexibility.
• KubeVela: A more focused tool specifically designed for managing applications. It provides Kubernetes abstractions to define and manage applications, components, and traits within Kubernetes. It offers strong community support but is less broad in its scope compared to Crossplane.

The audience voted for Crossplane. This choice underscores the desire for a platform that can manage a wide array of resources, not just those running directly on Kubernetes.

Admission Controller Policy

Next, the talk addressed how to enforce policies and prevent developers from making non-compliant or destructive requests.

• ValidatingAdmissionPolicy (VAP): This is a relatively new, built-in Kubernetes feature. It uses Common Expression Language (CEL) to define lightweight rules that the Kube API server can evaluate to validate incoming requests. At the time of the talk, it primarily focused on validation, with mutation capabilities under development.
• Kyverno: A more mature, dedicated policy engine for Kubernetes. It supports CEL but also offers its own Kyverno JSON Query Language, designed to feel like YAML. Kyverno provides extensive capabilities beyond just validation, including mutation, resource generation, and resource cleanup. It boasts significant community support. The recommendation was to start with VAP for simple cases and migrate to Kyverno for more full-featured needs.

The audience chose Kyverno, favoring its maturity and comprehensive feature set for robust policy enforcement.

One-Time Tasks / Pipelines

This section explored how the IDP can assist developers with one-time tasks, typically associated with CI/CD pipelines.

• Tekton: Specifically designed for CI use cases, where one task leads to another in a straightforward pipeline. It can handle more advanced scenarios but is optimized for simplicity and ease of getting started with basic pipelines.
• Argo Workflows: Designed for more complex, Directed Acyclic Graphs (DAGs), often found in AI/ML or data processing workflows. It supports advanced features like loops and conditionals, making it suitable for intricate, multi-step processes. While powerful, it is generally more complicated to use and understand initially.

The audience selected Argo Workflows, indicating a preference for a tool capable of handling potentially more complex and diverse pipeline requirements within the IDP.

Graphical User Interface (GUI)

Finally, the discussion turned to enhancing the developer experience with a GUI, acknowledging that while not strictly necessary, it greatly aids discoverability and management.

• Backstage: A very popular CNCF project (reportedly the second most contributed-to project after Kubernetes itself). It functions as a framework for building custom developer portals rather than a ready-made portal. It offers numerous existing plugins, but custom extensions require TypeScript development, which can introduce complexity for initial setup and customization.
• Port: A commercial solution offering a low-code/no-code approach to building IDP GUIs. It aims for ease of use and quicker setup, simplifying the creation of a developer portal.

The audience voted for Backstage, reflecting its popularity and the community's interest in building highly customizable, open-source developer portals.

Demo / Proof of Concept

▶ Watch: Crossplane: unifying any API under Kubernetes management (6:20)

Viktor Farcic conducted a live demonstration, building out an Internal Developer Platform (IDP) based on the audience's technology choices. The demo showcased the self-service capabilities and policy enforcement in action.

Crossplane for API Management

The first part of the demo illustrated how Crossplane extends the Kubernetes control plane to manage external resources.

1. GitHub Repository Claim: Viktor began by showing a pre-configured GitHubClaim which, when applied, created a GitHub repository. This established the pattern of using custom Kubernetes resources to provision external services.
2. AWS SQL Database Provisioning: The core of this segment was provisioning an AWS SQL database using a custom Kubernetes API called SQLClaim. Viktor applied a simple YAML manifest:

This manifest dramatically simplifies database provisioning for a developer, abstracting away the underlying complexities of AWS (VPCs, subnets, security groups, database engine versions, instance sizes). Viktor demonstrated that behind the scenes, Crossplane translated this SQLClaim into numerous AWS resources, showcasing the power of composition and abstraction.

1. Application Deployment: Following the database, an application was deployed using another custom Crossplane-managed API, CrossplaneApp (or a similar custom application claim). This allowed developers to define their application with minimal parameters like image, tag, and target environment (e.g., production), without needing to specify individual Kubernetes deployments, services, or ingresses. The goal was for this to trigger a Pull Request (PR) in the GitHub repository, demonstrating GitOps integration. While there was a brief hiccup in the live demo where the application didn't immediately appear, the underlying mechanism of applying a simplified manifest to trigger complex infrastructure and application deployment was clear.

Kyverno for Policy Enforcement

The demo then pivoted to Kyverno to enforce policies on application deployments.

1. Policy Definition: Viktor applied two Kyverno cluster-level policies targeting AppClaim resources:

• One policy required the scaling.enabled field to be set to true.
• Another policy required the scaling.min field (minimum replicas) to be greater than 1.

1. Policy Enforcement in Action: Viktor attempted to re-apply the previously defined AppClaim YAML, which initially lacked the required scaling parameters. The Kube API server, via the web hook configured for Kyverno, immediately rejected the request, providing clear error messages indicating which policies were violated ("Application scaling must be enabled," "Minimum replicas must be greater than one").
2. Compliance and Success: Viktor then modified the AppClaim YAML to include scaling.enabled: true and scaling.min: 4. Upon re-application, the Kyverno policies were satisfied, and the application was successfully deployed, demonstrating how policies guide developers towards compliant configurations and prevent missteps.

Argo Workflows for One-Time Tasks

The demonstration moved to automating pipelines using Argo Workflows.

1. Workflow Definition: Viktor presented a simplified Argo Workflow YAML, designed for CI/CD. This workflow included several steps:

• Checkout Code: Fetching application source code from a repository.
• Generate Tag: Creating a version tag for the build.
• Build Image: Using Kico (a tool for building container images without a Docker daemon) to build the application image.
• Run Unit Tests: Executing application tests.
• Git Push: Modifying a YAML file (e.g., updating the image tag) and pushing changes back to the Git repository, which Argo CD (or Flux) would then pick up for deployment.

1. Workflow Execution: Instead of triggering it from GitHub (due to time constraints), Viktor manually submitted the workflow to the Kubernetes cluster. The demo showed the workflow running, progressing through its defined tasks. This illustrated how the IDP could manage and execute complex, multi-step pipelines, abstracting the underlying compute and orchestration for developers.

Backstage for Graphical User Interface

The final segment showcased Backstage as the developer portal for the IDP.

1. Dynamic Discovery: Viktor demonstrated a dynamically configured Backstage instance. Unlike typical Backstage setups that require explicit catalog-info.yaml files, this instance was configured to automatically discover resources directly from the Kubernetes cluster.
2. Resource Visualization: Backstage displayed the CNCF demo app created earlier, showing its associated resources and a graphical representation (boxes and arrows) of how these components were related. This provides developers with an intuitive overview of their provisioned infrastructure and applications.
3. Self-Service Form Generation: A key feature demonstrated was Backstage's ability to dynamically generate forms for provisioning new resources. By inspecting the Kubernetes schemas of SQLClaim, AppClaim, and GitHubClaim, Backstage automatically created user-friendly forms. Developers could fill in fields like name, namespace, owner, and specific parameters (e.g., database version, size) through a guided interface.
4. GitOps Integration: Viktor emphasized that submitting these forms in Backstage would not directly apply changes to the cluster. Instead, it would push the generated YAML to Git. Argo CD would then detect these changes and synchronize them to the cluster, ensuring all platform operations adhere to the GitOps paradigm. This reinforces the principle of a read-only GUI driving a declarative, Git-centric workflow.

The demo, despite minor live challenges, effectively illustrated the power of an IDP built on Kubernetes APIs, offering self-service, policy enforcement, automated pipelines, and a streamlined developer experience.

Defensive Implications

▶ Watch: Kubevela: focused application API management tool (7:15)

Building an Internal Developer Platform (IDP), as demonstrated, has profound defensive implications, fundamentally enhancing the security posture of an organization's cloud-native environment. By centralizing control and abstracting complexity, IDPs can embed security and compliance from the ground up.

1. Policy Enforcement with Kyverno: This is perhaps the most direct defensive mechanism highlighted. Kyverno allows platform engineers to define granular policies that are enforced at the Kube API server level. This means security best practices can be codified and automatically applied to all resources provisioned through the platform. Examples include:

• Resource Limits: Preventing developers from provisioning excessively large or expensive resources, which could lead to denial-of-service vulnerabilities or cost overruns.
• Trusted Registries: Ensuring container images are only pulled from approved, secure registries, mitigating risks from untrusted or malicious images.
• Configuration Guardrails: Mandating specific security configurations (e.g., requiring network policies, disallowing root containers, enforcing read-only filesystems).
• Vulnerability Management: Preventing deployments with known outdated dependencies or critical CVEs by integrating with vulnerability scanning tools.
• Compliance Automation: Automatically validating that resources adhere to regulatory compliance standards (e.g., HIPAA, PCI DSS) before they are deployed.

1. Controlled Access to Infrastructure via Crossplane: By abstracting complex infrastructure (like AWS VPCs, subnets, and security groups) behind simplified Kubernetes APIs (e.g., SQLClaim), Crossplane significantly reduces the attack surface. Developers are granted permissions only to interact with these high-level custom resources, rather than having direct access to underlying cloud provider APIs. This enforces the principle of least privilege and prevents misconfigurations that could expose sensitive data or services.

1. Secure Supply Chain with GitOps and Automation: The integration of Argo Workflows and GitOps principles (e.g., pushing changes to Git for Argo CD to synchronize) ensures an auditable and immutable record of all infrastructure and application changes. This transparency aids in forensic analysis and rollback capabilities. Furthermore, the ability to bake security checks (e.g., unit tests, static analysis, image scanning) directly into automated pipelines means vulnerabilities can be identified and remediated early in the development lifecycle. The initial mention of Notary Project for image signing further strengthens the software supply chain by ensuring the integrity and authenticity of container images.

1. Runtime Security and Network Segmentation: While not explicitly detailed in the live demo, the speakers' "You Choose" show had already selected KubeArmor for runtime security and Cilium for network policy. These components are vital for defensive posture:

• KubeArmor: Provides host and container-level runtime protection, enforcing security policies based on application behavior and preventing unauthorized process execution or file access.
• Cilium: Offers highly granular network segmentation and policy enforcement at the CNI level, isolating workloads and preventing lateral movement in case of a breach.

1. Centralized Secrets Management: The use of External Secrets Operator (also from the "You Choose" stack) ensures that sensitive data like API keys, database credentials, and certificates are managed securely outside of Git, integrated with external secrets management systems, and injected into applications at runtime, minimizing exposure.

1. Workload Identity and Authentication/Authorization: The inclusion of Spiffe/Spire for workload identity and KCLO (authentication) / OpenFGA (authorization) in their overall IDP stack provides a robust framework for securing inter-service communication and controlling access to resources within the platform, regardless of where they run.

In essence, an IDP shifts security left, embedding it into the platform's design and automating its enforcement. This transforms security from a reactive bottleneck into a proactive, self-service capability, allowing developers to build securely by default without becoming security experts themselves.

Key Takeaways

• Kubernetes APIs are the Foundation of Modern IDPs: Leveraging Kubernetes' declarative model and ecosystem allows platform engineers to build a unified control plane for managing both on-cluster and external cloud resources.
• Abstraction Empowers Developers: Custom Kubernetes APIs (like SQLClaim and AppClaim) abstract complex infrastructure details, enabling developers to self-service their needs with simplified manifests, fostering autonomy and accelerating development.
• Policy Enforcement is Non-Negotiable: Tools like Kyverno are crucial for embedding security and compliance directly into the platform, ensuring that all provisioned resources adhere to organizational standards and preventing misconfigurations.
• Automated Pipelines are Essential for Efficiency: Integrating workflow engines like Argo Workflows into the IDP streamlines CI/CD processes, automating tasks from code checkout and image building to testing and GitOps-driven deployments.
• Developer Portals Enhance Experience and Discoverability: While not strictly mandatory, a GUI like Backstage significantly improves developer experience by offering dynamic visualization, self-service form generation, and a single pane of glass for understanding the IDP's capabilities and deployed resources.
• Platform Engineering Bridges Expertise Gaps: The IDP acts as the interface between specialized experts and application developers, ensuring that secure, compliant, and integrated capabilities are readily available in a self-service manner.

About the Speaker(s)

Whitney Lee is a Developer Advocate at VMware Tanzu and co-host of the "You Choose" streaming show. Known for her engaging presentations and ability to simplify complex topics, she guides audiences through the myriad of cloud-native choices, often with a humorous and energetic approach. Her role involves helping developers understand and utilize the technologies within the cloud-native ecosystem.

Viktor Farcic is a Principal Developer Advocate at Upbound and co-host of the "You Choose" show with Whitney Lee. He is also known for his "DevOps Toolkit" YouTube channel, where he explores various DevOps and cloud-native tools and practices. Viktor is a prolific author, speaker, and trainer, focusing on GitOps, Kubernetes, and developer experience. His expertise lies in demonstrating the practical application of these technologies to build efficient and scalable systems.