From 0 To Production-Grade With Kubernetes Native Development - Thomas Vitale & Kevin Dubois

From 0 To Production-Grade With Kubernetes Native Development

Speakers: Thomas Vitale, Engineer at Systematic; Kevin Dubois, Developer Advocate at Red Hat

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=07RnkzSc6Jg

Overview

This talk, "From 0 To Production-Grade With Kubernetes Native Development," by Kevin Dubois and Thomas Vitale, delves into the evolving landscape of software development for Kubernetes environments. It addresses the fundamental shift from traditional application deployment, often characterized by large, resource-intensive application servers, to a Kubernetes native development paradigm. The core premise is that applications running on Kubernetes clusters must be designed for rapid startup, efficient resource utilization, and resilience, which necessitates a different approach from developers.

The speakers highlight the challenges faced by application developers who need to deploy to Kubernetes without necessarily being Kubernetes experts. They present a comprehensive toolkit of open-source projects and methodologies, primarily from the CNCF landscape, that aim to simplify the developer experience, reduce cognitive load, and foster "developer joy." The talk covers everything from local containerization and development setup to production-grade deployment strategies, emphasizing fast feedback loops and seamless integration.

Ultimately, the session champions an approach where developers can focus on their application logic while leveraging powerful abstractions and automation to handle the complexities of Kubernetes. By demonstrating various tools and frameworks, Dubois and Vitale illustrate how to achieve a consistent and efficient development workflow that bridges the gap between local development and production deployment, ensuring applications are truly cloud-native from conception.

Background

▶ Watch: Introduction to Kubernetes Native Development principles (0:00)

Traditionally, software development often involved creating applications that ran effectively on a developer's local machine, then packaging them for deployment onto large, often monolithic, application servers. This model often led to resource-intensive applications, especially in environments like Java, where memory and CPU consumption might not be a primary concern during initial development. The "throw it over the wall" mentality, where developers handed off code to operations teams for deployment, further disconnected development practices from the realities of production environments.

The advent of Kubernetes introduced a paradigm shift, demanding applications that are designed to operate efficiently within a distributed, containerized environment. Key requirements for Kubernetes-native applications include:

• Fast Startup Times: Essential for rapid scaling, rescheduling, and fault recovery.
• Low Resource Usage: Maximizing cluster density and optimizing infrastructure costs.
• Resilience: Applications must be able to handle being scaled up, down, or rescheduled without disruption.

However, developing for Kubernetes presents significant challenges for application developers who are not necessarily infrastructure or operations experts. Deploying an application to Kubernetes typically requires defining multiple YAML manifests, including:

• Deployments: To manage application pods.
• Services: To expose applications within or outside the cluster.
• Probes: Readiness probes (to indicate when an application is ready to receive traffic), liveness probes (to check if an application is still running and healthy), and startup probes (to handle slow-starting applications) are crucial for Kubernetes to manage application lifecycle effectively. Without these, Kubernetes might route traffic to an unready application, leading to downtime.
• Configuration Management: Handling environment variables, connection strings, and sensitive data via ConfigMaps and Secrets. Developers need to understand how these are injected and consumed.
• Resource Management: Specifying resource requests (guaranteed minimums for CPU and memory) and resource limits (maximums to prevent resource exhaustion and ensure fair sharing). Misconfigured resources can lead to throttling or pod eviction.

The sheer volume and complexity of this YAML configuration, coupled with the need for a fast feedback loop during development, often overwhelm developers. The background problem, therefore, is how to empower developers to build and deploy Kubernetes-native applications efficiently, securely, and with joy, without requiring them to become Kubernetes YAML gurus. This talk explores solutions that abstract away this complexity, allowing developers to focus on business logic while still adhering to cloud-native best practices.

Key Findings

▶ Watch: Cloud Native Buildpacks: automatically create production-ready images (2:30)

The talk presents a robust set of tools and methodologies designed to streamline Kubernetes native development, focusing on developer experience and efficient deployment from local environments to production. The key findings and contributions can be summarized as follows:

1. Simplified Containerization with Buildpacks: The adoption of Cloud Native Buildpacks (specifically the pac cli) allows developers to create production-ready container images directly from source code without writing or maintaining Dockerfiles. This significantly reduces boilerplate and ensures images are performant and secure by default, supporting a wide range of programming languages and frameworks (e.g., Spring Boot, Quarkus).

1. Integrated Local Development Environment with Podman Desktop: Podman Desktop emerges as a central tool for local development. It not only manages containers locally but also facilitates the creation and management of local Kubernetes clusters (using kind or minikube). Crucially, it offers the ability to auto-generate Kubernetes manifests (Deployment, Service) from running containers and deploy them to the local cluster directly from the UI, abstracting much of the initial YAML complexity.

1. Framework-Specific Kubernetes Automation (Quarkus): For Java developers, Quarkus demonstrates advanced Kubernetes integration. Through specific extensions (quarkus-kubernetes, quarkus-kubernetes-config, quarkus-smallrye-health), Quarkus can automatically generate comprehensive Kubernetes YAML, including deployments, services, service accounts, role bindings, environment variables, resource requests/limits, and sophisticated readiness and liveness probes that account for application dependencies (e.g., Kafka, databases). This drastically reduces manual YAML writing and configuration errors.

1. Serverless on Kubernetes with Knative: The talk highlights Knative as a powerful CNCF project enabling serverless capabilities directly on Kubernetes. Knative Functions (via kn func cli) provide a polyglot approach to creating, building (leveraging Buildpacks), and deploying functions that automatically scale to zero when idle and scale up rapidly under load, abstracting ingress and pod management.

1. Local Dependency Management without Kubernetes (Testcontainers): A significant finding for local development is the use of Testcontainers. This polyglot tool allows developers to programmatically provision and manage lightweight containers for application dependencies (databases like PostgreSQL, messaging systems like RabbitMQ, identity management like Keycloak, or even AI models like Ollama) directly within their development or testing workflow. This eliminates the need for a full local Kubernetes cluster just for dependencies, reducing resource overhead and setup complexity.

1. Local Observability with Testcontainers: Extending the utility of Testcontainers, the speakers demonstrate its use in provisioning a full observability stack (e.g., OpenTelemetry collector and Grafana) locally. This enables developers to trace requests, visualize metrics, and debug issues (like LLM hallucinations) in their development environment, bringing production-grade observability practices earlier into the development cycle.

1. API Mocking and Testing with Micros: Micros, another CNCF project, is introduced for mocking and testing APIs in distributed systems. It supports various protocols (HTTP, messaging) and can be integrated into local development (potentially via Testcontainers) and CI/CD pipelines, providing consistent test data and scenarios across the development lifecycle. It even has a Kubernetes operator for cluster-wide mocking.

1. Production-Grade Service Binding with Service Binding API: For connecting applications to actual services in production, the Kubernetes Service Binding API provides a standardized, declarative mechanism. Developers declare their application's intent to consume a service (e.g., PostgreSQL, Ollama inference), and the platform fulfills that request, separating concerns and enabling a more automated, self-service model.

1. Developer Portal Integration (Backstage): The talk concludes by showcasing how a developer portal like Backstage can integrate these tools into a "golden path." Developers can bootstrap new projects, automatically configuring local environments with Testcontainers and Micros, and production deployments with the Service Binding API, providing a fully automated, opinionated, and seamless experience from day zero.

These findings collectively present a cohesive strategy for modern cloud-native development, focusing on automation, abstraction, and a superior developer experience across the entire software lifecycle.

Technical Deep Dive

▶ Watch: Demo: Build & run application locally with Buildpacks (4:00)

The technical depth of the talk spans several layers of the cloud-native stack, from local container management to advanced Kubernetes features and development workflows.

The journey begins with local containerization. Podman Desktop serves as the developer's local runtime, offering a user-friendly interface for managing containers. Beyond simple container execution, it facilitates the creation of local Kubernetes clusters using tools like kind or minikube. This capability is crucial for developers needing to test their applications in a Kubernetes-like environment without the overhead of a remote cluster. A significant feature demonstrated is the ability for Podman Desktop to auto-generate Kubernetes manifests (Deployment, Service) from a running container image and deploy them to the local cluster, abstracting the initial YAML writing process. The generated manifests include basic deployments and services, and Podman Desktop even configures an ingress controller out-of-the-box for easy local access without manual port forwarding.

For packaging applications into containers, the talk advocates for Cloud Native Buildpacks. This technology allows developers to transform application source code directly into production-ready container images without needing to write a Dockerfile. Buildpacks automatically detect the application's language and framework (e.g., Java, Go, Spring Boot, Quarkus) and apply best practices for security, performance, and layering. This results in smaller, more secure images with faster build times and improved patchability, as dependencies are separated into distinct layers. The pac cli is the command-line interface for interacting with Buildpacks, providing a simple pack build command.

Moving to actual Kubernetes deployment, the speakers emphasize the critical components beyond basic deployments:

• Readiness Probes: An HTTP GET, TCP socket, or exec command that Kubernetes periodically checks to determine if the application is ready to accept traffic. This prevents Kubernetes from sending requests to an application that has started its container but hasn't fully initialized its internal dependencies (e.g., database connections, Kafka consumers).
• Liveness Probes: Similar checks to determine if the application is still running and healthy after startup. If a liveness probe fails, Kubernetes will restart the pod, ensuring application availability.
• Startup Probes: Introduced for applications with long startup times, these delay liveness and readiness checks until the application has successfully started, preventing premature restarts.
• Resource Requests and Limits: These are crucial for cluster stability and efficient resource allocation. Resource requests guarantee a minimum amount of CPU and memory for a pod, ensuring it can run. Resource limits define the maximum CPU and memory a pod can consume; exceeding these can lead to CPU throttling or memory-related pod eviction.

To simplify the generation of these complex Kubernetes manifests, the talk highlights framework-specific solutions like Quarkus for Java. Quarkus, through extensions such as quarkus-kubernetes, quarkus-kubernetes-config, and quarkus-smallrye-health, can automatically generate a comprehensive Kubernetes YAML file. This includes:

• A Deployment resource with the application image.
• A Service to expose the application.
• A ServiceAccount and RoleBinding for necessary permissions (e.g., accessing secrets).
• Environment variables from ConfigMaps or Secrets.
• Configured resource requests and limits based on application properties.
• Dynamically generated readiness and liveness probes that intelligently detect and incorporate application dependencies (e.g., a database connection, a Kafka client), ensuring Kubernetes only considers the application ready when all its critical dependencies are functional.

For serverless patterns on Kubernetes, Knative is presented. Knative provides a set of building blocks for running serverless workloads, including intelligent auto-scaling (scaling to zero when idle) and request-driven execution. Knative Functions (kn func cli) abstract away much of the Knative complexity, allowing developers to create functions in various languages (Go example shown) and deploy them directly. Under the hood, Knative Functions leverage Buildpacks to containerize the function code, then deploy it as a Knative Service, which handles ingress, routing, and auto-scaling based on traffic.

The discussion then shifts to local development dependencies without a full Kubernetes cluster, introducing Testcontainers. This library allows developers to programmatically define and run lightweight, disposable Docker containers as part of their application's development or test lifecycle. For example, a Java application can declare a dependency on a PostgreSQL database, an Ollama LLM service, or a RabbitMQ message broker, and Testcontainers will spin up these services in Docker containers when the application starts. This provides a consistent and isolated environment for local development and integration testing. A notable demonstration involved using Testcontainers to provision an OpenTelemetry collector and Grafana alongside the application. This enables developers to visualize traces and metrics locally, gaining observability into their application's behavior (e.g., LLM interactions) without deploying to a remote cluster.

To address the challenge of integrating with external APIs and distributed systems, Micros is introduced. Micros is a CNCF project designed for API mocking and testing. It allows developers to define mock services for HTTP APIs, asynchronous messaging (e.g., Kafka), and other protocols. This is invaluable for testing microservices in isolation or for simulating complex external dependencies during development and CI/CD. Micros offers both a CLI for automation and a UI for configuration and inspection, and it can even run as a Kubernetes operator.

Finally, for production deployment and dependency management, the Kubernetes Service Binding API is highlighted. This API standardizes how applications declare their need for backing services (e.g., a database, an LLM inference service) and how those services are connected. It separates the application's intent from the platform's implementation, allowing platform teams to dynamically inject connection details (e.g., credentials, endpoints) into application pods. This declarative approach, combined with developer portals like Backstage, allows for "golden paths" where developers can bootstrap new projects with pre-configured local (Testcontainers, Micros) and production (Service Binding API) environments, drastically simplifying the entire development-to-production lifecycle.

Demo / Proof of Concept

▶ Watch: Demo: Deploy to local Kubernetes via Podman Desktop (5:00)

The talk features several compelling demonstrations and proofs of concept, illustrating the practical application of the discussed tools and methodologies:

1. Local Containerization and Deployment with Podman Desktop:

• Buildpacks in action: Thomas demonstrates a Java Spring Boot application. Using the bootBuildImage Gradle task (which leverages Buildpacks), he builds a production-grade container image without a Dockerfile.
• Running locally: The built image is then run locally using podman run, exposing it on port 8080. A curl request confirms the application is serving traffic.
• Deploying to local Kubernetes: Thomas shows how Podman Desktop can provision a local Kubernetes cluster (using kind). From the Podman Desktop UI, he selects the locally running container and chooses "Deploy to Kubernetes." Podman Desktop automatically generates the necessary Kubernetes YAML (Deployment, Service) and deploys the application, making it accessible through the cluster's ingress. A Kubernetes dashboard within Podman Desktop allows monitoring the deployed pod.

1. Automated Kubernetes Manifest Generation with Quarkus:

• Adding extensions: Kevin showcases a Java Quarkus application. He adds quarkus-kubernetes and quarkus-kubernetes-config extensions to the project.
• YAML generation: Upon building the application, Quarkus automatically generates a kubernetes.yaml file in the target/kubernetes directory. This YAML includes a Service Account, Role Binding, Deployment, and Service.
• Configuring resources and probes: Kevin then demonstrates how to define resource requests/limits and configure health endpoints (for database and Kafka dependencies) directly in application.properties. Adding the quarkus-smallrye-health extension automatically configures sophisticated liveness and readiness probes within the generated YAML, ensuring Kubernetes correctly manages the application's health and readiness for traffic.
• Deployment: The demo concludes by showing how quarkus deploy command can build the image, push it to a registry, and apply the generated Kubernetes YAML to a cluster, confirming that resource limits and health probes are correctly configured.

1. Serverless Functions with Knative:

• Creating a Go function: Kevin diversifies the demo by creating a Go application using kn func create.
• Building and deploying: The kn func build command builds the function into a container image (again, leveraging Buildpacks). Then, kn func deploy pushes the image to a registry and deploys it as a serverless function on an OpenShift cluster (running Knative).
• Auto-scaling to zero: After invoking the function with kn func invoke and receiving a response, the demo illustrates Knative's auto-scaling capability. After a period of inactivity, the pod scales down to zero, and then scales back up instantly when another invocation occurs, demonstrating the efficiency of serverless on Kubernetes.

1. Local Development with Testcontainers and Ollama:

• Integrating Ollama: Thomas presents a Java Spring Boot application that integrates with an AI model. Instead of relying on a remote service, the application is configured to use Testcontainers to provision an Ollama container locally when the application starts.
• Interacting with LLM: Running the application (via bootRun), the Ollama container is spun up. Thomas then sends an HTTP request to the application asking "what's the capital of England?". The application processes this via the locally running Ollama container, which provides an answer (though humorously incorrect: "London was established in 1962").
• Observing Testcontainers: The Podman Desktop UI confirms that the Ollama container is indeed running locally, provisioned by Testcontainers.

1. Local Observability with Testcontainers (OpenTelemetry & Grafana):

• Observability stack: Building on the previous demo, Thomas extends the Testcontainers setup to also provision an OpenTelemetry collector and Grafana locally, all without any manual configuration.
• Tracing LLM interaction: He sends the same "capital of England" request. Then, accessing the locally running Grafana dashboard (also provisioned by Testcontainers), he traces the request flow through the application. The trace clearly shows the interaction with the "small-llm2" model running in Ollama. This visualization helps diagnose the model's "hallucination," highlighting the importance of local observability for debugging AI applications.

1. Backstage for Project Bootstrapping:

• Golden Path: Thomas demonstrates a Backstage instance acting as a developer portal. He selects a "golden path" for an AI application.
• Configuring dependencies: Through the Backstage UI, he chooses an LLM provider (e.g., Mistral AI, Ollama) and a database type.
• Automated setup: When the project is bootstrapped, Backstage automatically configures the local development environment with Testcontainers and Micros settings for chosen dependencies and sets up Service Binding API configurations for the actual services when deployed to Kubernetes. This streamlines project setup and ensures consistency.

While Micros was introduced as a tool for API mocking, the talk did not include a specific step-by-step demonstration of its usage. Its functionality was described, and its integration into the broader development workflow (especially via Backstage) was implied.

Defensive Implications

▶ Watch: Why Podman deployment isn't production-grade; K8s YAML (6:05)

While the talk primarily focuses on developer experience and efficiency, many of the discussed tools and methodologies inherently contribute to a stronger defensive posture for applications deployed on Kubernetes:

1. Secure and Performant Image Builds with Buildpacks: Cloud Native Buildpacks are designed to produce secure, production-grade container images. By abstracting the Dockerfile process, they apply best practices for layering, dependency management, and minimal image size, reducing the attack surface. This "security by default" approach means developers don't have to be Dockerfile experts to create secure images, lowering the risk of misconfigurations that could introduce vulnerabilities.

1. Application Resilience through Kubernetes Probes: Properly configured readiness, liveness, and startup probes are fundamental for application resilience. They allow Kubernetes to accurately determine an application's health and availability. This prevents traffic from being routed to unready instances (reducing downtime), automatically restarts unhealthy instances (improving self-healing capabilities), and handles slow startups gracefully. From a defensive standpoint, this ensures the application remains operational and responsive even under stress or transient failures, mitigating denial-of-service scenarios caused by internal application issues.

1. Resource Management and Cluster Stability: Defining resource requests and resource limits for CPU and memory is critical for cluster stability. By providing these, developers help platform teams ensure fair resource allocation and prevent individual applications from monopolizing cluster resources. This guards against "noisy neighbor" problems, where a misbehaving application could degrade the performance or stability of other services on the same node, which could be exploited or lead to cascading failures.

1. Standardized Configuration and Secrets Handling: Tools like Quarkus extensions and the Service Binding API standardize how configuration and secrets are managed. Quarkus can automatically generate YAML that integrates with Kubernetes ConfigMaps and Secrets. The Service Binding API provides a declarative, platform-managed way to bind applications to backing services, ensuring that sensitive connection details are injected securely and consistently, rather than being hardcoded or manually managed by developers. This reduces the risk of exposing credentials or misconfiguring access to critical services.

1. Early Observability for Proactive Defense: The demonstration of using Testcontainers to spin up a local OpenTelemetry and Grafana stack is a powerful defensive tool. By enabling full observability (tracing, metrics) in the development environment, developers can:

• Identify and fix issues early: Catch performance bottlenecks, unexpected behavior (like LLM hallucinations), and integration problems before they reach production.
• Understand system behavior: Gain deep insights into how their application interacts with dependencies, which is crucial for troubleshooting and identifying potential attack vectors or vulnerabilities.
• Improve debugging: Tracing requests through distributed systems locally helps pinpoint the root cause of issues, making applications more robust.

1. Reduced Cognitive Load and Error Prevention: By abstracting away complex Kubernetes YAML and offering opinionated "golden paths" through tools like Quarkus and Backstage, the cognitive load on developers is significantly reduced. Less manual YAML means fewer opportunities for human error, misconfigurations, or security oversights. A streamlined, automated process leads to more consistent and potentially more secure deployments.

1. Consistent Testing Environments with Testcontainers and Micros: Using Testcontainers for local dependencies and Micros for API mocking ensures that development and testing environments closely mirror production. This consistency helps uncover integration issues and potential vulnerabilities that might otherwise only surface in production, allowing for proactive remediation.

In essence, the "Kubernetes native development" approach, championed in this talk, fosters a culture of building resilient, observable, and well-managed applications from the ground up, significantly enhancing their defensive posture against both operational failures and potential security threats.

Key Takeaways

• Embrace Kubernetes Native Development: Shift from traditional "works on my machine" development to building applications specifically designed for Kubernetes, focusing on fast startup, low resource usage, and resilience.
• Automate Containerization and Deployment: Leverage tools like Cloud Native Buildpacks for simplified, secure, and performant image creation without Dockerfiles, and Podman Desktop for easy local container management and one-click deployment to local Kubernetes clusters.
• Reduce Kubernetes Cognitive Load for Developers: Utilize framework-specific extensions (e.g., Quarkus) to automatically generate comprehensive Kubernetes YAML, including intelligent probes and resource configurations, abstracting away YAML complexity.
• Enable Serverless on Kubernetes: Adopt Knative and Knative Functions to build and deploy auto-scaling, polyglot serverless workloads directly on Kubernetes, simplifying operational concerns for functions.
• Simplify Local Development with Testcontainers: Use Testcontainers to programmatically provision and manage application dependencies (databases, LLMs, observability stacks like OpenTelemetry/Grafana) locally, eliminating the need for a full local Kubernetes cluster for development and testing.
• Standardize Production Service Bindings: Employ the Kubernetes Service Binding API to declaratively connect applications to backing services in production, promoting separation of concerns and automated credential injection.
• Foster Developer Experience and Joy: Integrate these tools into cohesive workflows, potentially via developer portals like Backstage, to provide "golden paths" that reduce friction, accelerate feedback loops, and allow developers to focus on business logic.

About the Speaker(s)

Kevin Dubois is a Developer Advocate at Red Hat. With over 20 years of experience in software development, Kevin has navigated the transition from traditional development paradigms to the modern cloud-native world. His expertise lies in helping developers understand and adopt new technologies, particularly within the Kubernetes ecosystem.

Thomas Vitale works at Systematic, a software company. He is deeply passionate about all things cloud-native and Java-related. Thomas is the author of a book on cloud-native Java and is currently co-authoring a new book titled "Developer Experience on Kubernetes" with Mauricio Salatino, indicating his strong focus on improving the developer journey in Kubernetes environments.