Flux Ecosystem Evolution - Stefan Prodan, ControlPlane & Sanskar Jaiswal, Kong

Flux Ecosystem Evolution - Stefan Prodan, ControlPlane & Sanskar Jaiswal, Kong

Speakers: Stefan Prodan, ControlPlane; Sanskar Jaiswal, Kong

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=jikiO3CC7Zw

Overview

This talk by Stefan Prodan from ControlPlane and Sanskar Jaiswal from Kong delves into the significant advancements and new tools within the Flux ecosystem, a leading GitOps continuous delivery solution for Kubernetes. The presentation highlights not only the core Flux project but also its crucial companion, Flagger, for progressive delivery, and introduces the nascent yet powerful Flux Operator and the new Resource Set API. The speakers emphasize the ecosystem's evolution towards more robust, secure, and developer-friendly GitOps practices, addressing challenges like inconsistent user experiences during rollouts, secure Flux bootstrapping at scale, and facilitating ephemeral testing environments.

The session showcases how the Flux ecosystem is expanding beyond basic continuous delivery to encompass advanced deployment strategies, enhanced operational resilience, and streamlined developer workflows. Prodan and Jaiswal, who have collaborated on Flux and Flagger for years, demonstrate practical applications of these tools, including a novel approach to session affinity during canary deployments and the declarative management capabilities of the Flux Operator. Their work underscores a commitment to simplifying complex Kubernetes operations while maintaining high standards of security and reliability.

The importance of this talk lies in its comprehensive update on a critical cloud-native project that underpins many organizations' deployment pipelines. By introducing features like OCI artifact support for air-gapped environments, GitHub App authentication, and the declarative management of Flux itself, the speakers illustrate a clear path towards more mature, scalable, and secure GitOps implementations. For anyone managing Kubernetes clusters and striving for efficient, safe software delivery, understanding these evolutions is paramount to leveraging the full potential of the Flux ecosystem.

Background

▶ Watch: Introduction to Flux ecosystem and Flagger's role (0:00)

The foundation of this discussion rests on GitOps, an operational framework that takes DevOps best practices and applies them to infrastructure automation. At its core, GitOps uses Git as the single source of truth for declarative infrastructure and applications. Flux, as a continuous delivery tool, extends the Kubernetes API to automatically synchronize the desired state defined in Git repositories (or other external sources like container registries or S3 buckets) with the actual state of a cluster. This means any change committed to Git is automatically picked up and applied to the Kubernetes cluster, driving consistency and auditability.

While Flux efficiently deploys changes, directly applying every change to all users can introduce significant risks. This is where Flagger comes into play. Developed over seven years, Flagger is a standalone tool designed for progressive delivery, acting as a critical safeguard in production environments. It intercepts Kubernetes deployment events and implements advanced deployment strategies that the native Kubernetes deployment controller lacks, such as canary releases, A/B testing, and blue/green deployments. Flagger ensures that new application versions are gradually rolled out, monitored against predefined Service Level Objectives (SLOs), and automatically rolled back if performance degrades, protecting users from regressions.

The workflow typically involves Flux image automation detecting a new container image in a registry, updating the image tag in Git, and committing this change. Flux then applies this updated desired state to the cluster. If Flagger is configured, it takes over from the standard Kubernetes deployment controller. Instead of an immediate full rollout, Flagger orchestrates a gradual traffic shift to the new version, testing its stability with metrics from providers like Prometheus. Only after passing these checks does Flagger proceed with a full rollout. This disconnects continuous deployment from the actual release, allowing for guarded, progressive releases that minimize user impact from potential mistakes. This established synergy forms the bedrock upon which the latest ecosystem enhancements are built.

Key Findings

▶ Watch: Flagger's role in the progressive delivery workflow (4:10)

The talk reveals several key advancements across the Flux ecosystem, addressing both operational efficiency and user experience in GitOps.

Firstly, Flagger has significantly expanded its networking integration capabilities. A major highlight is the addition of full support for Gateway API, including specific implementations for AWS Gateway API. This unification simplifies how Flagger programs traffic across diverse ingress and service mesh solutions, reducing the need for bespoke integrations. Furthermore, Flagger now supports Knative, a serverless platform for Kubernetes, enabling progressive delivery for Knative services despite their unique deployment model.

A crucial problem identified and solved by Flagger is the inconsistent user experience during standard weighted traffic shifting in progressive delivery. When traffic is randomly distributed, users might be routed to an older version of an application, then a newer version, and then back to the older one. Flagger addresses this by introducing a novel deployment strategy that combines weighted routing with session affinity. By injecting a cookie name into responses, Flagger ensures that once a user interacts with a canary (new) version, they consistently stick to that version, providing a seamless experience.

The introduction of the Flux Operator marks a significant step towards declarative and scalable management of Flux itself. Initially driven by the demand from OpenShift users, the Flux Operator provides a higher-level abstraction, the FluxInstance API, for deploying and configuring Flux across potentially hundreds of clusters. This operator simplifies advanced configurations like sharding and multi-tenancy. A groundbreaking feature of the Flux Operator is its support for OCI artifacts as the desired state, decoupling Git from direct production dependencies and enabling GitOps in air-gapped environments. It also automates Flux upgrades, critical for maintaining security posture against CVEs, and introduces guardrails to prevent accidental self-destruction of the Flux system.

Building on the Flux Operator, the new Resource Set API enables self-service ephemeral environments. This API allows users to define high-level rules, such as creating a HelmRelease in a dedicated namespace for every pull request with a specific label, using the Git SHA as the image tag. This empowers developers to test code and configuration changes in isolated, production-like environments before merging, significantly reducing the risk of introducing issues to main branches.

Finally, Flux 2.5, the latest core Flux release, introduces important security and extensibility features. These include GitHub App authentication, moving away from less secure personal access tokens (PATs). It also adds the ability to define custom health checks using Common Expression Language (CEL), providing powerful and flexible dependency management for complex deployments. Furthermore, OCI reconciliation notifications have been completed, ensuring that Flux can report the successful reconciliation of desired states even when sourced from OCI registries.

Technical Deep Dive

▶ Watch: Introducing Flux Operator for CI and ephemeral environments (5:50)

The innovations presented span across Flagger, Flux Operator, and the core Flux project, each offering sophisticated technical solutions to common GitOps challenges.

Flagger's Session Affinity

The core technical challenge addressed by Flagger's new session affinity feature is ensuring a consistent user experience during progressive delivery. Traditional weighted routing randomly distributes traffic, leading to users oscillating between old and new application versions. Flagger solves this by introducing a cookie-based session affinity mechanism.

When configuring a Canary Custom Resource, users can now specify a cookieName. Flagger then takes over the traffic programming. For each new canary session, Flagger generates a unique cookie value. When a user's request is routed to the canary workload, Flagger injects this specific cookie into the response. Subsequent requests from that user, possessing this cookie, are then consistently directed to the canary, even if the overall traffic distribution still heavily favors the primary workload. This "stickiness" is maintained throughout the canary analysis. The talk also mentions that Flagger creates a separate cookie for the primary workload, allowing for fine-grained load balancing between the two.

This feature is implemented to work seamlessly with various networking technologies. It explicitly supports Gateway API (including AWS Gateway API) and Istio Virtual Services. The speakers highlighted the significance of Gateway API, as it provides a unified abstraction for defining networking rules, simplifying Flagger's integration with future service meshes or ingress controllers without requiring extensive modifications to Flagger itself. This positions Flagger as a "traffic programmer" that leverages standardized APIs for greater interoperability.

Flux Operator

The Flux Operator fundamentally changes how Flux is deployed and managed, especially at scale. It provides a declarative, Kubernetes-native way to bootstrap and configure Flux, moving beyond the Flux CLI or Terraform provider. The central abstraction is the FluxInstance API, which allows users to define the desired state of their Flux installation using a concise Kubernetes manifest, replacing potentially thousands of lines of YAML. This API streamlines advanced configurations such as sharding, multi-tenancy, and persistent storage for Flux's internal artifacts.

A standout technical innovation is the support for OCI artifacts as the desired state for GitOps. This decouples the runtime dependency on a Git repository for continuous delivery. Instead of Flux directly pulling from Git, a CI pipeline can use the flux push command (part of the Flux CLI) to package the Git repository's state (Kubernetes manifests, Helm charts, etc.) into an OCI image and push it to a container registry. Flux then reconciles from this OCI image. Benefits include:

• Resilience: The system can continue operating and even perform rollbacks if the Git server is temporarily unavailable.
• Security: OCI artifacts can be signed, allowing Flux to verify their authenticity.
• Air-gapped environments: This is critical for highly regulated or disconnected environments where direct Git access is not feasible. The OCI registry acts as the trusted, air-gapped source of truth.

The Flux Operator also automates Flux upgrades. By specifying a SemVer expression (e.g., spec.distribution.version: ">=2.x <3.x") in the FluxInstance object, the operator automatically keeps Flux components updated to the latest secure versions. This is vital for security, as Flux runs with cluster-admin privileges and is often the first tool deployed, making it a critical attack surface if outdated. The operator also incorporates guardrails, making it "really hard" to accidentally delete or misconfigure the Flux system, mitigating the risk of "suicide" scenarios where a misconfigured Git commit could instruct Flux to delete itself.

Resource Set API

The Resource Set API, built on top of the Flux Operator, addresses the long-standing request for ephemeral environments for pre-merge testing. The motivation for a new API, rather than extending existing Flux APIs like HelmRelease, stems from the conceptual mismatch: HelmRelease reflects Helm's capabilities, while ephemeral environments often depend on higher-level concepts like pull requests (or GitLab's merge requests), which are specific to Git hosting platforms, not generic Git.

The ResourceSet API provides a flexible framework for defining self-service environments. A typical use case involves instructing the Flux Operator to "scan this repo, watch for pull requests that have a label, and for each such pull request, create a HelmRelease object in an existing namespace." The HelmRelease would then use the Git SHA of the pull request as the image tag, ensuring that both code changes (in the container image) and configuration changes (in the Helm chart) are deployed together.

The API is designed to be unopinionated, meaning it doesn't dictate how an ephemeral environment is provisioned. Users can configure it to:

• Create a HelmRelease per pull request.
• Create an entirely new namespace per pull request.
• Even provision a new cluster using Cluster API or Crossplane definitions within the ResourceSet to target the HelmRelease to that new cluster.

This flexibility allows organizations to tailor ephemeral environments to their specific needs, enabling comprehensive testing of changes before they are merged into the main branch.

Flux 2.5 Features

The latest core Flux release, Flux 2.5, introduces several impactful features:

• GitHub App authentication: This enhances security by allowing Flux to authenticate with GitHub using an application-specific token rather than a personal access token (PAT), which often carries broader permissions and is tied to an individual user. This is a significant improvement for enterprise security and compliance.
• Custom health checks with Common Expression Language (CEL): Flux now supports CEL for defining advanced health checks and dependency conditions. This is particularly useful for complex deployment scenarios involving Crossplane or Cluster API, where resources might need to be checked for readiness in specific sequences (e.g., ensure a cluster is provisioned before deploying applications onto it). CEL, already integrated into Kubernetes for policy agents, offers a powerful and flexible way to extend Flux's API behavior without adding hundreds of new fields.
• OCI reconciliation notifications: This completes the OCI integration, allowing Flux to provide feedback on the successful reconciliation of a desired state, even when that state was sourced from an OCI container registry. This ensures end-to-end visibility and traceability, connecting the reconciled state back to the original Git commit via metadata.

Demo / Proof of Concept

▶ Watch: Flagger's new support for Gateway API and KNative (6:40)

Sanskar Jaiswal provided a live demonstration of Flagger's new session affinity feature, showcasing its ability to provide a consistent user experience during a canary rollout.

The demo environment consisted of a Kubernetes cluster with Istio ingress gateway API already set up. A Canary object was defined, targeting a podinfo deployment. Crucially, this Canary definition included a cookieName specification, which Flagger would use to establish session stickiness.

Initially, the podinfo application was running version 6.0.0. Sanskar then triggered a canary rollout by updating the podinfo deployment to version 6.0.1. Behind the scenes, Flagger began its magic: scaling up the canary workload running 6.0.1 and configuring the HTTP route to gradually shift traffic. More importantly, it started injecting the specified cookies into responses.

To illustrate the session affinity, Sanskar opened two separate incognito browser tabs. When he loaded the application in both tabs, they were initially served by the primary workload (6.0.0). However, as Flagger started shifting a small percentage of traffic to the canary, both tabs eventually hit the canary workload (6.0.1). The key takeaway was that after this initial hit to the canary, subsequent reloads in both incognito tabs always directed traffic to the canary, even though the overall weighted traffic distribution (e.g., 90% primary, 10% canary) still favored the primary. This demonstrated that once a user (represented by an incognito tab with its unique session) experienced the new version, they would continue to experience it, fulfilling the goal of a consistent user experience during progressive delivery.

The demo clearly highlighted how Flagger automates the entire process, from scaling the canary to configuring the ingress with appropriate response headers and cookies, all based on a simple cookieName declaration in the Canary resource.

Defensive Implications

▶ Watch: Explaining Flagger's progressive delivery (canary) mechanism (8:10)

The advancements in the Flux ecosystem offer significant defensive implications, enhancing security, resilience, and operational safety for Kubernetes deployments.

Progressive Delivery with Flagger: Flagger's core function is a critical defensive measure. By enabling canary releases, A/B testing, and automatic rollbacks based on SLOs, it acts as a robust shield against introducing faulty code or configurations into production. The new session affinity feature further strengthens this by preventing user experience churn, which can lead to user frustration and increased support costs, indirectly bolstering user trust and system stability. By catching issues early and minimizing impact, Flagger directly reduces the blast radius of deployment failures.

Flux Operator for Secure and Resilient GitOps: The Flux Operator introduces several layers of defense for the GitOps control plane itself:

• Declarative Management and Standardization: By providing a standardized, declarative way to deploy and manage Flux via the FluxInstance API, organizations can ensure consistent and auditable deployments of their GitOps tooling across all clusters. This reduces configuration drift and human error.
• OCI Artifacts for Resilience and Air-Gapped Security: Decoupling Flux's runtime dependency on Git through OCI artifacts provides critical resilience against Git server outages. For highly regulated sectors or those operating in air-gapped environments, OCI support is a fundamental security requirement, enabling a secure supply chain for configurations where direct external access is prohibited. The ability to sign OCI artifacts further adds a layer of tamper detection.
• Automated Upgrades for CVE Mitigation: Flux runs as critical infrastructure with high privileges (often cluster-admin). The Flux Operator's automatic upgrade capability, driven by SemVer expressions, is paramount for security. It ensures that Flux components are consistently updated to the latest versions, promptly patching CVEs and addressing security vulnerabilities before they can be exploited. Delaying upgrades on such a critical component is a significant security risk.
• Guardrails Against Self-Destruction: The operator's design makes it "really hard" to accidentally misconfigure Flux to delete itself or the entire cluster. This "suicide prevention" mechanism is a crucial operational defense, particularly in automated, GitOps-driven environments where a single erroneous commit could have catastrophic consequences.

Resource Set API for Pre-Production Risk Reduction: The Resource Set API fosters a proactive defensive posture by enabling self-service ephemeral environments. By allowing developers to test new features, configuration changes, and even infrastructure modifications in isolated, production-like environments (e.g., per pull request), organizations significantly reduce the risk of introducing bugs or security vulnerabilities into main branches and, subsequently, production. This shifts defect detection left, making it cheaper and safer to remediate issues.

Flux 2.5 Security Enhancements:

• GitHub App Authentication: Moving from personal access tokens (PATs) to GitHub App authentication is a direct security upgrade. GitHub Apps operate with specific, granular permissions, reducing the attack surface compared to PATs which often have broader scopes and are tied to individual user accounts. This aligns with the principle of least privilege.
• CEL for Custom Health Checks: The integration of Common Expression Language (CEL) for health checks allows for more sophisticated and robust validation of deployed resources and their dependencies. This means that critical services can be checked with greater precision before being considered "ready," preventing applications from receiving traffic when underlying dependencies are not fully functional, thereby improving overall system stability and reliability.

In summary, the Flux ecosystem's evolution provides a comprehensive suite of tools and features that not only streamline operations but also embed robust defensive capabilities throughout the software delivery lifecycle, from pre-merge testing to resilient production deployments and continuous security patching.

Key Takeaways

• Flagger Enhances Progressive Delivery: Flagger continues to evolve as a critical tool for progressive delivery, adding support for modern networking APIs like Gateway API (including AWS Gateway API) and integrating with serverless platforms like Knative.
• Session Affinity Solves User Experience Issues: A significant innovation in Flagger is the introduction of cookie-based session affinity during canary deployments, ensuring a consistent user experience by "sticking" users to the new application version once they encounter it, preventing disruptive toggling between old and new UIs.
• Flux Operator for Scalable and Secure Flux Management: The Flux Operator provides a declarative, Kubernetes-native way to deploy and manage Flux at scale, simplifying complex configurations, automating upgrades for security, and introducing guardrails against accidental misconfigurations.
• OCI Artifacts Decouple Git and Enable Air-Gapped GitOps: The Flux Operator's support for OCI artifacts as the desired state offers crucial benefits for resilience against Git outages, enables secure GitOps in air-gapped environments, and allows for signed artifact verification.
• Resource Set API Empowers Ephemeral Environments: The new Resource Set API in Flux Operator facilitates the creation of self-service, on-demand ephemeral environments (e.g., per pull request), allowing for comprehensive testing of code and configuration changes before merging to production, significantly reducing deployment risk.
• Flux 2.5 Boosts Security and Extensibility: The latest core Flux release, Flux 2.5, introduces vital security enhancements like GitHub App authentication and powerful extensibility through Common Expression Language (CEL) for custom health checks, alongside improved OCI reconciliation notifications.

About the Speaker(s)

Stefan Prodan is affiliated with ControlPlane and is a prominent Flux maintainer. He has been deeply involved in the Flux project and its ecosystem for many years, including a four-year collaboration with Sanskar Jaiswal on both Flux and Flagger. His work focuses on advancing GitOps practices and building tools that extend Kubernetes' capabilities for continuous delivery.

Sanskar Jaiswal is associated with Kong and has been a key contributor to the Flagger and Flux projects. He has worked alongside Stefan Prodan for four years, with Flagger itself having been under development for nearly seven years. Sanskar's expertise lies in progressive delivery, traffic management, and integrating these capabilities with various networking technologies within the Kubernetes ecosystem.