Project Lightning Talk: external-secrets: Zero Trust Secrets Management with ESO - Moritz Johner

Project Lightning Talk: external-secrets: Zero Trust Secrets Management with ESO - Moritz Johner

Speakers: Moritz Johner, Maintainer and Original Creator of External Secrets Operator

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=9mX9PvNNDjk

Overview

In the dynamic landscape of cloud-native applications, managing sensitive data like API keys, database credentials, and certificates within Kubernetes clusters presents a persistent security challenge. Traditional methods often involve static credentials or manual processes, which are prone to misconfiguration, leakage, and lack of proper rotation. Moritz Johner, a maintainer and original creator of the external-secrets operator (ESO), addressed this critical issue in his KubeCon EU lightning talk, "Zero Trust Secrets Management with ESO."

This talk introduced ESO as a pivotal tool for bridging the gap between external, secure secret vaults and Kubernetes' native secret management. Johner elucidated how ESO enables organizations to adopt a zero-trust security model for their secrets, ensuring that workloads can access necessary credentials without ever exposing them directly or relying on static, long-lived access keys within the cluster. By automating the synchronization of secrets from enterprise-grade vaults into Kubernetes as native Secret objects, ESO significantly enhances both security posture and operational efficiency.

The core premise of ESO is built on the assumption that engineering teams centralize their secrets in a robust, external vault solution. ESO then acts as a secure intermediary, running as a Kubernetes workload itself, to fetch these secrets and make them consumable by other cluster components. This approach eliminates the need for applications to directly interact with external vaults, simplifying access patterns while maintaining a strong security perimeter. Johner's presentation highlighted ESO's architecture, its extensive provider support, and its suite of features designed to streamline secret lifecycle management in multi-tenant Kubernetes environments.

Background

▶ Watch: Introduction to external-secrets and core assumption (0:00)

The proliferation of microservices and containerized applications, particularly within Kubernetes, has amplified the complexity of secret management. Every application component often requires access to various secrets—database passwords, API tokens, TLS certificates, and more. Historically, developers might embed these secrets directly into application code, configuration files, or use Kubernetes Secret objects populated manually or via CI/CD pipelines with static credentials. These practices introduce significant security risks: hardcoded secrets are difficult to rotate, prone to accidental exposure in version control, and lack granular access control. Kubernetes Secret objects, while a step forward, are merely base64 encoded by default and not encrypted at rest without additional configuration, nor do they inherently provide mechanisms for dynamic rotation or integration with external vaults.

The industry has largely converged on the necessity of using dedicated, secure secret vaults (such as AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or HashiCorp Vault) for storing sensitive data. These vaults offer robust encryption, auditing, access control, and secret rotation capabilities. However, integrating these external vaults directly with Kubernetes workloads presents its own set of challenges. Applications would typically need to include vault client libraries, manage authentication tokens, and handle secret retrieval logic, adding complexity and increasing the attack surface. Furthermore, granting Kubernetes pods direct, privileged access to a central vault requires careful credential management for the pods themselves, often leading back to the problem of static credentials for vault access.

This is the problem space that the external-secrets operator (ESO) was designed to address. ESO operates on the fundamental assumption that an organization already leverages a secure, centralized vault for all its secrets. Its purpose is not to be a vault, but to act as a secure, Kubernetes-native bridge to these vaults. ESO eliminates the need for individual applications to possess vault-specific credentials or logic, instead providing a mechanism for the cluster itself to securely fetch and manage secrets on behalf of its workloads. This architecture shifts the burden of vault integration from application developers to the platform layer, enabling a more secure, consistent, and manageable approach to secrets in Kubernetes.

Key Findings

▶ Watch: Zero-trust authentication with Kubernetes service accounts (0:50)

The external-secrets operator (ESO) introduces a robust and flexible framework for zero-trust secrets management within Kubernetes, addressing the inherent challenges of integrating external secret vaults. Moritz Johner's talk underscored several key findings and contributions that make ESO a compelling solution:

Firstly, ESO operates as a Kubernetes operator, meaning it extends the Kubernetes API with custom resources and continuously reconciles the desired state with the actual state. Its core function is to run as a workload inside the cluster, reach out to an external secret vault, fetch specified secrets, and then create or update a standard Kubernetes Secret object with that data. This approach allows existing Kubernetes workloads to consume secrets natively, either by referencing them from an Ingress resource, mounting them as files in a pod, or consuming them as environment variables, without any modification to their code or requiring direct vault integration.

A pivotal finding highlighted by Johner is ESO's implementation of zero-trust authentication with external vaults. Instead of relying on static credentials (e.g., API keys, username/password pairs) stored within Kubernetes—which themselves become secrets to manage—ESO leverages Kubernetes service accounts to authenticate with the chosen secret provider. This is achieved by exchanging a JSON Web Token (JWT), associated with the service account, directly with the vault provider. This mechanism ensures that the operator, and by extension the SecretStore custom resource, does not possess any long-lived, static credentials, significantly reducing the attack surface. This capability is widely supported by modern secret providers, making ESO highly secure and adaptable.

The operator boasts impressive compatibility, supporting "about like 30ish providers" as of the talk, including major cloud offerings like AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and self-hosted solutions like HashiCorp Vault. This extensive support ensures that most organizations can integrate ESO with their existing secret infrastructure.

ESO introduces two primary Custom Resource Definitions (CRDs) to manage secrets:

• SecretStore: This resource defines how ESO authenticates with and connects to a specific external secret provider. It encapsulates the provider type, region (if applicable), and the authentication method (e.g., a Kubernetes service account).
• ExternalSecret: This resource specifies which secret to fetch from the configured SecretStore and how it should be materialized as a Kubernetes Secret object within the cluster. It allows for mapping secret keys, defining target names, and specifying refresh intervals.

Crucially, ESO performs reconciliation on a regular basis (e.g., hourly, every 10 minutes, or even every minute). This continuous synchronization ensures that any changes to the secret in the external vault are promptly reflected in the corresponding Kubernetes Secret, facilitating dynamic secret rotation and maintaining consistency.

Beyond its core synchronization capabilities, ESO offers a rich set of additional features:

• Secret Rotation: While the vault itself manages the actual secret rotation, ESO's reconciliation loop ensures that updated versions are propagated to Kubernetes, allowing workloads to gracefully handle new credentials.
• Secret Distribution: ESO enables secure distribution of secrets across multiple namespaces within a cluster or even between different Kubernetes clusters, supporting platform-as-a-service models.
• Templating: The ability to template secrets allows for flexible transformation and formatting of fetched data before it becomes a Kubernetes Secret.
• Aggregation and Extraction: ESO can fetch, aggregate, and extract specific values from structured secrets (e.g., JSON objects) stored in the vault.
• Multi-tenancy: The entire system is built with multi-tenancy in mind, allowing platform teams to provide secure secret management as a service to various engineering teams and namespaces.

These findings collectively demonstrate ESO's comprehensive approach to modern secret management, transforming a complex security challenge into a streamlined, automated, and secure process within Kubernetes.

Technical Deep Dive

▶ Watch: Key features: rotation, distribution, templating, multi-tenancy (2:10)

The external-secrets operator (ESO) fundamentally transforms how Kubernetes workloads access sensitive data by acting as a secure, automated intermediary between internal cluster resources and external secret vaults. Its architecture is designed for both security and operational efficiency, leveraging Kubernetes-native constructs to achieve a zero-trust model.

At its core, ESO runs as a standard workload inside your Kubernetes cluster. This operator, once deployed, continuously monitors Custom Resources (CRs) that define desired secret states. When an ExternalSecret resource is created or updated, the operator springs into action. Its first step is to communicate with the external secret vault. This communication is facilitated by a SecretStore resource, which acts as a blueprint for connecting to a specific provider.

The most critical technical aspect of ESO is its zero-trust authentication mechanism. Instead of requiring users to embed static API keys or long-lived credentials within the cluster for vault access, ESO leverages Kubernetes service accounts. Each SecretStore can be configured to use a specific service account within a particular namespace. When ESO needs to authenticate with an external vault (e.g., AWS Secrets Manager, HashiCorp Vault), it obtains a JSON Web Token (JWT) associated with that service account. This JWT is then presented to the external vault provider for authentication. The vault, configured to trust the Kubernetes API server (or an OIDC issuer associated with it), validates the JWT and grants temporary, scoped access based on the service account's permissions. This eliminates the need to store any static, high-privilege credentials within the Kubernetes cluster for vault access, drastically reducing the risk of credential compromise. For example, in the demo, a service account in the backend namespace was used to authenticate with AWS Secrets Manager.

The interaction flow is as follows:

1. Deployment: The ESO controller is deployed into the Kubernetes cluster, often in a dedicated control plane namespace.
2. SecretStore Definition: An administrator or platform team defines a SecretStore resource. This resource specifies the external secret provider (e.g., provider: aws: region: us-east-1), and crucially, the authentication method. For cloud providers, this typically involves specifying auth: serviceAccountRef: name: <service-account-name>. This service account must have appropriate Identity and Access Management (IAM) roles (for AWS) or equivalent permissions configured in the external vault to read secrets.
3. ExternalSecret Definition: Developers or application teams define ExternalSecret resources in their respective namespaces. This resource references a SecretStore and dictates which specific secret (or parts of a secret) to fetch from the external vault and how it should be mapped to a Kubernetes Secret. For instance, an ExternalSecret might specify secretStoreRef: name: aws-secrets-manager, then data: - secretKey: dbUser remoteRef: key: myapp/db-creds property: username.
4. Reconciliation Loop: ESO continuously monitors ExternalSecret resources. Upon detection of a new or modified ExternalSecret, or at a defined interval (e.g., refreshInterval: 1h), the operator performs the following steps:

• Authenticates with the external vault using the service account's JWT as defined in the referenced SecretStore.
• Fetches the specified secret data from the vault.
• Creates or updates a standard Kubernetes Secret object in the same namespace as the ExternalSecret. The data within this Kubernetes Secret is base64 encoded, as is standard for Kubernetes Secrets.
• Updates the status of the ExternalSecret to reflect its synchronization state (e.g., Ready, Synchronized).

This reconciliation loop is vital for maintaining secret rotation. While the external vault handles the actual rotation of the secret itself, ESO ensures that the updated secret versions are promptly pulled into Kubernetes. Workloads consuming these secrets can then be configured to restart or reload when the underlying Kubernetes Secret changes, effectively implementing secure, automated secret rotation without application-level logic.

Furthermore, ESO offers advanced capabilities like secret distribution across namespaces, allowing a single SecretStore to serve multiple ExternalSecret instances in different namespaces, potentially with different access policies. Templating allows for sophisticated transformations of secret values, enabling the creation of complex configuration files or certificates from simple vault entries. The ability to fetch, aggregate, and extract secrets from structured data (e.g., a single JSON secret in a vault containing multiple key-value pairs) provides immense flexibility for diverse application requirements. All these features are built with multi-tenancy in mind, empowering platform teams to offer robust, self-service secret management to development teams while maintaining central control and security standards.

Demo / Proof of Concept

▶ Watch: Demo: Configuring secrets from AWS Secrets Manager (3:00)

Moritz Johner presented a concise yet effective pre-recorded demonstration of the external-secrets operator (ESO) in action, showcasing its core functionality using AWS Secrets Manager. The demo aimed to illustrate how ESO securely fetches secrets from an external vault and materializes them as native Kubernetes Secret objects without relying on static credentials within the cluster.

The scenario began with a pre-existing secret stored in AWS Secrets Manager. This secret contained two key-value pairs: DB user and DB password. The specific password value used in the demo was "mingle mingle," a simple but illustrative example.

The demonstration proceeded through the following key steps:

1. SecretStore Configuration: Johner first highlighted a SecretStore resource. This resource was configured to connect to AWS Secrets Manager in the us-east-1 region. Crucially, the authentication method specified was a Kubernetes service account named backend-sa residing in the backend namespace. The speaker emphasized that "There are no static secrets there, no static credentials. We're just using the JSON web token," reinforcing the zero-trust authentication model. This SecretStore defines how ESO connects to AWS.

1. ExternalSecret Creation: Next, an ExternalSecret object was defined and applied to the Kubernetes cluster. This resource specified which secret to fetch from AWS Secrets Manager (by its name, implicitly defined by the SecretStore and the ExternalSecret's configuration) and how its components (DB user and DB password) should be mapped into a Kubernetes Secret. Applying this ExternalSecret resource triggers the ESO controller.

1. Status Verification: After applying the ExternalSecret, the demo showed checking its status. The output of kubectl get externalsecret confirmed that the resource was Ready and Synchronized. This indicates that ESO successfully authenticated with AWS Secrets Manager, fetched the secret data, and created the corresponding Kubernetes Secret.

1. Kubernetes Secret Inspection: Finally, the demo revealed the resulting Kubernetes Secret object. As expected, it contained the DB user and DB password keys, with their values base64 encoded. To prove the integrity of the fetched data, the speaker then demonstrated decoding the base64-encoded password. The decoded output clearly showed "mingle mingle," verifying that the secret was correctly retrieved from AWS Secrets Manager and made available within the Kubernetes cluster.

The simplicity and directness of the demo effectively highlighted ESO's value proposition: securing and automating the flow of secrets from an external vault into Kubernetes, making them accessible to workloads in a familiar, native format, all while adhering to robust zero-trust principles for authentication. The entire process was seamless, requiring only Kubernetes manifests and no manual intervention for secret fetching or credential management.

Defensive Implications

▶ Watch: Join the external-secrets community and project booth (4:15)

The external-secrets operator (ESO) introduces several profound defensive implications for organizations operating Kubernetes clusters, significantly enhancing their overall security posture regarding sensitive data.

Firstly, and most critically, ESO facilitates the elimination of static credentials within Kubernetes for accessing external secret vaults. By leveraging Kubernetes service accounts and JSON Web Tokens (JWTs) for authentication, the operator prevents the need to store long-lived API keys, access tokens, or other sensitive credentials directly within the cluster's configuration or as Kubernetes Secret objects themselves, which would then require their own management and protection. This drastically reduces the attack surface by removing a common vector for credential theft and abuse. An attacker gaining access to the cluster would not immediately find vault access credentials, as the authentication is dynamic and tied to the identity of the ESO operator's service account, rather than static secrets.

Secondly, ESO promotes centralized secret management within dedicated, secure vaults while enabling decentralized, Kubernetes-native consumption. This separation of concerns means that security teams can enforce robust policies, auditing, and lifecycle management (including mandatory rotation) at the vault level, which is purpose-built for such tasks. Application developers, on the other hand, consume secrets through familiar Kubernetes Secret objects, simplifying their workflow without needing to interact directly with the vault or embed vault client libraries. This consistency minimizes the chances of developers bypassing security best practices due to complexity.

Thirdly, the use of service accounts for authentication allows for the enforcement of least privilege. The Kubernetes service account assigned to the SecretStore can be granted only the precise permissions required to read specific secrets or secret paths from the external vault. This fine-grained access control ensures that even if the ESO operator or its service account were compromised, the scope of potential damage would be limited to the secrets it is explicitly authorized to access, rather than the entire vault.

Fourthly, ESO's continuous reconciliation loop provides automated secret hygiene. By periodically fetching updated secrets from the vault, ESO ensures that Kubernetes Secret objects reflect the latest versions. This is crucial for implementing automated secret rotation, a fundamental security practice. When a secret is rotated in the external vault, ESO propagates that change to Kubernetes, allowing applications to consume the new credentials without manual intervention or downtime (assuming applications are designed to gracefully handle secret changes). This automation reduces the risk of using stale or compromised secrets for extended periods.

Fifthly, the integration with external vaults inherently leverages the auditing capabilities of those vaults. Every access request made by ESO to fetch a secret is logged by the external vault, providing a clear audit trail of who (or what, in this case, the ESO service account) accessed which secret and when. This enhanced visibility is invaluable for security monitoring, compliance, and forensic analysis.

Finally, for multi-tenant Kubernetes environments, ESO offers a secure platform service. Platform teams can configure SecretStore resources to define secure connections to vaults, and then delegate ExternalSecret creation to individual development teams within their namespaces. This empowers development teams to manage their application's secrets independently, but always within the guardrails and security policies established by the platform team and enforced by ESO, preventing ad-hoc, insecure secret management practices.

While ESO significantly enhances security, defenders must also consider securing the ESO operator itself. This includes ensuring the operator runs with appropriate resource limits, its service account adheres to least privilege, and its logs are monitored. However, by abstracting direct vault access and enabling zero-trust authentication, ESO provides a powerful defense-in-depth mechanism that elevates the overall security posture of Kubernetes deployments.

Key Takeaways

• Zero-Trust Secret Management: ESO enables a zero-trust approach to secrets in Kubernetes by bridging external, secure secret vaults (like AWS Secrets Manager, HashiCorp Vault) to native Kubernetes Secret objects.
• Service Account Authentication: It leverages Kubernetes service accounts and JSON Web Tokens (JWTs) for authenticating with external vaults, eliminating the need for static credentials within the cluster and significantly reducing the attack surface.
• Extensive Provider Support: ESO supports approximately 30 different secret providers, including major cloud services (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) and self-hosted solutions, making it highly adaptable to diverse enterprise environments.
• Kubernetes-Native Integration: Secrets are defined via SecretStore and ExternalSecret Custom Resources, and then materialized as standard Kubernetes Secret objects, allowing existing workloads to consume them without modification.
• Automated Lifecycle Management: ESO continuously reconciles secrets, ensuring that changes and rotations in the external vault are automatically propagated to Kubernetes, enhancing secret hygiene and operational efficiency.
• Multi-Tenancy and Advanced Features: Designed with multi-tenancy in mind, ESO offers features like cross-namespace secret distribution, templating, and extraction from structured data, empowering platform teams to provide secure secret management as a service.

About the Speaker(s)

Moritz Johner is a prominent figure in the Kubernetes ecosystem, known as one of the maintainers and original creators of the external-secrets operator. His work focuses on enhancing the security and operational efficiency of secret management within cloud-native environments. Through his contributions to ESO, Johner has played a key role in developing a solution that bridges the gap between enterprise-grade secret vaults and Kubernetes, promoting zero-trust principles for sensitive data handling. His expertise lies in building robust, scalable, and secure infrastructure tools for modern containerized applications.