etcd V3.6.0 and etc... Benjamin Wang, Ivan Valdes Castillo, Siyuan Zhang, Arka Saha & Ciprian Hacman

etcd V3.6.0 and etc... Benjamin Wang, Ivan Valdes Castillo, Siyuan Zhang, Arka Saha & Ciprian Hacman

Speakers: Benjamin Wang, Ivan Valdes Castillo, Siyuan Zhang, Arka Saha & Ciprian Hacman

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=_xoDbpm-Qks

Overview

This talk provides a comprehensive update on the long-awaited etcd 3.6.0 release and introduces the nascent etcd operator 0.1.0. Presented by a diverse team of etcd maintainers and contributors, including Benjamin Wang, Ivan Valdes Castillo, Siyuan Zhang, Arka Saha, and Ciprian Hacman, the session delves into the significant new features, critical bug fixes, and substantial performance improvements that define etcd 3.6.0. It also highlights the community's renewed efforts to standardize etcd cluster management within Kubernetes through a dedicated operator.

The release of etcd 3.6.0 marks a pivotal moment, being the first major update in nearly four years since 3.5.0. This extended development cycle underscores the complexity and critical importance of etcd as the foundational distributed key-value store for Kubernetes clusters. The talk meticulously unpacks the technical advancements, such as the full migration to the V3 store, robust downgrade capabilities, and Kubernetes-style feature gates, which collectively aim to enhance the stability, operability, and developer experience of etcd.

Crucially, the speakers address a significant upgrade issue discovered during the 3.6.0 release candidate phase, detailing its root cause and the mandatory pre-upgrade steps users must take. Beyond core etcd, the session introduces the community-driven etcd operator, a project designed to streamline the deployment, management, and lifecycle of etcd clusters in Kubernetes environments, promising a more consistent and robust operational experience for the broader ecosystem.

Background

▶ Watch: Introduction to etcd 3.6.0 and operator 0.1 (0:00)

etcd serves as the distributed brain for Kubernetes, storing all cluster state, configurations, and metadata. Its stability, performance, and operational ease are paramount to the health and reliability of any Kubernetes deployment. For years, etcd 3.5.0 had been the prevailing stable release, leading to a prolonged period of anticipation for a new major version. This extended timeline brought several challenges and highlighted areas ripe for improvement.

Prior to 3.6.0, etcd grappled with a legacy V2 store data format, which, while deprecated since release 3.4, remained the source of truth for metadata in 3.5. This dual-store architecture (V2 for metadata, V3 for membership) introduced complexities and potential inconsistencies, particularly during upgrades. The absence of robust, officially supported downgrade mechanisms meant that operators faced significant risks when attempting to roll back etcd versions, often necessitating complex backup and restore procedures. Furthermore, the existing approach to feature development, relying on experimental flags, proved difficult to track and manage, leading to abandoned features and breaking changes when flags graduated.

The operational landscape for etcd within Kubernetes was also fragmented. While numerous third-party etcd operators existed, there was no single, community-standardized solution. This led to varied quality, maintenance levels, and inconsistent user experiences, making it challenging for organizations to adopt a reliable, long-term strategy for managing etcd. Performance, particularly concerning memory consumption, had also been a recurring complaint from users, driven by default configurations and internal mechanisms that could lead to excessive resource usage under certain workloads. Finally, the release process itself was largely manual, contributing to delays and making it harder to rapidly address security vulnerabilities or integrate new features. These collective issues underscored the urgent need for a comprehensive update and a more mature approach to etcd's development and operational management.

Key Findings

▶ Watch: Feature: Migration to W3 store (replacing W2) (3:00)

The etcd 3.6.0 release, despite a delayed general availability, represents a monumental step forward, addressing several long-standing architectural and operational challenges. A core finding is the ongoing and critical migration to the V3 store as the definitive source of truth, with the legacy V2 store being completely removed in 3.6.0. This move simplifies the data architecture and improves consistency.

A significant new capability is the full support for downgrades, allowing users to revert to the immediate prior minor version (e.g., 3.6 to 3.5) through a well-defined, two-stage process. This drastically reduces operational risk. The project also adopted Kubernetes-style feature gates, moving away from experimental flags to a more structured alpha, beta, and GA lifecycle, promising better feature tracking and fewer breaking changes.

For improved observability and integration with container orchestration platforms, etcd 3.6.0 introduces granular liveZ and readyZ health check endpoints, aligning with Kubernetes API expectations. The talk also highlighted the new V3 discovery protocol, which leverages the etcd client SDK 3 for cluster bootstrapping, deprecating the older V2 client-based discovery and the unmaintained discovery.io service.

A critical finding, and the primary reason for the 3.6.0 GA delay, was an upgrade issue from etcd 3.5 to 3.6, manifesting as "too many leaders." This bug, introduced in etcd 3.5.1 and affecting versions up to 3.5.19, stemmed from leader promotion only being persisted in the V2 store, not the V3 store. The fix was released in etcd 3.5.20, making it a mandatory pre-upgrade step for all users.

On the performance front, etcd 3.6.0 demonstrates consistently better performance compared to 3.5.20/21 across various read/write ratios and connection counts. Notably, it shows significantly reduced RAM consumption, primarily due to a default snapshot-count change from 100,000 to 10,000 and other memory optimization efforts. The team also reported advancements in automating the etcd release process, aiming for faster, more secure, and less error-prone releases.

Finally, a major new initiative is the formation of the etcd operator working group and the release of etcd operator 0.1.0. This first community-driven operator aims to provide a standardized, modern solution for managing etcd clusters within Kubernetes, starting with basic cluster creation and customization, with an ambitious roadmap for future enhancements like TLS, advanced upgrades, and disaster recovery.

Technical Deep Dive

▶ Watch: Feature: Full support for downgrades (3.6 to 3.5) (4:00)

The etcd 3.6.0 release introduces several fundamental technical shifts and refinements designed to enhance stability, performance, and operability.

V2 to V3 Store Migration

A cornerstone of etcd 3.6.0 is the complete deprecation and removal of the legacy V2 store. The V2 store, identifiable by its .snap suffix, had been marked deprecated since etcd 3.4 but persisted as the source of truth for metadata in etcd 3.5. In 3.5, while the V3 store (the bolt database file) became the source of truth for membership data, the V2 store still played a critical role during bootstrap. With 3.6.0, the enable-v2 flag has been entirely removed, making it impossible to enable the V2 store. The migration effort is ongoing, with 3.6.0 still bootstrapping etcd on the V2 store and then replaying the WAL (Write-Ahead Log) records onto the latest V2 snapshot. The ambitious goal for etcd 3.7 is to fully bootstrap etcd directly on the V3 store and replay WAL records starting from a consistent index, completely eliminating the V2 store dependency. This architectural simplification is expected to improve data consistency and reduce operational complexity.

Downgrade Support

etcd 3.6.0 is the first version to fully support downgrades, a crucial feature for disaster recovery and flexible cluster management. The downgrade process is structured into two main stages:

1. Data Schema Migration: The etcd cluster's data schema is migrated to be compatible with the target downgrade version.
2. Rolling Binary Replacement: Each etcd member's binary or image is replaced with the target version, one by one.

From a user perspective, this involves three distinct steps:

1. Validation: Users must validate the downgrade using the etcdctl downgrade validate command or the client SDK API. Crucially, only one minor version downgrade is supported at a time (e.g., 3.6 to 3.5 is allowed, but 3.6 to 3.4 is not).
2. Enable Downgrade: Once validated, etcdctl downgrade enable is executed. etcd automatically migrates the schema. Operators must wait until the storage version is confirmed to have changed to the target version (e.g., 3.5 in the example provided).
3. Binary Replacement: After schema migration, the etcd binary or container image for each member is replaced sequentially.

Kubernetes-Style Feature Gates

To streamline feature development and management, etcd 3.6.0 fully adopts Kubernetes-style feature gates. Previously, new features were introduced with an experimental- prefix flag, which had to be removed upon graduation, creating a breaking change and making feature tracking difficult. The new system allows features to progress through Alpha, Beta, and GA (General Availability) stages, with users enabling or disabling features via a unified feature-gates flag. All existing experimental flags have been migrated to this new system. This change is accompanied by etcd adopting the Kubernetes Enhancement Proposal (KEP) process for new feature development, promising greater transparency, less risk of breaking changes, and a faster development cycle.

Granular Health Check Endpoints

etcd 3.6.0 introduces two new health check endpoints: liveZ and readyZ. Before this, etcd 3.4 and 3.5 only had a single health endpoint, which couldn't distinguish between a process that was alive but not ready to serve traffic and one that needed a restart.

• The liveZ endpoint indicates whether the etcd process is still alive and does not require a restart.
• The readyZ endpoint reflects whether the process is ready to serve client traffic.

These endpoints make etcd fully compliant with Kubernetes API expectations for liveness and readiness probes, enabling more sophisticated and robust health monitoring in containerized environments.

V3 Discovery Protocol

Cluster bootstrapping with discovery services has been updated with the new V3 discovery protocol. This protocol utilizes the etcd client SDK 3 to communicate with the discovery service. The older V2 discovery, based on etcd client V2, is now deprecated. Furthermore, the public discovery.io service, which was commonly used for V2 discovery, is no longer maintained. This shift ensures that cluster bootstrapping leverages modern etcd client capabilities and aligns with the V3 API.

Critical Upgrade Issue (3.5 to 3.6)

A significant hurdle in the 3.6.0 release cycle was the discovery of a critical upgrade issue. When attempting to upgrade from etcd 3.5 to 3.6, clusters could fail, often manifesting as "too many leaders." The root cause was a bug introduced in etcd 3.5.1 where leader promotion was only persisted in the V2 store and not correctly mirrored to the V3 store. Since etcd 3.5 used the V2 store as the source of truth for metadata, this went unnoticed until the 3.6.0 upgrade, where the V3 store becomes the primary truth. This bug affected all patch versions from 3.5.1 up to 3.5.19. The fix was implemented in etcd 3.5.20. Consequently, a mandatory prerequisite for upgrading to etcd 3.6.0 is that users must first upgrade their etcd 3.5 clusters to version 3.5.20 or a higher patch release.

Performance Improvements

etcd 3.6.0 delivers substantial performance enhancements, particularly in memory usage. Comparative heatmaps and power line charts between 3.5 and 3.6 release candidate 3 consistently show 3.6 outperforming 3.5 in both read and write operations across various value sizes and connection counts.

• RAM Usage: There's a notable reduction in memory consumption in 3.6.0. This is largely attributed to a change in the default snapshot-count from 100,000 in 3.5 to 10,000 in 3.6. A pull request by Marik (PR 18825) also contributed by reducing the number of raft snapshots held in memory, leading to more frequent compaction.
• CPU Usage: While RAM improvements are significant, CPU usage shows only marginal improvement between the versions.

These optimizations mean that etcd 3.6.0 processes complete operations faster and utilize fewer resources, leading to a more efficient and stable distributed store.

Automated Release Process

The etcd team has made significant strides in automating the release process, transitioning from a highly manual approach that could take over a day for a single release (as seen with early 3.5 releases). The goal, as put by James (part of the release team), is to make releases "more and more boring" through automation, which is a positive indicator for reliability. This automation also includes plans for periodic jobs to scan for vulnerabilities in etcd's dependencies, aiming to improve security and enable faster CVE response times.

etcd Operator 0.1.0

A new etcd operator working group was formed about a year ago to address the fragmented landscape of etcd operators. The goal is to provide a community-standardized operator for managing etcd within Kubernetes. The etcd operator 0.1.0 is the first release from this group.

• Scope: Designed exclusively for running etcd inside Kubernetes, not for bare-metal clusters.
• Current Capabilities (0.1.0): Bootstraps the repository, enables automated updates, includes basic tests and release scripts, and supports creating new etcd clusters with customizable options and various CSI drivers for storage.
• Future Roadmap:
• 0.2.0: Enable TLS communication through certificate management, supporting providers like cert-manager and auto-providers. Fortify upgrades with more tests for patch and minor versions.
• 0.3.0 and beyond: Implement disaster recovery for cluster members, periodic and on-demand backups, and the creation/restoration of new clusters from available backups.

This initiative represents a concerted effort to bring a modern, compliant, and robust operator solution to the etcd community.

Demo / Proof of Concept

▶ Watch: Feature: New liveZ and readyZ health check endpoints (8:00)

The KubeCon talk primarily presented findings through detailed slides and visual data, such as read/write heatmaps and power line charts, which served as visual proofs of concept for the performance improvements in etcd 3.6.0. These charts effectively demonstrated the reduced memory consumption and increased throughput compared to etcd 3.5. While there wasn't a live, interactive demonstration of features like downgrades or the etcd operator in action, Siyuan Zhang's explanation of the new feature gates was delivered via a pre-recorded video segment, showcasing the new development approach. The focus was on conveying the technical advancements and the results of extensive testing rather than a live operational demo.

Defensive Implications

▶ Watch: Critical upgrade issue from etcd 3.5 to 3.6 (10:00)

The release of etcd 3.6.0, coupled with the introduction of the etcd operator, carries several critical implications for defenders and operators managing Kubernetes environments. Understanding and acting on these points is essential for maintaining robust, secure, and performant clusters.

First and foremost, the upgrade path to etcd 3.6.0 demands meticulous attention. The identified bug in etcd 3.5.1 through 3.5.19, where leader promotion was not correctly persisted in the V3 store, necessitates a mandatory pre-upgrade step. Defenders must ensure that their etcd 3.5 clusters are upgraded to etcd 3.5.20 or a higher patch version before attempting any upgrade to 3.6.0. Failure to do so risks cluster instability, data corruption, or "too many leaders" scenarios, which can lead to severe service disruptions. Operators should consult the official etcd documentation and blog posts related to this upgrade path for precise instructions.

The introduction of dedicated liveZ and readyZ health check endpoints provides a significant advantage for monitoring and automated remediation. Defenders should update their Kubernetes liveness and readiness probes to leverage these new endpoints. This granular distinction allows for more intelligent health checks; a process might be alive (liveZ healthy) but temporarily unable to serve traffic (readyZ unhealthy), allowing the scheduler to temporarily route traffic away without unnecessarily restarting the pod. This can improve application availability and reduce cascading failures.

Performance improvements, particularly the reduction in RAM usage due to the default snapshot-count change (from 100,000 to 10,000) and other memory optimizations, are directly beneficial. Defenders can expect etcd 3.6.0 to operate with a smaller memory footprint, potentially allowing for tighter resource limits or more headroom for other critical components on the same nodes. However, it's still crucial to monitor etcd's resource consumption closely, especially for large or high-traffic clusters, and tune configurations as needed. The warning about etcd being designed for metadata, not large-scale data, remains pertinent; while the database quota is configurable (e.g., up to 8GB or 16GB), increasing it significantly will still impact memory due to snapshotting and internal caching.

The new downgrade support offers a crucial safety net. While upgrades should always be thoroughly tested, the ability to perform a validated downgrade to the immediate prior minor version (e.g., 3.6 to 3.5) provides a robust fallback mechanism. Defenders should integrate the etcdctl downgrade validate and etcdctl downgrade enable commands into their operational runbooks for emergency rollback scenarios, ensuring they understand the schema migration process.

For those managing etcd within Kubernetes, the etcd operator 0.1.0 represents a strategic shift towards standardized, community-driven management. While 0.1.0 is not yet production-ready, defenders should closely track its development. Future versions promising TLS communication (via cert-manager or auto-providers), fortified upgrades, and comprehensive disaster recovery features will significantly enhance the security, reliability, and operational ease of etcd clusters in Kubernetes. Early engagement with the working group can also provide valuable feedback and influence the operator's development to meet specific organizational security requirements.

Finally, the commitment to automating the release process and incorporating periodic vulnerability scanning in etcd's development pipeline is a positive security signal. This proactive approach aims to identify and address CVEs in etcd's dependencies more rapidly, allowing defenders to patch their clusters more quickly and reduce their exposure to known vulnerabilities. Defenders should stay informed about etcd security advisories and promptly apply patches as they become available.

Key Takeaways

• Major Milestone Release: etcd 3.6.0 is a significant release after nearly four years, introducing crucial features like full V3 store migration, robust downgrade support, and Kubernetes-style feature gates, greatly enhancing etcd's stability and operability.
• Critical Upgrade Prerequisite: Users must upgrade their etcd 3.5 clusters to version 3.5.20 or higher before attempting to upgrade to 3.6.0 to avoid a severe leader promotion bug that affects versions 3.5.1 through 3.5.19.
• Enhanced Operational Visibility: The new liveZ and readyZ health check endpoints provide granular insights into etcd's state, enabling more sophisticated and reliable liveness and readiness probes in Kubernetes environments.
• Significant Performance Gains: etcd 3.6.0 demonstrates consistently better performance, particularly a notable reduction in RAM usage, primarily due to the default snapshot-count being lowered from 100,000 to 10,000.
• Community-Driven Operator: The new etcd operator 0.1.0 marks the beginning of a standardized, community-backed solution for managing etcd clusters within Kubernetes, with a clear roadmap for critical features like TLS, advanced upgrades, and disaster recovery.
• Metadata-Centric Design: etcd remains optimized for metadata storage, and while database size limits are configurable, larger databases will inherently increase memory consumption due to internal mechanisms like snapshotting and caching.

About the Speaker(s)

• Benjamin Wang: A key figure in the etcd project, serving as an etcd maintainer and the etcd code tag lead, guiding its technical direction and development.
• Ivan Valdes Castillo: Vice President of Engineering at Inar Technologies, an active etcd contributor, the new SIG etcd chair, and heavily involved in etcd's CI/CD tooling. He also co-leads the newly formed etcd release team.
• Siyuan Zhang: A reviewer for etcd, contributing to the quality and consistency of the codebase, and part of the team from Google.
• Arka Saha: A software engineer at VMware by Broadcom, actively contributing to etcd and responsible for making downstream releases of etcd and Kubernetes.
• Ciprian Hacman: A prominent Kubernetes maintainer who contributes to various open-source projects including Kops, node-problem-detector, and cloud providers, bringing extensive experience from the broader Kubernetes ecosystem.