SIG Security: Succession Pla ...Cailyn Edwards, Iain Smart, Rory McCune, Tabitha Sable & Mahé Tardy

SIG Security: Succession Planting for a Flowering Future

Speakers: Cailyn Edwards, Co-chair, Kubernetes SIG Security, Ozero by Octa; Iain Smart, Consultant, Amberwolf, Co-project lead, SIG Third Party Audit; Rory McCune, Security Lead, Data Dog, Co-lead, SIG Security Docs; Tabitha Sable, Co-chair, Kubernetes SIG Security; Mahé Tardy, Software Engineer, Cisco, SIG Security Tooling Representative

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=0p-sZT0LWOg

Overview

This talk, "Succession Planting for a Flowering Future," presented by key maintainers of Kubernetes SIG Security, delves into the critical, community-driven efforts to enhance the security posture of the Kubernetes project and its vast user base. The speakers, including co-chairs Tabitha Sable and Cailyn Edwards, along with sub-project leads Iain Smart, Rory McCune, and Mahé Tardy, articulate SIG Security's foundational philosophy: that security within an open-source ecosystem as expansive as Kubernetes cannot be dictated by a single entity but must be a collective endeavor. The talk uses the metaphor of "succession planting" to illustrate the ongoing growth, nurturing, and evolution of the SIG, emphasizing the importance of fostering new contributors and ensuring the long-term health and resilience of Kubernetes security initiatives.

The presentation provides a comprehensive update on the diverse activities undertaken by SIG Security's sub-projects, spanning third-party security audits, documentation improvements, and the development of security tooling. It highlights specific achievements, ongoing projects, and future plans, all aimed at making Kubernetes inherently more secure and providing users with the necessary resources to operate their clusters safely. The talk is particularly relevant for anyone involved in Kubernetes deployment, management, or development, offering insights into the proactive measures being taken to address security challenges in a rapidly evolving cloud-native landscape. It underscores the continuous commitment of the Kubernetes community to security as a shared responsibility, inviting new members to contribute their expertise and passion to this vital work.

Background

▶ Watch: What is SIG Security and its community approach? (1:06)

Kubernetes, as a cornerstone of modern cloud-native infrastructure, presents unique and significant security challenges due to its complexity, distributed nature, and widespread adoption across diverse environments. The sheer volume of its codebase, comprising "multiple millions of lines of Go and various other programming languages," means that no single individual or organization can realistically oversee its security in isolation. As Tabitha Sable eloquently puts it, "everyone who works on Kubernetes has the ability to make Kubernetes insecure. Anyone can write a bug. Anyone can write bad advice on a blog post. Anyone can write a design that has failed to take adversarial nature into account." This fundamental truth underpins the existence and philosophy of Kubernetes SIG Security.

Formed on the principle that if everyone can introduce insecurity, then everyone also has the power to contribute to making it secure, SIG Security adopts a community-based approach. Its primary goal is to provide a collaborative space for individuals within the Kubernetes community to share security interests and concerns, facilitating co-working with other SIGs (Special Interest Groups) to implement security improvements in their respective domains. This collaborative model is crucial for a project of Kubernetes' scale, ensuring that security considerations are integrated across the entire ecosystem rather than being siloed.

Historically, SIG Security has tackled various issues, demonstrating its impact. One notable example cited by Mahé Tardy and Tabitha Sable was the successful effort to remove the security context deny admission controller. This controller, while nominally a security feature, had become an "attractive nuisance" because enabling it prevented modern clusters from functioning correctly. Despite its obsolescence, its mere presence led to compliance burdens, forcing organizations to write extensive justifications for not using an outdated feature. Through a "long road down the Kubernetes change control process" involving SIG Security and SIG Auth, this problematic component was finally removed, streamlining compliance and reducing potential misconfigurations. This illustrates SIG Security's role not just in adding new security features, but also in refining and removing outdated or counterproductive elements from the project. The continuous evolution of Kubernetes demands a proactive and adaptive security strategy, which SIG Security endeavors to provide through its multi-faceted initiatives.

Key Findings

▶ Watch: Core philosophy: Community contribution to Kubernetes security. (3:20)

The talk, rather than presenting novel research findings, highlights the ongoing and critical contributions of Kubernetes SIG Security through its various sub-projects. These represent the "key findings" in terms of continuous improvement, community engagement, and proactive security posture for the Kubernetes ecosystem.

1. Sustained Third-Party Security Audits: SIG Security recognizes the limitations of internal review and consistently commissions external security firms to perform dedicated security reviews of the Kubernetes project and its codebase. This ongoing commitment ensures fresh, adversarial eyes scrutinize the project for vulnerabilities. Past audits in 2018 (with Atredis and Trail of Bits) and 2021 (with NCC Group) have yielded significant findings, with current efforts focused on a 2025 audit by Shielder, funded by the Open Source Technology Improvement Fund (OSTIF). This structured approach to external validation is a cornerstone of Kubernetes' security assurance.

1. Comprehensive Security Documentation: The SIG Security Docs sub-project is actively working to improve the security content of the official Kubernetes website. This includes developing hardening guides (e.g., for the scheduler), identifying and rectifying stale or incorrect security information, and contributing to the resolution of security-related issues across the project. This effort directly empowers users to configure and manage their clusters more securely, leveraging the collective wisdom of the community. A notable "finding" from this work was the discovery of the Kubernetes API server's self-signed loopback certificate having a 12-month lifespan, which, if not refreshed by a restart, causes the API server to crash—a critical operational detail uncovered through documentation review.

1. Robust Security Tooling and Vulnerability Disclosure: The SIG Security Tooling sub-project maintains and develops essential security tools and processes. This includes running the Sneak scanner on Kubernetes release images and managing the official CVE feed. This feed, available as a web page, JSON feed, and RSS feed, provides timely updates on Kubernetes vulnerabilities. Ongoing work aims to achieve near real-time updates via webhooks and to transition to the Open Source Vulnerability (OSV) format for enhanced machine readability and integration with other security tools. The initiation of the Go-check project also signifies an internal effort to leverage static analysis for code quality and security.

1. Community-Driven Vulnerability Management: Beyond external audits, the SIG maintains a HackerOne program for responsible disclosure and encourages direct reporting to security@kubernetes.io. Their process includes not only fixing identified issues but also diligently reviewing and addressing past audit findings—categorizing them as fixed, intentional by design (requiring documentation), or needing a Kubernetes Enhancement Proposal (KEP) for significant architectural changes. This holistic approach ensures that security concerns are systematically managed from initial report to resolution or documentation.

These ongoing initiatives collectively form the "findings" of SIG Security: a continuously improving security posture, a transparent vulnerability management process, and a wealth of accessible security knowledge for the community.

Technical Deep Dive

▶ Watch: Introduction to SIG Security's subprojects and structure. (4:40)

Kubernetes SIG Security's strength lies in its structured approach to tackling security challenges through dedicated sub-projects: Third-Party Audit, Documentation, and Tooling. Each sub-project employs specific technical methodologies and resources to achieve its objectives.

Third-Party Audit Sub-project

Led by Iain Smart, this sub-project is responsible for orchestrating external security reviews of the Kubernetes project. The rationale is clear: while the internal community is vigilant, the sheer scale and complexity of the codebase necessitate "extra eyes on the codebase" from dedicated security vendors.

Historically, two major audits have been conducted:

• 2018 Audit: Performed by Atredis and Trail of Bits. Findings are documented under Kubernetes GitHub issue 81146.
• 2021 Audit: Conducted by NCC Group. Findings are available under Kubernetes GitHub issue 118980.

The current focus is on the 2025 Kubernetes audit, a significant undertaking designed to be more targeted than previous efforts. Instead of attempting to review the "entire codebase," which is deemed inefficient, the SIG has adopted a strategy of focusing on "different projects from various different SIGs." This involves:

1. RFP Process: Approaching other SIGs to identify components or sub-projects they would like to have reviewed. The Request for Proposal (RFP) documentation for this audit is publicly available in the SIG Security GitHub repository.
2. Vendor Selection: Working with the Open Source Technology Improvement Fund (OSTIF), which recommended several vendors. Shielder was selected as the chosen vendor for the 2025 audit.
3. Ongoing Review: Shielder is currently reviewing "quite a lot of the sub-projects." The specific list of reviewed components is documented within the SIG Security project.
4. Findings and Disclosure: The audit is already yielding findings, which are being managed through a responsible disclosure process. This ensures that vulnerabilities are fixed before public release of the audit report, expected by the "tail end of this year."

A crucial aspect of the audit sub-project is the ongoing work to address findings from previous audits. Many issues are tracked, and their status is meticulously managed:

• Closed and Fixed: Demonstrating tangible security improvements.
• Closed as "Won't Fix": These are often "pointy edges" or intentional design choices that might pose a risk if misused. In such cases, the SIG ensures these are "documented elsewhere in the Kubernetes documentation" to guide users away from potential pitfalls.
• Needs a KEP (Kubernetes Enhancement Proposal): These findings require "significant effort" to fix, often necessitating architectural changes to avoid "breaking changes in Kubernetes." These represent opportunities for new contributors to engage with the project on impactful security improvements.

SIG Security Docs Sub-project

Rory McCune leads the documentation efforts, emphasizing that "the act of writing something down... will lead you to better understand something." The sub-project's mission is twofold:

1. Improve Kubernetes Website Security Content: Collaborating with other SIGs (like SIG Docs, SIG Auth, SIG Node) to ensure the official Kubernetes website provides accurate, up-to-date, and comprehensive security information. This is critical as the website is a "primary resource" for users configuring and managing clusters.
2. Develop Standalone Security Content: Creating resources like threat models and white papers that might not fit directly into the main documentation but are crucial for a deeper understanding of Kubernetes security. These are often hosted in the SIG Security GitHub repository.

Key initiatives and technical details include:

• Hardening Guides: A long-running project to develop detailed hardening guides for different aspects of Kubernetes. An example cited is the recently completed guide for the Kubernetes scheduler, which delved into the "security implications of different parameters and different features." This process not only produces valuable documentation but also serves as a learning opportunity for contributors.
• Website Issue Review: Actively reviewing existing open issues on the Kubernetes website that relate to security. This process has led to the discovery of obscure but critical details, such as:
• The Kubernetes API server uses a self-signed certificate for loopback calls (internal communication).
• This certificate has a 12-month lifespan.
• If the API server runs for more than 12 months without a restart, this certificate will become invalid, causing the API server to crash. This specific detail highlights the value of deep documentation dives in uncovering operational security risks.

The sub-project emphasizes collaboration, as SIG Security "don't own the code." They work with code-owning SIGs to ensure technical accuracy and with SIG Docs for adherence to website style and standards.

SIG Security Tooling Sub-project

Mahé Tardy represents the tooling sub-project, which focuses on building and improving security through code and cross-SIG collaboration. The sub-project fosters a learning environment for new contributors through regular working and learning sessions.

Technical activities and tools include:

• Sneak Scanner Integration: Running the Sneak scanner on Kubernetes release images to identify vulnerabilities in dependencies. Recent efforts have focused on:
• Migrating the scanning scripts from test-infra to the SIG Security repository for better ownership and management.
• Moving the scan jobs to "less trusted clusters" to enhance the security posture of the scanning process itself.
• Official CVE Feed: A critical resource for the community, providing an auto-refreshing list of Kubernetes CVEs.
• Availability: Accessible as a dedicated web page on the Kubernetes website, a JSON feed, and an RSS feed.
• Current Status: The initial goals to make the feed generally available (GA) are mostly complete.
• Future Enhancements: The primary remaining task is to achieve near real-time updates. Currently, the feed can be up to 12 hours late if the website is not rebuilt. The proposed solution is to use a webhook to trigger website rebuilds upon new CVE publication.
• OSV Format Adoption: A significant new initiative is the transition to the Open Source Vulnerability (OSV) format. This was prompted by feedback from projects like Aqua's CVE feed OSV, which consume the existing feed but require a more structured format. The goal is to collaborate with the Kubernetes Security Response Committee (SRC) to issue initial CVEs directly in the OSV format, moving away from the current "free form" text in GitHub issues, thereby improving machine readability and integration with automated security tools.
• Go-check Project: An initiative to run the Go-check project (a static analysis tool for Go code) on the Kubernetes codebase. While the tool has been updated, the SIG is actively seeking contributors to help "using this scanning results" to identify and address findings, indicating a need for expertise in interpreting static analysis outputs.

These technical efforts demonstrate SIG Security's commitment to both reactive (CVE management, audits) and proactive (documentation, static analysis) security measures, continuously evolving to meet the demands of the Kubernetes ecosystem.

Demo / Proof of Concept

▶ Watch: Key initiative: Revamping the OWASP Kubernetes Top 10. (7:15)

The talk "Succession Planting for a Flowering Future" did not feature a live technical demonstration or a traditional proof of concept. Instead, the presentation focused on providing an overview of the Kubernetes SIG Security's ongoing initiatives, sub-projects, and future plans, showcasing the collaborative work and the tangible outputs of the community's efforts through discussion rather than interactive display.

Defensive Implications

▶ Watch: Deep dive into the Third-Party Audit subproject. (8:00)

The work of Kubernetes SIG Security carries profound defensive implications for anyone operating, developing, or securing Kubernetes environments. The insights and resources provided by the SIG are directly actionable for improving the security posture of cloud-native deployments.

1. Leverage Official Security Resources:

• Subscribe to the CVE Feed: Cluster operators and security teams should immediately subscribe to the official Kubernetes CVE feed (RSS, JSON, or webpage). This ensures timely awareness of newly disclosed vulnerabilities, enabling rapid assessment and patching of affected clusters. As the feed aims for near real-time updates and OSV format adoption, its utility will only increase for automated vulnerability management.
• Consult Kubernetes Security Documentation: Regularly refer to the official Kubernetes website's security sections and the SIG Security's white papers and hardening guides. These resources provide authoritative guidance on secure configuration, threat models, and best practices. The example of the API server's 12-month certificate lifespan highlights that obscure but critical operational security details are documented here.
• Review Audit Reports: Access and review the public reports from the third-party security audits (e.g., 2018, 2021). These reports detail specific vulnerabilities and recommended mitigations, offering a valuable external perspective on potential weaknesses in Kubernetes components.

1. Proactive Configuration and Hardening:

• Implement Hardening Guides: Actively follow the hardening guides developed by SIG Security, such as the one for the scheduler. These guides provide prescriptive advice on configuring Kubernetes components securely, reducing the attack surface, and mitigating common risks.
• Stay Updated: The continuous nature of security work means that best practices evolve. Regularly updating Kubernetes clusters to the latest secure versions and reviewing configurations against current recommendations is paramount. The removal of the problematic security context deny admission controller, for instance, simplifies compliance and reduces the risk of misconfiguration by eliminating an "attractive nuisance."

1. Contribute to Community Security:

• Participate in SIG Security: For security professionals, developers, or even new contributors, joining SIG Security offers a direct avenue to influence and improve Kubernetes security. This includes contributing to documentation, helping to analyze Go-check results, contributing to KEPs for complex fixes, or even reviewing code. The community-driven model means that every contribution, regardless of experience level, enhances the collective security posture.
• Responsible Disclosure: If vulnerabilities are discovered, utilize the official HackerOne program or email security@kubernetes.io. Following responsible disclosure guidelines ensures that issues are addressed systematically and securely, protecting the broader community.

1. Understand Kubernetes' Security Philosophy:

• Recognize that Kubernetes security is a shared responsibility. While the project provides a secure foundation, the ultimate security of a deployment depends heavily on how it is configured and managed.
• Be aware of "intentional by design" aspects that might have security implications and ensure these are understood and accounted for in your specific threat model and risk assessment.

By embracing these defensive implications, organizations and individuals can significantly enhance their ability to secure Kubernetes, aligning their practices with the proactive and community-driven security efforts of Kubernetes SIG Security.

Key Takeaways

• Community-Driven Security is Paramount: Kubernetes SIG Security operates on the principle that security for a project of Kubernetes' scale is a collective responsibility, not a dictated mandate, fostering contributions from all experience levels.
• Continuous External Validation is Essential: Regular third-party security audits (e.g., Shielder for the 2025 audit, funded by OSTIF) provide critical external scrutiny, identifying vulnerabilities and driving continuous improvement.
• Robust Documentation Empowers Users: SIG Security actively improves the official Kubernetes documentation, providing crucial hardening guides and uncovering deep-seated operational security details (like the API server's 12-month self-signed certificate expiry).
• Official CVE Feed is a Critical Resource: The SIG maintains and continuously enhances the official Kubernetes CVE feed (web, RSS, JSON), with plans for near real-time updates and transition to the OSV format for improved automation and machine readability.
• Opportunities for Contribution are Abundant: From reviewing audit findings and contributing to KEPs to improving documentation and analyzing tooling results (Sneak scanner, Go-check), SIG Security welcomes new contributors of all backgrounds.
• Proactive Security Management is Key: The SIG actively removes outdated or problematic features (like the security context deny admission controller) and systematically addresses audit findings, ensuring Kubernetes' security posture evolves responsibly.

About the Speaker(s)

Tabitha Sable is one of the co-chairs of Kubernetes SIG Security. She plays a pivotal role in creating a collaborative space for the community to improve Kubernetes security, and she helped lead the effort to remove the outdated security context deny admission controller.

Cailyn Edwards is the newest and "greenest" co-chair of Kubernetes SIG Security. In her day job, she works on security at Ozero by Octa. She emphasizes the SIG's nurturing environment for new contributors, drawing on her own journey from KubeCon attendee to co-chair.

Iain Smart is a consultant at Amberwolf and one of the co-project leads for the SIG Security Third-Party Audit sub-project. He is responsible for coordinating external security reviews of the Kubernetes codebase, having previously been involved in the delivery team for the 2021 audit with NCC Group.

Rory McCune is a security lead at Data Dog and one of the co-leads of the SIG Security Docs sub-project. He is passionate about improving the security content of the Kubernetes website and white papers, believing that the act of writing documentation significantly enhances understanding of the project's security aspects.

Mahé Tardy works at Cisco as a software engineer and represents the SIG Security Tooling sub-project. He is involved in initiatives such as running the Sneak scanner on release images, maintaining the official CVE feed, and exploring the adoption of the OSV format for vulnerability reporting.