Keynote: 221B Cloud Native Street - Ricardo Rocha & Katie Gamanji

Keynote: 221B Cloud Native Street - Ricardo Rocha & Katie Gamanji

Speakers: Ricardo Rocha, Computer Engineer, CERN; Katie Gamanji, Senior Engineer, Apple

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=jUChVGvSB5g

Overview

In this insightful keynote from KubeCon EU, Katie Gamanji, a Senior Engineer at Apple, and Ricardo Rocha, a Computer Engineer at CERN, both prominent members of the Cloud Native Computing Foundation (CNCF) Technical Oversight Committee (TOC), offered a comprehensive update on the state and strategic direction of the cloud native ecosystem. Titled "221B Cloud Native Street," the talk metaphorically positioned the TOC as detectives, investigating the health, growth, and future challenges of the vast CNCF landscape. The core focus was on the TOC's critical role in providing technical vision, steering the ecosystem, and guiding projects through various maturity levels.

The presentation highlighted the exponential growth of the CNCF, which now encompasses over 214 projects, having doubled its project count in just three years after taking six years to reach the first hundred. This rapid expansion presents both immense opportunities and significant scaling challenges for the community and its governance bodies. Gamanji and Rocha detailed the TOC's proactive measures to adapt its processes, restructure key technical groups, and identify emerging gaps within the cloud native space to ensure sustained health and innovation for the next decade. Their insights are crucial for anyone invested in the stability, security, and future trajectory of cloud native technologies, from project maintainers and contributors to end-users and enterprise architects.

The importance of this talk extends beyond mere organizational updates; it delves into the foundational mechanisms that ensure the integrity and reliability of the cloud native stack. By refining project maturity models, streamlining due diligence, and fostering a more collaborative environment, the TOC directly influences the quality, stability, and ultimately, the security posture of the software powering modern distributed systems. The speakers underscored that while technology is the gravitational point, it is the community and its well-defined processes that truly enable its success and secure its future.

Background

▶ Watch: Speakers introduce TOC's vision and technical steer (0:00)

The Cloud Native Computing Foundation (CNCF) has established itself as the premier open-source foundation for cloud-native technologies, fostering an ecosystem that now boasts over 214 projects. These projects are categorized into three distinct maturity levels: Sandbox, Incubated, and Graduated. Each level signifies a different stage of development, adoption, and community commitment, providing crucial guidance for both project maintainers and end-users.

Sandbox projects represent early-stage, greenfield ideas addressing niche problem spaces. They are experimental, offering innovative solutions without necessarily having widespread adoption or diverse contributions. Moving to Incubated projects, the criteria become more stringent, requiring evidence of production adoption and, critically, contributions from multiple organizations to ensure vendor neutrality and broader community engagement. Finally, Graduated projects are considered mature, stable, and widely adopted, having "crossed the chasm" from early adopters to the early majority. These projects are deemed foundational and are expected to be here to stay. Currently, the CNCF landscape comprises 134 Sandbox projects, 36 Incubated, and 31 Graduated projects.

An often-overlooked but vital aspect of project lifecycle management discussed by Gamanji is archival. Sometimes, projects do not gain sufficient momentum in terms of contributions or adoption. The TOC actively manages this process, encouraging maintainers to learn from these experiences and re-channel their energy into existing ecosystem projects or new initiatives. Currently, 13 projects have been archived, a testament to the TOC's commitment to maintaining a healthy and relevant project portfolio rather than simply accumulating projects indefinitely.

The exponential growth of the CNCF ecosystem — taking six years to reach 100 projects and then only three more years to reach 200 — presented a significant challenge: scaling the community and its governance processes at the same pace as technological growth. Ricardo Rocha likened this challenge to scaling infrastructure, where increasing load necessitates continuous process improvement. In response, the TOC initiated a comprehensive revision of its internal processes over the past two years. These improvements included the introduction of Domain Technical Reports and General Technical Reports, enhanced pre-checks for project submissions, and refined due diligence procedures, including adopter interviews and surveys. The goal was to better manage the influx of new projects and facilitate the maturation of existing ones, ensuring that the TOC could effectively guide the ecosystem without becoming a bottleneck.

Key Findings

▶ Watch: Why archival is a natural, healthy project lifecycle step (2:00)

The TOC's recent efforts have yielded several critical insights and tangible improvements across the CNCF ecosystem:

1. End-User Understanding and Reliance on Project Maturity Levels:

• A survey conducted by the TOC revealed that a majority of end-users have a good understanding of the meaning behind the Sandbox, Incubated, and Graduated maturity levels and actively rely on them when selecting projects for their technology stacks.
• However, feedback also indicated a desire for more granular information. End-users expressed interest in understanding a project's progress within an incubation phase (e.g., how close it is to graduation) and preferred to link specific project releases to their corresponding maturity levels at the time of release. This suggests a need for more dynamic and versioned maturity indicators.
• Internally, organizations use these levels in varied ways. Some utilize them as an initial data point, performing extensive additional due diligence. Others integrate maturity levels as one criterion within more complex internal matrices for project selection, demonstrating the levels' utility but also the need for organizations to tailor their evaluation processes.

1. Significant Improvement in Project Review Throughput:

• The revamped TOC processes, including improved due diligence, pre-checks, and technical reports, demonstrated remarkable success in managing the project review queue.
• A major achievement was announced on February 25th, clearing the entire Sandbox project review queue, a significant backlog that had previously strained TOC resources. This accomplishment was met with widespread relief and positive feedback from TOC members.
• As of the time of the talk, the Sandbox queue was down to just five projects, a manageable number that could be processed within a single review period. Similarly, the backlog for moves to incubation and graduation was also well under control, indicating that the new processes are effectively scaling to meet the demands of the growing ecosystem.

1. Necessity and Structure of the TAG Reboot:

• The existing structure of Technical Advisory Groups (TAGs), micro-communities focusing on specific domains like observability or security, was identified as not scaling effectively to meet the evolving demands of the TOC and the broader community.
• This recognition led to the initiation of a "TAG Reboot" workload, detailed in issue 1527. The primary objective is to restructure and revitalize the TAGs to ensure they can effectively support the cloud native ecosystem for the next decade.
• Key changes include reducing the number of TAGs from eight to five, introducing Community Groups, and formally defining Sub-projects (long-lived efforts like project review and contributor strategy) and Initiatives (short-lived efforts with clear objectives and exit criteria, such as the AI white paper working group). This new structure aims to foster increased collaboration, both within the CNCF and with external foundations, by allowing anyone to open an initiative with clear stewardship and goals.
• The reboot also aims to fill identified ecosystem gaps, including multicluster management and observability, cost management and sustainability, and infrastructure provisioning and secret management, by encouraging community contributions and collaboration in these critical areas.

These findings collectively underscore the TOC's proactive approach to governance, ensuring that the CNCF ecosystem remains robust, responsive, and well-positioned for future growth and innovation.

Technical Deep Dive

▶ Watch: Survey reveals end-user reliance on project maturity levels (3:30)

The technical depth of this keynote lies not in a specific vulnerability or system architecture, but in the sophisticated governance mechanisms and process improvements designed to manage a rapidly expanding, highly diverse technical ecosystem. The TOC's work is a masterclass in scaling an open-source foundation, adopting principles akin to robust software engineering for organizational health.

One of the core technical challenges addressed was the efficient and fair evaluation of projects seeking to join or advance within the CNCF. The TOC refined its due diligence process significantly. This now includes:

• Adopter Interviews: Direct engagement with organizations already using a project in production. This provides real-world feedback on stability, maintainability, and value.
• End-User Surveys: Broader data collection to gauge community perception, understanding of maturity levels, and specific pain points or feature requests. This quantitative data complements qualitative interviews.
• Pre-checks: A set of criteria and evaluations performed before a full due diligence review, streamlining the initial vetting process and ensuring projects meet basic readiness requirements.
• Domain Technical Reports (DTRs) and General Technical Reports (GTRs): These comprehensive reports provide in-depth technical analysis of projects, assessing their architecture, design principles, adherence to cloud-native best practices, and alignment with the CNCF vision. DTRs focus on specific technical domains, while GTRs offer broader technical evaluations. These reports are crucial for informed decision-making by the TOC.

The most significant technical restructuring discussed was the TAG Reboot. The existing eight Technical Advisory Groups (TAGs), each comprising Subject Matter Experts (SMEs) in domains like Observability, Security, or Environmental Sustainability, were struggling to scale. Their original mandate involved producing white papers, forming working groups, and guiding projects. The reboot aims to optimize their effectiveness by:

1. Consolidating TAGs: Reducing the number of TAGs from eight to five. This consolidation is intended to reduce overhead, foster broader collaboration across related domains, and create more impactful, better-resourced groups. The specific new TAG structures were not detailed in the transcript but the intent is clear: to streamline and focus efforts.
2. Introducing Community Groups: These are broader forums for community engagement and discussion, providing a platform for initial ideas and collaboration before formalizing into more structured efforts.
3. Formalizing Sub-projects: These are defined as long-lived efforts requiring continuous stewardship. The talk identified "project review" and "contributor strategy" as existing sub-projects. This formalization ensures dedicated resources and ongoing attention to critical, enduring operational aspects of the CNCF. This is akin to defining core modules or services within a larger software system that require continuous maintenance.
4. Defining Initiatives: These are short-lived efforts with very clear objectives and explicit exit criteria. An example cited was the "Artificial Intelligence working white paper." The critical design principle here is to increase collaboration within the community and with external foundations. Anyone can propose an initiative, provided it has clear stewardship and well-defined goals. This allows for agile, focused efforts to address specific, time-bound problems or opportunities without creating permanent overhead. It's a modular approach to problem-solving, enabling rapid iteration and cross-foundation partnerships.

A new organizational entity, the Technical Advisory Board (TAB), was also introduced. Composed of end-users, the TAB's mandate is to establish closer feedback loops between projects and end-users, develop reference architectures to guide cloud-native adoption, and create a structured process for identifying ecosystem gaps. This board acts as a crucial bridge, translating real-world end-user needs and challenges back into the TOC's strategic planning and project incubation processes. The development of reference architectures, in particular, is a direct technical contribution, providing validated, best-practice blueprints for deploying cloud-native infrastructure, similar to architectural patterns in software design.

The identified ecosystem gaps—multicluster management and observability, cost management and sustainability, and infrastructure provisioning and secret management—are critical technical areas. The TOC's intention to actively encourage contributions and movement in these areas through the TAG reboot and initiatives demonstrates a strategic technical vision, aiming to fill voids in the cloud-native toolkit. For instance, the lack of robust, standardized secret management tooling has long been a challenge in cloud-native deployments, and surfacing this as a priority signals a direct intent to foster technical solutions.

In essence, the "technical deep dive" here is into the meta-architecture of the CNCF itself: how it is designed to evolve, manage its components (projects, TAGs), and interact with its users, all to ensure the robust and secure functioning of the cloud-native landscape.

Demo / Proof of Concept

▶ Watch: User feedback to enhance project maturity level clarity (4:40)

This keynote address, delivered by members of the CNCF Technical Oversight Committee, was a strategic update on the organizational health, process improvements, and future direction of the cloud native ecosystem. As such, it did not include a live technical demonstration or a proof of concept of any specific software or vulnerability. The focus was entirely on presenting findings, strategic initiatives, and structural changes within the CNCF governance.

Defensive Implications

▶ Watch: TOC successfully cleared the backlog of sandbox project reviews (6:50)

While the keynote did not directly discuss security vulnerabilities or specific defensive tools, the foundational work presented by the TOC has profound defensive implications for the entire cloud-native ecosystem. A robust, well-governed, and healthy open-source foundation is a prerequisite for secure software development and deployment.

1. Project Maturity as a Security Indicator: The structured Sandbox, Incubated, and Graduated maturity levels serve as an indirect but critical security signal.

• Graduated projects, having undergone rigorous due diligence, extensive production adoption, and contributions from multiple organizations, generally imply a higher level of scrutiny, code quality, and community support. This often translates to more robust security practices, better vulnerability management, and a wider array of eyes on the codebase. Defenders can generally place more trust in graduated projects for critical production workloads.
• Incubated projects offer a moderate level of assurance, with some production adoption and diverse contributions, suggesting a growing security posture.
• Sandbox projects, being experimental, inherently carry higher risk. Defenders should approach Sandbox projects with extreme caution, performing thorough security audits, and understanding the potential for rapid changes or less mature security practices. The TOC's effort to provide more granular information on progress within levels and link maturity to specific releases will further empower defenders to make more informed risk assessments.

1. Enhanced Due Diligence and Vetting: The TOC's improved processes, including adopter interviews, end-user surveys, pre-checks, and comprehensive Domain/General Technical Reports, directly contribute to a more secure ecosystem. By thoroughly vetting projects before they advance through maturity levels, the TOC helps filter out projects with inadequate security practices, poor code quality, or a lack of community engagement, which are all risk factors. A more rigorous intake and progression process means that projects entering or maturing within the CNCF are more likely to adhere to higher security standards.

1. Strategic Role of Technical Advisory Groups (TAGs): The TAG Reboot has significant defensive implications, particularly for the Security TAG.

• A well-structured and efficiently operating Security TAG can produce vital white papers, best practices guidelines, and reference architectures specifically focused on cloud-native security. These resources are invaluable for defenders seeking to implement secure configurations, understand emerging threats, and adopt robust security strategies.
• By streamlining TAGs and fostering Initiatives, the CNCF can rapidly convene experts to address emergent security challenges, conduct security audits across projects, or develop targeted security tooling. For example, a temporary initiative could be formed to address a widespread vulnerability or to develop a security-focused white paper on a novel threat vector.
• The TAGs also provide direct guidance to projects moving through maturity levels, allowing security experts to influence project design and implementation from an early stage, embedding security by design.

1. Addressing Ecosystem Gaps (Secret Management): The explicit identification of infrastructure provisioning and secret management as a critical ecosystem gap is a direct call to action for improved defensive capabilities. Secure secret management is a cornerstone of cloud-native security, protecting sensitive data like API keys, database credentials, and certificates. A lack of robust, standardized, and integrated solutions in this area poses significant risks. By highlighting this gap, the TOC is encouraging the community to contribute and innovate in this crucial defensive space, which could lead to better tools, best practices, and more secure deployments.

1. Community Health and Collaboration: A thriving, well-governed open-source community, as fostered by the TOC's efforts, is inherently more secure. More contributors mean more eyes on the code, increasing the likelihood of identifying and fixing vulnerabilities. Strong community engagement facilitates quicker response times to security incidents, better vulnerability reporting mechanisms, and a shared responsibility for security across the ecosystem. The emphasis on cross-foundation collaboration through initiatives can also lead to shared security intelligence and coordinated defensive strategies across broader open-source landscapes.

1. Guidance from the Technical Advisory Board (TAB): The TAB's focus on reference architectures and identifying ecosystem gaps from an end-user perspective directly supports defensive efforts. Reference architectures can provide tested, secure blueprints for deploying cloud-native applications, helping organizations avoid common misconfigurations and security pitfalls. By identifying gaps based on real-world operational challenges, the TAB helps prioritize the development of tools and practices that directly enhance security.

In summary, the TOC's strategic work, by fostering project maturity, improving vetting processes, empowering specialized security groups, and highlighting critical technical gaps, creates a more resilient and secure foundation upon which cloud-native applications can be built and operated. Defenders benefit from a better-vetted project landscape, clearer guidance, and a community actively working to close security gaps.

Key Takeaways

• The CNCF ecosystem is experiencing exponential growth, now exceeding 214 projects, necessitating continuous adaptation of governance processes to scale effectively.
• The TOC has successfully streamlined project review processes, clearing the Sandbox project queue and improving throughput for incubation and graduation, demonstrating effective scaling of community efforts.
• End-users largely understand and rely on CNCF project maturity levels, but desire more granular information, such as progress within a level and linking maturity to specific project releases.
• The TAG Reboot is a critical initiative to restructure Technical Advisory Groups (from 8 to 5), introduce Sub-projects and Initiatives, and foster increased collaboration to address scaling challenges and ecosystem gaps.
• Key ecosystem gaps identified for future focus include multicluster management and observability, cost management and sustainability, and infrastructure provisioning and secret management.
• The newly formed Technical Advisory Board (TAB), composed of end-users, aims to establish closer feedback loops, develop reference architectures, and systematically identify ecosystem gaps, complementing the TOC's strategic work.

About the Speaker(s)

Katie Gamanji is a Senior Engineer at Apple and a distinguished member of the Cloud Native Computing Foundation (CNCF) Technical Oversight Committee (TOC). Her expertise lies in steering the technical vision and providing guidance for projects within the expansive cloud native ecosystem. Her contributions include overseeing project maturity, community health assessments, and the strategic evolution of CNCF processes.

Ricardo Rocha is a Computer Engineer at CERN and also serves as a valued member of the CNCF Technical Oversight Committee (TOC). His work within the TOC focuses on ensuring the technical health and scalability of the CNCF landscape, including refining due diligence processes, managing project maturation, and addressing the challenges posed by the rapid growth of cloud native technologies. Together, Katie and Ricardo provide crucial leadership in shaping the future of cloud native computing.