The Bricks That Make Us – How the LEGO Group Avoids 50 Medio... Thomas Øther Rasmussen & Paul Farver

The Bricks That Make Us – How the LEGO Group Avoids 50 Mediocre Kubernetes Implementations

Speakers: Thomas Øther Rasmussen, Platform Engineer; Paul Farver, Platform Engineer

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=weOIwt1lbvI

Overview

In a compelling KubeCon EU talk, Thomas Øther Rasmussen and Paul Farver from the LEGO Group unveiled their strategic approach to platform engineering, demonstrating how a large, decentralized organization can successfully standardize its containerized workloads on a single, opinionated Kubernetes platform. Titled "The Bricks That Make Us," the presentation delves into the challenges of managing a vast digital estate with numerous autonomous product teams and the pitfalls of fragmented infrastructure solutions.

The core of their message revolves around a pragmatic, user-centric methodology that combines robust technical architecture with a "sales mindset" for internal adoption and a strong emphasis on continuous support. This talk is particularly relevant for organizations grappling with platform sprawl, inconsistent developer experiences, and the difficulty of driving standardization without stifling innovation. It provides a blueprint for fostering a collaborative environment where platform teams act as enablers rather than gatekeepers, ultimately leading to greater efficiency, resilience, and developer satisfaction.

Background

▶ Watch: Speakers introduce themselves and LEGO/Kubernetes connection (0:00)

The LEGO Group's digital organization is a sprawling ecosystem, comprising 237 distinct product teams and approximately 2,000 colleagues. This structure is characterized by an exceptionally high degree of autonomy; product teams are empowered to choose the best technology and hosting solutions for their specific applications, even if that means running on an "old VM somewhere in a basement." While this autonomy fosters innovation and allows teams to optimize for their unique needs, it inevitably leads to a diverse, and often fragmented, infrastructure landscape.

Before the introduction of their new platform, LEGO faced the challenge of "50 mediocre Kubernetes implementations" – a situation where numerous teams had independently deployed and managed their own container orchestration solutions, leading to inconsistencies, duplicated effort, and varying levels of operational maturity. The speakers highlighted the classic XKCD comic where adding a new standard often just creates "one more standard to compete with," a trap they were determined to avoid. Further complicating matters was a strategic company goal to migrate away from an existing data center, leaving some product teams without a clear exit strategy for their applications. These factors underscored the urgent need for a unified, robust, and appealing container platform that could offer a superior alternative to the existing patchwork of solutions.

Key Findings

▶ Watch: Understanding user autonomy, variety, and existing platforms (4:00)

The central finding of the LEGO Group's journey is that avoiding numerous mediocre Kubernetes implementations requires a dual-pronged strategy: building a single, highly opinionated platform and then actively "selling" it to internal users with empathy and robust support. They named their solution the Novous Container Platform (Latin for "new"), symbolizing a fresh start.

Their approach is characterized by several key principles:

1. Opinionated Design with GitOps Enforcement: The platform is designed to be highly opinionated, enforcing GitOps as the sole method for deploying and managing applications. This means all configurations and application definitions must reside in Git repositories, eliminating direct kubectl apply commands. This decision, while initially met with some resistance from teams accustomed to direct server access, significantly enhances consistency, auditability, and resilience.
2. Seamless Multi-Environment Experience: The platform aims to provide a consistent experience across diverse environments, including on-premise data centers, factory edge sites, and various cloud regions. This seamlessness is crucial for accommodating the varied operational needs of LEGO's product teams.
3. Strategic User Migration Archetypes: Recognizing that not all users are the same, the platform team identified three distinct archetypes during initial migrations:

• Lift Users: Teams under time pressure, reluctant to re-architect, who needed their applications "lifted" directly onto Novous. These teams often require ongoing support.
• Experimenters: Enthusiastic teams willing to re-architect and adapt their CI/CD pipelines to fully leverage the platform's recommendations. These users quickly become self-sufficient.
• Custom Feature Users: Teams with unique, business-critical requirements (e.g., factory molding machines) that necessitated custom features or "technical debt" within the platform to ensure business continuity.

1. Internal "Sales Mindset" for Adoption: The platform team adopted strategies inspired by sales and marketing to drive adoption. This involved proactive engagement through workshops, participation in internal events, and building personal connections with user teams to understand their pain points.
2. Proactive and Empathetic Support: A cornerstone of their success is a dedicated, structured support system. This includes a public support channel with a support rotation of platform engineers and a "rent an engineer" program where platform engineers embed with product teams. This approach ensures users feel heard and supported, fostering trust and collaboration.
3. Empowered Team Structure: The platform team itself operates with a lean meeting schedule, empowering pairs of engineers to make significant decisions about the platform. This frees up time for direct user interaction and platform development, rather than endless internal discussions.

Ultimately, the LEGO Group's key finding is that a successful internal platform is not just about superior technology; it's equally about a commitment to user experience, proactive engagement, and building strong, empathetic relationships with the teams it serves.

Technical Deep Dive

▶ Watch: Strategic driver: Migrating from data centers, providing exit strategy (5:45)

The Novous Container Platform is the LEGO Group's answer to consolidating its containerized workloads, designed from the ground up to be highly opinionated and user-centric. At its core, Novous aims to abstract away the complexity of raw Kubernetes while providing a powerful, standardized environment for containerized applications.

The platform's technical foundation is built on several key architectural decisions:

1. GitOps Enforcement: This is the most significant technical mandate. All application deployments and configurations must be defined in Git repositories. The platform leverages a GitOps operator (though the specific tool, e.g., Argo CD or Flux, is not explicitly named in the transcript, the principle remains) to continuously synchronize the desired state defined in Git with the actual state of the Kubernetes clusters. This approach eliminates manual kubectl apply operations, ensuring immutability, auditability, and consistency across environments. For instance, if a data center in Frankfurt were to experience an outage, the Git-backed configurations allow for rapid spinning up of new clusters and immediate workload recovery by reapplying the known good state.

1. Curated Namespaces, Not Clusters: Instead of providing each product team with its own Kubernetes cluster, Novous offers curated, isolated namespaces within shared, managed clusters. This design choice simplifies infrastructure management for the platform team while providing product teams with a secure, logically separated environment. These namespaces come pre-integrated with essential services.

1. Integrated Secret Management: To further simplify the developer experience and enhance security, the platform includes its own secret management solution. This abstracts the complexities of handling sensitive data, allowing users to focus on their application logic without needing deep knowledge of Kubernetes secrets or external secret stores. The solution is built "on top to support our users," making it simple and easy for them.

1. Multi-Environment Deployment: Novous is engineered for deployment across various environments. This includes traditional data centers, public cloud regions, and critical factory edge sites. For factory deployments, the platform runs on OpenShift clusters on vSphere. The choice of OpenShift, a commercial Kubernetes distribution, likely provides additional enterprise features, security, and support for the demanding industrial environments. The redundancy offered by Kubernetes is a key selling point for applications running in factories that operate continuously.

1. "LEGO Bricks" Approach: The platform itself is constructed using a modular, composable "LEGO bricks" philosophy. This implies that various components and services are integrated to form a cohesive platform, allowing for flexibility and future expansion. While specific "bricks" beyond secret management are not detailed, this approach suggests a microservices-like architecture for the platform's internal components.

By providing a highly opinionated, GitOps-driven, and abstracted Kubernetes experience, the Novous Container Platform not only reduces the operational burden on individual product teams but also enforces best practices for deployment, security, and resilience across the entire digital organization. The focus on abstracting the underlying Kubernetes complexity ensures that developers, regardless of their Kubernetes expertise, can quickly deploy and manage their applications.

Demo / Proof of Concept

▶ Watch: Introducing Novous, the new opinionated container platform (6:05)

The talk itself did not feature a live, technical demonstration of the Novous Container Platform in action, nor did it present a traditional proof of concept. Instead, the speakers described the platform's capabilities and the strategies employed for its adoption.

However, the LEGO Group's approach to demonstrating value and proving the platform's efficacy was through onboarding workshops. These workshops served as a practical "proof of concept" for potential users. They were meticulously designed to allow engineers, even those with no prior Kubernetes knowledge, to get a production-ready application up and running on the Novous platform from code in less than an hour. This rapid onboarding experience effectively showcased the platform's ease of use and immediate value, acting as a powerful internal marketing tool. Initially conducted as guided online or in-person sessions, these workshops were later converted into self-hosted templates, allowing teams to request a workshop environment and follow a guide independently, further scaling the "proof of concept" delivery. While not a live demo during the conference, these workshops represent the LEGO Group's practical demonstration of their platform's capabilities to their internal stakeholders.

Defensive Implications

▶ Watch: Platform's core principle: Enforcing GitOps for resiliency (6:20)

The LEGO Group's adoption of the Novous Container Platform, with its highly opinionated and GitOps-driven architecture, carries significant defensive implications that enhance the overall security posture and operational resilience of the organization.

1. Standardization and Reduced Attack Surface: By consolidating numerous fragmented Kubernetes implementations into a single, managed platform, the LEGO Group drastically reduces the overall attack surface. Instead of 50+ diverse and potentially inconsistently configured clusters, security teams can focus their efforts on securing and monitoring one well-defined platform. This standardization minimizes configuration drift, ensures consistent security policies, and simplifies vulnerability management.

1. Enhanced Auditability and Compliance: The enforcement of GitOps means that all changes to application deployments and infrastructure configurations are version-controlled and traceable within Git. This provides an immutable audit trail, making it significantly easier to track who made what changes, when, and why. This level of auditability is crucial for compliance requirements (e.g., PCI DSS, GDPR) and for post-incident analysis. The speakers explicitly mentioned that running on Novous allows teams to "check off this entire list of compliance points," which is a major selling point for users and a huge defensive benefit for the company.

1. Improved Resiliency and Disaster Recovery: The GitOps model, where the desired state is declared in Git, inherently boosts resilience. In the event of a catastrophic failure (e.g., a data center outage), new infrastructure can be provisioned, and applications can be rapidly redeployed to a known good state by simply applying the Git repository. This reduces recovery time objectives (RTO) and recovery point objectives (RPO), ensuring business continuity.

1. Centralized Security Controls: The Novous platform provides curated, isolated namespaces rather than individual clusters. This allows the platform team to implement and enforce centralized security controls, such as network policies, role-based access control (RBAC), and admission controllers, consistently across all tenants. The integrated secret management solution ensures sensitive data is handled securely and uniformly, reducing the risk of hardcoded credentials or insecure secret storage.

1. Proactive Vulnerability Management: A standardized platform enables more efficient and effective vulnerability scanning and patching. The platform team can implement automated scanning for known vulnerabilities in container images and Kubernetes components. When vulnerabilities are discovered, patches can be applied consistently across the entire platform, rather than relying on individual product teams to update their disparate environments.

1. Reduced Human Error: The GitOps model minimizes manual interventions, thereby reducing the potential for human error in configurations that could lead to security vulnerabilities or operational outages. Automation ensures that deployments adhere to predefined security best practices.

In essence, the Novous Container Platform transforms a potentially chaotic and insecure landscape into a well-governed, resilient, and defensible environment. Defenders can leverage the platform's inherent properties to achieve a higher level of security assurance than would be possible with fragmented, self-managed solutions.

Key Takeaways

The LEGO Group's journey with the Novous Container Platform offers invaluable lessons for any organization embarking on or struggling with platform engineering initiatives. Here are the key takeaways:

• Embrace a "Sales Mindset" for Internal Adoption: Building a great platform isn't enough; you must actively "sell" its value to internal product teams. This involves proactive engagement, clear communication, and demonstrating how the platform solves their pain points and simplifies their work, while maintaining empathy for their challenges.
• Highly Opinionated Platforms Drive Consistency and Resilience: Enforcing strict architectural patterns, such as GitOps, is crucial for achieving consistency, auditability, and robust disaster recovery capabilities. While initially challenging for some users, the long-term benefits in terms of reliability and security are substantial.
• Understand and Cater to User Archetypes: Recognize that different user teams have varying needs, skill levels, and willingness to adapt. Tailor migration strategies (e.g., "lift," "experimenter," "custom feature") to accommodate these differences, ensuring a smoother transition and higher adoption rates.
• Prioritize and Invest Heavily in User Support: Building strong, personal connections with users and providing accessible, responsive support is paramount. The "support rotation" and "rent an engineer" programs exemplify how investing in support fosters trust, gathers crucial feedback, and ultimately reduces the overall need for reactive support by catching issues early.
• Abstract Complexity, Provide Value: The platform should abstract away the underlying complexities of Kubernetes, offering a curated experience (e.g., isolated namespaces, integrated secret management) that empowers developers to focus on their applications rather than infrastructure minutiae. Compliance sign-offs are a significant value proposition that simplifies developers' work.
• Empower Your Platform Team: Structure the platform team to minimize bureaucratic overhead and empower engineers to make decisions. The LEGO Group's model of empowered pairs and minimal fixed meetings frees up valuable time for user interaction and platform innovation.

About the Speaker(s)

Thomas Øther Rasmussen is a Platform Engineer at the LEGO Group, with a deep personal and professional connection to the brand. His journey with LEGO bricks began at a very young age, and his passion for combining computers with LEGO evolved into a career six years ago when he joined the company. Thomas has held various platform engineering roles within the LEGO Group, bringing a wealth of experience in building and scaling digital infrastructure. He is a key driver behind the Novous Container Platform, focusing on its strategic direction and the technical implementation of its core principles.

Paul Farver is a Platform Engineer in the LEGO container platform team. He describes creativity as a significant part of his personality and his family's, finding a perfect fit in the creative challenges of Kubernetes and LEGO. Paul's role involves hands-on work with the platform, and he played a crucial part in developing and implementing the strategies for user adoption and support, including the support rotation and "rent an engineer" program. His perspective on fostering user connections and empathy is central to the platform's success.