Do Your Containers Even Lift – A Hardening Guide for K8s Containers - Cailyn Edwards & Daniel Murphy

Do Your Containers Even Lift – A Hardening Guide for K8s Containers

Speakers: Cailyn Edwards, Security Engineer, Ozero by Octa; Daniel Murphy, Senior Security Engineer, Ozero by Octa

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=lj_qgsb4h38

Overview

In their KubeCon EU talk, "Do Your Containers Even Lift – A Hardening Guide for K8s Containers," Cailyn Edwards and Daniel Murphy, both Senior Security Engineers at Ozero by Octa, present a comprehensive, entry-level guide to securing containerized applications within Kubernetes environments. The presentation, framed around a fitness metaphor, aims to transform a "squishy" and vulnerable container into a "rock-solid, show-worthy" one by systematically addressing common security weaknesses and misconfigurations.

The talk delves into the inherent risks of deploying containers without adequate security considerations, highlighting how easily default configurations or common development practices can expose applications to significant threats. Edwards and Murphy provide a clear breakdown of fundamental container and Kubernetes security concepts before demonstrating practical hardening techniques that cover image creation, Kubernetes deployment configurations, and cluster-level policy enforcement.

This article explores the critical insights and actionable strategies shared by Edwards and Murphy, emphasizing the importance of intentional security practices throughout the container lifecycle. By showcasing a tangible transformation from an insecure "Squishy Bin" to a hardened "Catching Container," the speakers underscore the necessity of proactive security measures to prevent compromise, protect sensitive data, and maintain the integrity of cloud-native applications.

Background

▶ Watch: Introduction, speakers, and talk agenda (0:00)

The Kubernetes and broader software landscape are rife with complex, often overloaded terminology, making a shared understanding of security concepts crucial. Edwards and Murphy initiated their talk by defining key terms: Kubernetes cluster (a collection of worker nodes), node (worker machine), pod (smallest deployable resource in Kubernetes where applications live), and container (a lightweight unit with application code, runtime, libraries, and configuration). They further clarified base images (layered foundational images), dependencies (OS or programming language packages, direct or indirect), vulnerabilities (software weaknesses), exploits (tools to leverage vulnerabilities), and scanners (tools to detect insecure configurations, secrets, or vulnerable packages).

The speakers then presented a stark picture of the prevailing security posture in container ecosystems, citing several alarming statistics from recent studies:

• A 2021 study revealed that nearly half of the official Docker Hub library images contained at least one vulnerability with a Proof of Concept (PoC) exploit, demonstrating practical usability for attackers.
• In 2023, research found that approximately 30% of popular Docker Hub images were based on a parent image outdated by over a month, with 70% of "latest" versions built from parent images with a median staleness of more than 5.5 months. This highlights the dangers of relying on mutable tags like latest.
• Further data from 2020 indicated that the latest tag is often unreliable; nearly 12% of images studied had a latest tag that didn't point to the actual latest version, and about 4% had a gap of more than five versions.
• A 2023 study uncovered over 55,000 secrets across more than 28,000 images on Docker Hub and unsecured public registries. These included TLS and SSH private keys, cloud API keys, and compromised certificates, with hundreds of thousands of hosts still actively using these vulnerable keys to secure services like HTTPS, SSH, and PostgreSQL.
• Most recently, a 2024 study identified over a million exposed container components on the public internet, including Kubelet, the Kubernetes API server, image registries, Docker runtime sockets, and etcd clusters. A significant majority were running outdated versions and using default configurations—a recipe for disaster. Alarmingly, 86% of exposed Kubernetes API servers allowed anonymous authentication, a default setting since version 1.6 if no other authorization mode is explicitly configured. Thousands of these exposed components are already on public IP blocklists, indicating active exploitation by threat actors.

To illustrate these widespread issues, Edwards and Murphy introduced "Squishy Bin," a Flask API application containerized with a simple, yet insecure, Dockerfile and a basic Kubernetes deployment YAML. This "soft, fuzzy" container serves as the baseline for demonstrating how common misconfigurations make applications vulnerable to compromise.

Key Findings

▶ Watch: Container security terms: base images, dependencies, vulnerabilities (3:15)

The initial assessment of "Squishy Bin" using industry-standard security scanning tools immediately revealed critical vulnerabilities and misconfigurations, validating the statistical concerns outlined in the background. Edwards and Murphy employed two open-source tools to analyze their example container:

1. Trivy: An open-source vulnerability scanner from Aqua Security, used to scan the container image for known vulnerabilities and misconfigurations.

• The scan results showed a number of identified vulnerabilities, including a particularly high-severity flaw with a CVSS score of 8.8. This vulnerability was a Remote Code Execution (RCE) issue found in the package management library of the container. Such a high CVSS score indicates a critical vulnerability that could allow an attacker to execute arbitrary code on the container, potentially leading to full system compromise.
• When deployed as an operator within the Kubernetes cluster, Trivy further extended its scanning capabilities to analyze cluster components, workloads, underlying infrastructure, and Role-Based Access Control (RBAC) configurations. The initial Trivy config report for "Squishy Bin"'s deployment highlighted several critical misconfigurations:
• An immutable root file system was not enforced, allowing potential runtime modifications by an attacker.
• A lack of specific security contexts meant the container was running with excessive privileges.
• The ability to escalate privileges was present, a common attack vector for gaining root access.
• The container was running as the root user, which grants maximum permissions and significantly increases the impact of any compromise.

1. TruffleHog: Another open-source tool from Truffle Security, designed to scan for secrets lurking within various data sources, including container images.

• TruffleHog quickly identified a "super secret password" directly embedded within the pip config file of the "Squishy Bin" image. This demonstrated a classic security oversight where sensitive information is inadvertently baked into container layers, making it discoverable and exploitable.

These findings collectively painted a clear picture of "Squishy Bin" as a highly vulnerable target. The presence of a high-severity RCE, exposed secrets, and fundamental misconfigurations like running as root and lacking robust security contexts, underscored the urgent need for comprehensive hardening measures. This initial state served as a compelling "before" snapshot for the subsequent "montage" of security improvements.

Technical Deep Dive

▶ Watch: Startling statistics on container image vulnerabilities (4:07)

The core of Edwards and Murphy's talk focused on the practical steps taken to transform "Squishy Bin" into a "Catching Container." This involved a multi-pronged approach, addressing vulnerabilities at the image build stage, Kubernetes deployment configuration, and cluster-level policy enforcement.

Quick Wins: Patching and Image Management

The initial hardening steps were "quick wins" centered on image hygiene:

• Applying Scanner Recommendations: Acting on the output from tools like Trivy, the first step was to update the image version to patch identified vulnerabilities.
• Adhering to Semantic Versioning (SemVer): When updating dependencies, it's crucial to understand SemVer (Major.Minor.Patch) to anticipate potential breaking changes. A change in the major version number often indicates backward incompatibility.
• Reviewing Release Notes: For critical dependencies, release notes provide vital information about non-breaking but impactful changes.
• Avoiding Alpha/Beta in Production: Emphasizing stability, the speakers warned against deploying pre-release software versions to production environments.
• Thorough Application Testing: Despite best intentions, updates can introduce regressions, so testing applications post-update is non-negotiable.
• Leveraging End-of-Life Information: Tools like endoflife.date provide crucial information on project support dates, helping teams avoid using outdated or unsupported software.
• Automating Dependency Updates: Open-source tools like Renovate can automate various types of dependency updates, including security-relevant ones, reducing manual overhead and human error. Edwards even shared a personal anecdote of accidentally pulling an alpha version during demo prep, reinforcing the need for automated guardrails.

Kubernetes Component Hardening

The next set of changes focused on securing the Kubernetes deployment itself:

• Namespace Isolation: A fundamental Kubernetes security strategy, simply placing "Squishy Bin" into its own namespace provided a boundary for resource sharing and allowed for bespoke security policies for the workload. This prevents applications from interfering with or gaining unauthorized access to resources in other namespaces.
• Pod Security Admission Controller (PSAC): Native to Kubernetes since version 1.25, PSAC allows cluster administrators to enforce Pod Security Standards at the namespace level. The speakers demonstrated configuring namespace labels to enforce, audit, and warn at the baseline security level.
• enforce: Blocks the creation of pods that violate the specified security standard.
• audit: Records violations in the audit logs but allows the pod to be created.
• warn: Displays a warning to the user upon pod creation but still allows it.
• The demo showed how a "risky business pod" with an unconfined security context (the most permissive) was initially allowed with a warning under warn but subsequently denied when enforce was active.
• Admission Controllers (Validating and Mutating): Beyond PSAC, more granular control can be achieved using admission controllers through tools like Gatekeeper (based on Open Policy Agent - OPA), Kyverno, or Kubewarden.
• Validating Admission Controllers: These perform checks on resource changes and validate them against predefined policies. The example used Gatekeeper to enforce a specific seccomp (secure computing mode) profile. A policy was applied requiring pods to use either runtime/default (the basic default profile) or a localhost profile (a custom profile available on the node disk). A pod attempting to run with an unconfined seccomp profile was denied. Importantly, the speakers highlighted the ability to create exemptions for specific workloads that legitimately require more permissive security policies, allowing for "secure by default" while maintaining flexibility.
• Mutating Admission Controllers: These controllers can modify resources based on criteria. The demo showcased a policy that automatically sets the seccomp profile to runtime/default if it's not explicitly defined in a pod's configuration. This is a powerful mechanism to ensure minimum security requirements are met without burdening developers with manual configuration.

Dockerfile Refinements

Significant improvements were made to the Dockerfile to enhance image security:

• Build-Time Secrets: Instead of copying a pip config file (which contained a secret) into the image and then attempting to delete it in a subsequent layer (leaving it in the build history), the team leveraged Docker's --mount=type=secret feature. This mounts the secret only during the build process, making it available at a specified path without embedding it into any image layer, thus preventing secret leakage.
• Non-Root User: The USER instruction was added to the Dockerfile, ensuring the application runs as a non-root user. This significantly limits the potential damage an attacker can inflict if they compromise the container.
• Health Checks: A HEALTHCHECK instruction was added, allowing the container runtime to periodically check the application's health and ensure it's still responsive, improving reliability and detectability of issues.
• .dockerignore: Though not explicitly shown in the demo, the importance of a .dockerignore file was stressed. This file prevents unnecessary or sensitive files (like .env files containing secrets, or build artifacts like virtual environments or node_modules) from being copied into the container image.
• Trusted Host Configuration: A specific trusted-host artifact related to the demo environment was mentioned, advising general users to ignore it as it facilitates insecure TLS communication.

Kubernetes Deployment Configuration Hardening

The final set of technical changes focused on the Kubernetes deployment YAML:

• Image Pinning with Digests: Instead of relying on mutable tags (e.g., myimage:latest), the image was pinned to a specific SHA256 digest. This cryptographic hash uniquely identifies the exact image content. If a threat actor were to gain push access to an image registry and poison an image under an existing tag, a deployment pinned to a digest would fail, preventing the deployment of the compromised image.
• High User and Group IDs: Running containers with very high user and group IDs (runAsUser, runAsGroup) helps avoid overlap with host system UIDs/GIDs, reducing the risk of privilege escalation attacks that rely on matching host user contexts.
• Read-Only Root Filesystem: Setting readOnlyRootFilesystem: true prevents the application from writing to its own root filesystem at runtime. This restricts an attacker's ability to modify core system files or install malicious software post-compromise.
• Blocking Privilege Escalation: allowPrivilegeEscalation: false explicitly prevents processes within the container from gaining more privileges than their parent process, mitigating a common Linux privilege escalation technique.
• Dropping Capabilities: Linux capabilities grant fine-grained permissions beyond basic user/group IDs. The deployment was configured to drop unnecessary capabilities (e.g., NET_RAW, SYS_ADMIN) and add only those strictly required by the application. This adheres to the principle of least privilege.
• Seccomp Profile Application: Explicitly applying a seccomp profile (e.g., runtime/default) limits the system calls a container can make, significantly reducing the attack surface.
• Resource Limits: Setting resources.limits for CPU and memory prevents resource exhaustion attacks (e.g., ReDoS or cryptocurrency miners) that could otherwise consume cluster resources, impacting availability and incurring unexpected costs.

These detailed technical interventions, applied methodically, transformed "Squishy Bin" from a highly vulnerable application into a significantly more resilient "Catching Container."

Demo / Proof of Concept

▶ Watch: Discovery of valid secrets in public images (6:00)

The culmination of the talk was a compelling "before and after" demonstration, showcasing the tangible security improvements achieved by applying the hardening techniques. The "Squishy Bin" container, initially riddled with vulnerabilities and misconfigurations, was transformed into the "Catching Container," a much more secure entity.

The process involved repeating the initial security scans with Trivy and TruffleHog after all the Dockerfile and Kubernetes deployment changes had been implemented.

Before (Squishy Bin):

• Trivy Vulnerability Scan: Revealed multiple known vulnerable packages, most notably the high-severity CVSS 8.8 Remote Code Execution (RCE) vulnerability in the package management library.
• TruffleHog Secret Scan: Successfully identified a clear-text "super secret password" embedded within the pip config file, a critical secret leakage.
• Trivy Operator (CIS Benchmark Scan): The cluster-level scan of the "Squishy Bin" deployment against the CIS Kubernetes Benchmark reported numerous failures. These included issues like an unenforced immutable root filesystem, missing security contexts, potential for privilege escalation, and the container running as the root user.

After (Catching Container):

• Trivy Vulnerability Scan: Post-hardening, the Trivy scan reported no more known vulnerable packages installed in the image. This was achieved by updating dependencies and ensuring a minimal, patched base image.
• TruffleHog Secret Scan: The TruffleHog scan found no secrets within the "Catching Container" image. The use of build-time secrets mounting ensured that sensitive information was never baked into the image layers.
• Trivy Operator (CIS Benchmark Scan): The CIS benchmark scan of the "Catching Container"'s namespace and deployment returned a clean report, with all checks passing. This indicated successful implementation of security contexts, non-root user execution, read-only filesystem, dropped capabilities, seccomp profiles, and appropriate resource limits.

The speakers acknowledged that achieving a perfect 100% pass rate on a CIS benchmark across an entire multi-tenant cluster might not always be feasible due to diverse workload requirements and inherent compromises. However, they emphasized that applying even a subset of these controls significantly improves the security posture: "Don't let perfect be the enemy of good here. Applying some of these controls is still going to put you in a much better place than doing nothing at all." The demonstration effectively illustrated that a systematic approach to container and Kubernetes hardening yields substantial and measurable security benefits.

Defensive Implications

▶ Watch: Millions of exposed Kubernetes API servers and Kublets (7:00)

Edwards and Murphy transitioned from technical demonstrations to broader strategic advice for security professionals, emphasizing how to scale these hardening practices across an organization. Their recommendations centered on three key pillars: establishing secure defaults, understanding developer needs, and fostering a strong security culture.

Paved Paths and Guard Rails

The fundamental principle is to make secure choices the easy, default choices, thereby creating "paved paths" for developers and "guard rails" to prevent missteps.

• Security as Default: Security engineers should focus on building systems where secure configurations are the norm, rather than manually reviewing every Dockerfile or Kubernetes YAML.
• Private Registry of Blessed Images: Maintain an internal registry of vetted, hardened base images that teams can confidently use. This ensures a secure starting point for all applications.
• Regular Patching and Upgrades: Implement processes for continuous patching and upgrading of images and dependencies. This includes leveraging tools like Renovate and monitoring endoflife.date to stay ahead of known vulnerabilities.
• Enforce Slim and Minimal Images: Encourage or enforce the use of minimal base images (e.g., Alpine) and ensure that only necessary dependencies are included and vetted. This reduces the attack surface by minimizing the software installed in the container.
• Automated Scanning in CI/CD Pipelines: Integrate vulnerability, secret, and misconfiguration scanners (like Trivy and TruffleHog) directly into the CI/CD pipeline. Configure these pipelines to alert teams when containers need patching or upgrading, preventing insecure images from reaching production. Security should be a continuous process, not a one-time review.

Understanding Customer (Developer) Use Cases

Effective security must be empathetic to the needs of developers, the "customers" of security teams. Reducing friction and integrating security into existing workflows is paramount.

• Reduce Friction: Security controls should complement, not hinder, development processes. Meeting teams where they are and building upon existing workflows increases adoption.
• Communicate Early and Often: Involve developers in security discussions from the outset. Deploying controls without prior communication can disrupt work and lead to resentment.
• Threat Modeling Sessions: Running threat modeling sessions with development teams is highly effective. It allows security teams to deeply understand the application's architecture and data flow, while empowering developers to "think like the baddies." This collaborative approach fosters buy-in and leads to more relevant and actionable security recommendations.

Building a Security Culture

Beyond technical controls and workflow integration, cultivating a company-wide security culture is essential for long-term success.

• Educate and Encourage Buy-in: Use threat modeling and other interactive sessions to educate teams about security risks and demonstrate the value of security practices. When teams understand the "why," they are more likely to enthusiastically adopt security measures.
• Show Security's Value: Security, when successful, is often "silent" – it prevents incidents that never happen. Security teams need to actively communicate their contributions and impact to maintain visibility and support.
• Security Advocate/Champion Programs: Establish formal or informal programs where members from various teams are educated about security. These "champions" can then represent security interests within their teams, provide early feedback on new initiatives, and act as a bridge between development and security. This helps security teams get involved earlier in the development lifecycle, preventing last-minute "no-sorry" situations.
• Security Educational Sessions and Events: Organize "Cyber Month" events, Capture The Flag (CTF) competitions, and regular educational sessions to make security engaging and accessible to everyone, fostering a sense of shared responsibility.

By implementing these defensive strategies, organizations can build robust, scalable security programs that protect their containerized applications while empowering their development teams.

Key Takeaways

• Containers are Inherently Vulnerable Without Intentional Hardening: Default configurations and common development practices often lead to critical security flaws, including high-severity vulnerabilities and exposed secrets.
• Automated Scanning is Non-Negotiable: Tools like Trivy for vulnerability and misconfiguration scanning, and TruffleHog for secret detection, are essential for identifying security issues early in the development lifecycle.
• Leverage Kubernetes Native Security Features: The Pod Security Admission Controller (PSAC) and robust Admission Controllers (e.g., Gatekeeper) provide powerful, declarative mechanisms for enforcing security policies at the cluster and namespace levels.
• Implement Secure Dockerfile Practices: Adopt best practices like using build-time secrets, running applications as non-root users, adding health checks, and leveraging .dockerignore to create inherently more secure container images.
• Harden Kubernetes Deployment Configurations: Pin images to SHA256 digests, enforce read-only root filesystems, block privilege escalation, drop unnecessary Linux capabilities, apply seccomp profiles, and set resource limits to significantly reduce the attack surface and impact of a compromise.
• Build a Proactive Security Culture: Establish "paved paths" for secure development, integrate automated scanning into CI/CD, understand developer use cases, and foster buy-in through education and advocate programs to embed security throughout the organization.

About the Speaker(s)

Cailyn Edwards is a Security Engineer at Ozero by Octa. Beyond her work, she is a dedicated co-chair of Kubernetes SIG Security, actively contributing to the security posture of the Kubernetes project. Edwards is also a CNCF ambassador, passionate about open-source and cloud-native technologies, and a self-proclaimed enthusiastic dog mom and Sounder fan.

Daniel Murphy is a Senior Security Engineer at Ozero by Octa. He shares a passion for security with his colleague Cailyn Edwards. Outside of his professional life, Daniel enjoys photography, coffee, outdoor adventures, and is also a massive Sounder fan.