Level Up Your CI/CD: Building a secure pipeline with OSS

Level Up Your CI/CD: Building a secure pipeline with OSS

Speakers: Andoni Alonso Fernández; Paco Sanchez (Progress)

Conference: Cloud Village

YouTube: https://www.youtube.com/watch?v=MLfOlO3Vjeg

Overview

In this comprehensive talk titled "Level Up Your CI/CD: Building a secure pipeline with OSS" at Cloud Village, Andoni Alonso Fernández and Paco Sanchez, both formerly working together and now with Paco at Progress, delivered a practical workshop on integrating robust security measures into modern Continuous Integration/Continuous Delivery (CI/CD) pipelines. The session emphasized the critical importance of a "shift left" security approach, advocating for the early detection and remediation of vulnerabilities in the software development lifecycle (SDLC). The speakers provided a detailed blueprint for what they term a "perfect pipeline," outlining a series of security scans and checks that span the entire deployment process, from pre-commit to post-deployment runtime analysis.

The core objective of the talk was to equip attendees with the knowledge and open-source tools necessary to construct highly secure and efficient CI/CD pipelines. By demonstrating practical examples and offering two alternative open-source tools for each security stage, Alonso Fernández and Sanchez illustrated how to embed security seamlessly without significantly impeding development velocity. This approach not only prevents supply chain compromises and ensures compliance but also empowers development teams to build inherently secure products, moving away from a reactive "security says no" mentality to one of proactive enablement.

The relevance of this talk is underscored by the escalating frequency and impact of supply chain attacks and data breaches, making CI/CD pipelines a prime target for adversaries. By focusing on open-source solutions, the speakers made advanced pipeline security accessible to a broader audience, demonstrating that robust defenses do not necessarily require expensive commercial tools. The workshop's hands-on nature, coupled with a repository of examples, provided a tangible framework for organizations to fortify their development processes against both external threats and internal misconfigurations.

Background

▶ Watch: Introduction and talk agenda overview (0:59)

The modern software development landscape is increasingly dominated by CI/CD pipelines, which represent an automated, streamlined approach to software delivery. At its core, CI/CD aims to accelerate the Software Development Life Cycle (SDLC) by automating the building, testing, and deployment of applications. Continuous Integration (CI) involves developers merging code frequently into a central repository, followed by automated builds and tests. Continuous Delivery (CD) or Continuous Deployment ensures that all changes can be reliably and rapidly released to production environments.

However, this acceleration brings inherent security challenges. The pipeline itself, with its elevated access to source code, secrets, and infrastructure, becomes a critical attack surface. A compromise here can lead to widespread data breaches, unauthorized access, and supply chain attacks, as tragically demonstrated by numerous high-profile incidents in recent months. The speakers highlighted that the fundamental principle driving pipeline security is to "find vulnerabilities when they are cheapest and easiest to fix." This concept, often referred to as "shift left" security, advocates for integrating security checks as early as possible in the development process, dramatically reducing the cost and effort of remediation compared to discovering issues later in the cycle, such as during a penetration test or, worse, after a production incident.

Beyond preventing external attacks, robust pipeline security also fosters a culture of continuous security. It enables development teams to build secure products by default, rather than encountering security as an afterthought or a bottleneck. This proactive stance, as emphasized by the speakers, ensures compliance with regulatory standards (e.g., PCI) and helps prevent internal issues, such as accidental hardcoded secrets or misconfigured deployments. The challenge lies in integrating these security steps efficiently, avoiding excessive pipeline execution times that could frustrate developers and undermine the very agility CI/CD aims to achieve. The talk built upon the existing understanding that while CI/CD optimizes speed and reliability, security must be an embedded, continuous component, not an optional add-on.

Key Findings

▶ Watch: CI/CD/CS explained: find vulnerabilities cheapest to fix (2:00)

The talk's primary "key finding" is the comprehensive architectural blueprint for a "perfect pipeline" that systematically embeds security throughout the entire software delivery lifecycle. This blueprint, visualized as a three-bucket model, serves as a practical framework for organizations to implement continuous security. The buckets are:

1. Pre-Deploy: Focuses on early detection, including pipeline security scans, code scans, secret scans, and Infrastructure as Code (IaC) scans. The emphasis here is on preventing insecure configurations and vulnerabilities from even entering the build process.
2. Build and Deploy: Integrates security checks during the build phase, primarily through container scans, and ensures secure deployment of infrastructure.
3. Post-Deploy: Addresses the dynamic and runtime aspects with runtime infrastructure scans and end-to-end tests, catching issues that might emerge after deployment or due to configuration drift.

Another significant finding highlighted by the speakers is the nuanced and often problematic application of Artificial Intelligence (AI) in security scanning today. While acknowledging its future potential, they caution against directly using large language models (LLMs) for primary vulnerability scanning. Their key observations regarding current AI limitations include:

• Non-deterministic results: AI scans can return different findings each time, making them unreliable for consistent security enforcement.
• High cost: Current AI solutions for code scanning are often expensive both in terms of computational resources and monetary cost.
• Performance issues: AI-driven scans are not yet optimal in terms of speed and efficiency.
• Auditability concerns: The "black box" nature of some AI models makes it difficult to audit and verify their findings for compliance purposes.

Instead, they suggest that AI is currently more effective as a post-processing step, where it can analyze and confirm the exploitability or context of vulnerabilities identified by traditional, deterministic security tools.

Finally, the talk underscored that even seemingly innocuous elements within a CI/CD setup, such as unpinned third-party actions or overly permissive OIDC trust policies, can introduce severe vulnerabilities. The CTF example, where a GitHub Actions workflow allowed any fork from a specific user and branch to assume a highly privileged AWS role, served as a stark demonstration of how subtle misconfigurations in pipeline security can lead to P0 vulnerabilities and full production compromise. This emphasizes that the security of the pipeline itself is the baseline for all subsequent security measures.

Technical Deep Dive

▶ Watch: Why pipeline security is crucial: cost, supply chain, compliance (4:00)

The core of the talk revolved around constructing a "perfect pipeline" by integrating various security stages. This pipeline is broadly categorized into three phases: pre-deploy, build/deploy, and post-deploy. The speakers provided concrete examples and open-source tool alternatives for each step, primarily using GitHub Actions as the CI/CD platform, while noting that the principles are platform-agnostic.

1. Pipeline Security

This foundational step focuses on securing the CI/CD platform and its configurations. As the "baseline" of the SDLC, the pipeline often has elevated access to secrets, code, and infrastructure. Common vulnerabilities include:

• Hardcoded secrets: Directly embedding credentials in pipeline scripts or repositories.
• Excessive permissions: Over-granting access to pipeline jobs or service accounts, enabling developers or attackers to bypass security controls.
• Untrusted actions/modules: Using third-party components (e.g., GitHub Actions, Terraform modules) that are unverified, unpinned to specific versions, or potentially malicious. A critical vulnerability arises when actions are not pinned to a specific commit SHA, allowing maintainers to introduce breaking changes or malicious code into a widely used action, which then automatically runs in dependent pipelines.
• Insecure triggers: Misconfigured repository settings that allow unreviewed code (e.g., from pull requests) to trigger sensitive pipelines, potentially granting access to internal networks or resources.

A compelling example from a Cloud Village CTF highlighted an OIDC (OpenID Connect) trust policy vulnerability. A Terraform configuration for an AWS role granted sts:AssumeRoleWithWebIdentity permissions to any repository named "warden-roo" on a specific branch, regardless of the owner. This meant an attacker could simply fork the "warden-roo" repository, create the specified branch, and then use a GitHub Actions workflow to assume the highly privileged AWS role, gaining unauthorized access to production. The fix involves explicitly defining the repository owner in the OIDC condition. The speakers also recommended crawler, an open-source scanner for GitHub settings, to identify misconfigurations like unprotected main branches or direct pushes to master.

2. Secret Detection

The objective here is to prevent sensitive data from being committed to repositories. Despite widespread awareness, secrets continue to leak. GitGuardian reports indicate common leaked secrets include database connections, AWS credentials, GitHub access tokens, and, more recently, OpenAI and Anthropic API keys.

Beyond traditional secrets, the talk emphasized detecting other sensitive data:

• Webhooks: Slack webhooks, for instance, can be used to impersonate users and facilitate phishing.
• Cloud Account IDs: While not secrets, AWS Account IDs can be used for enumeration of resources, making an attacker's job easier. Tools like aws-i.com (similar to Shodan for public AWS resources) demonstrate their utility.
• Internal Documents/PII: These should never reside in code repositories.

Tools like GitLeaks and TruffleHog were suggested. A practical demonstration involved a canary token embedded in the repository. Once committed, the token triggered alerts within minutes, showcasing the speed at which leaked secrets are exploited. The recommended fix is simply removing the secret from the repository and ensuring it is managed securely (e.g., via a secret manager).

3. Infrastructure as Code (IaC) Scan

IaC scans aim to identify misconfigurations and vulnerabilities in infrastructure definitions (e.g., Terraform, CloudFormation, Kubernetes manifests) before deployment. This approach treats infrastructure like any other software, enabling versioning, peer review, and automated testing.

Common IaC issues include:

• Access Control: Overly permissive IAM policies or security groups (e.g., open to 0.0.0.0/0).
• Encryption: Lack of encryption at rest for storage buckets.
• Network Security: Exposing services to the internet unnecessarily.
• Compliance/Governance: Missing logging, alerting, or specific resource tags.

Recommendations include encrypting remote backends, using state locking, marking sensitive data in outputs, and pinning provider versions. A crucial technical point was the Terraform plan remote code execution vulnerability. While terraform plan is generally considered safe, certain data sources (e.g., data "external" or data "template_file") can execute arbitrary code during the planning phase. If a third-party module contains such a resource, it could compromise the CI/CD environment. The speakers pointed to resources like "Living Off The Pipeline" for a comprehensive list of tools with similar RCE capabilities. Tools demonstrated for IaC scanning included Trivy and Checkov. A specific vulnerability highlighted was an AWS Security Group ingress rule open to 0.0.0.0/0 for an ECS task.

4. Container Scan

This step involves scanning container images (e.g., Docker, OCI) after they are built but before deployment. The goal is to detect vulnerabilities within the image's filesystem, dependencies, and configuration.

Common container issues:

• Vulnerable Dependencies: Outdated libraries or packages with known CVEs.
• Secrets: Residual secrets accidentally baked into the image layers.
• Excessive Privileges: Running containers as root or with unnecessary capabilities.

If an attacker compromises a container, excessive privileges or access to secrets within the image can lead to further system compromise. The default Dockerfile often uses root, making it a prime target. The demonstration used Trivy and Grype to scan a Node.js application container. The scans revealed numerous vulnerabilities due to an extremely old Node.js base image and outdated npm dependencies. The fix involved simply bumping the Node.js version to a more recent, secure release.

5. Runtime Infrastructure Scan

This is the final security step in the "perfect pipeline," performed after infrastructure and applications are deployed. Its purpose is to detect security issues that IaC scans might miss, such as:

• Configuration Drift: Manual changes made directly in the cloud console that deviate from IaC definitions.
• Dynamic Issues: Vulnerabilities that only manifest at runtime or are discovered through dynamic analysis.
• Provider Problems: Unexpected behavior from cloud providers.

While many organizations schedule these scans or perform one-time assessments, integrating them into the pipeline ensures continuous validation of the deployed environment's security posture. This step essentially acts as a continuous Cloud Security Posture Management (CSPM) check. The speakers demonstrated using Prowler and ScoutSuite against a live AWS account. Due to the nature of default AWS configurations often being "open," numerous findings were expected, although the role provided for the workshop was read-only to prevent unauthorized modifications. This step highlights the ongoing need for vigilance even after successful deployment.

Demo / Proof of Concept

▶ Watch: Don't forget internal threats and private repositories (6:00)

The talk was structured as a hands-on workshop, encouraging attendees to fork a provided GitHub repository and actively participate in building a secure CI/CD pipeline using open-source tools. For each security step, the speakers presented two alternative tools and guided participants through enabling the corresponding GitHub Actions workflow.

1. Pipeline Security:

• Tools: gitleaks (for general pipeline configuration issues, though primarily a secret scanner) and trivy (for misconfigurations in YAML files).
• Vulnerability Demonstrated: An unpinned GitHub Action (actions/checkout@master). The master branch reference meant that any changes pushed to the action's master branch would automatically execute in the participant's pipeline without explicit review.
• Fix: Pinning the action to a specific commit SHA or a version tag (e.g., actions/checkout@v3).
• Challenge/Extra: A CTF-like challenge involving an OIDC token vulnerability in an AWS IAM role definition, where a permissive trust policy allowed assumption by any repository with a specific name/branch, demonstrating how to gain unauthorized AWS access. Another "extra ball" was crawler, an open-source tool for scanning GitHub organization and repository settings for security misconfigurations.

1. Secret Detection:

• Tools: GitLeaks and TruffleHog.
• Vulnerability Demonstrated: A US canary token (a synthetic credential designed to alert on access) hardcoded directly into the repository.
• Fix: Removing the canary token from the repository. The immediate alerts generated upon the token's public exposure underscored the urgency of secret management.

1. Infrastructure as Code (IaC) Scan:

• Tools: Trivy and Checkov.
• Vulnerability Demonstrated: An AWS Security Group with an ingress rule open to 0.0.0.0/0 (the entire internet) for an ECS task, indicating a critical network exposure.
• Fix: Restricting the ingress rule to specific IP ranges or internal security groups. The speakers noted that trivy for some reason didn't detect this specific issue in their demo, highlighting the need for multiple scanning tools or careful configuration.
• Extra: A terraform plan remote code execution example, showing how seemingly benign terraform plan commands can execute arbitrary code if certain data sources are maliciously crafted within a module.

1. Container Scan:

• Tools: Trivy and Grype.
• Vulnerability Demonstrated: A Docker image built upon an extremely old Node.js base image which contained numerous known vulnerabilities and outdated npm dependencies.
• Fix: Bumping the Node.js base image version to a more recent, patched release.

1. Runtime Infrastructure Scan:

• Tools: Prowler and ScoutSuite.
• Vulnerability Demonstrated: Scanning a live AWS account to identify real-world misconfigurations and non-compliant settings that might have resulted from configuration drift or default insecure settings.
• Fix: (Not directly demonstrated due to the read-only nature of the provided AWS role and the volume of potential findings). The purpose was to show the detection of runtime issues.

The workshop's structure, providing a repository with pre-configured workflows and instructions, allowed attendees to experience the integration of these security tools firsthand, making the technical concepts highly practical and actionable.

Defensive Implications

▶ Watch: The "perfect pipeline" diagram: pre-deploy, build, post-deploy stages (7:15)

The insights from this talk offer several critical defensive implications for organizations aiming to secure their CI/CD pipelines and broader SDLC:

1. Embrace "Shift Left" Security: The most significant takeaway is to integrate security checks as early as possible. This means moving beyond reactive security assessments to embedding automated scans for code, secrets, IaC, and containers within every pull request and commit, making vulnerabilities cheaper and easier to fix.
2. Secure the Pipeline Itself: Treat the CI/CD pipeline as a critical attack surface. Implement rigorous security for the pipeline configuration, including:

• Pinning all third-party actions/modules to specific commit SHAs or immutable versions to prevent supply chain compromises from upstream changes.
• Applying least privilege to pipeline service accounts and roles (e.g., OIDC trust policies) to limit potential damage if compromised.
• Protecting main branches, enforcing code reviews, and preventing direct pushes.
• Scanning pipeline configurations for misconfigurations, using tools like crawler for GitHub settings.

1. Automate Secret Detection and Management: Implement automated secret scanning tools (e.g., GitLeaks, TruffleHog) within the CI/CD pipeline to catch credentials before they are committed. Crucially, educate developers on secure secret management practices, emphasizing the use of dedicated secret managers over hardcoding.
2. Validate Infrastructure as Code (IaC): Integrate IaC scanning tools (e.g., Trivy, Checkov) to proactively identify misconfigurations, overly permissive access controls, and encryption gaps in infrastructure definitions before deployment. Be aware of advanced IaC vulnerabilities like terraform plan remote code execution and vet all third-party modules rigorously.
3. Harden Container Images: Implement container scanning (e.g., Trivy, Grype) to detect vulnerable dependencies, embedded secrets, and excessive privileges in container images. Prioritize using minimal, official, and regularly updated base images (e.g., Alpine versions, up-to-date Node.js images) and build images with least privilege principles (e.g., non-root users).
4. Implement Runtime Infrastructure Monitoring: Extend security beyond deployment with runtime infrastructure scans (e.g., Prowler, ScoutSuite). This helps detect configuration drift, manual changes, and dynamic vulnerabilities that IaC scans might miss, ensuring continuous compliance and security posture management.
5. Optimize for Developer Experience: While adding security steps, be empathetic to developer workflows. Optimize pipeline execution times by parallelizing jobs, running less critical scans on a schedule (e.g., nightly), or conditionally executing jobs based on code changes. The goal is to enable secure development, not hinder it.
6. Strategic Use of AI: Currently, leverage AI as a post-processing step to confirm exploitability or contextualize findings from deterministic security tools, rather than relying on it for primary, non-deterministic vulnerability detection.

By implementing these defensive strategies, organizations can build resilient and secure CI/CD pipelines that protect against a wide range of threats, improve compliance, and foster a proactive security culture.

Key Takeaways

• Shift Left Security is Paramount: Integrating security checks early in the CI/CD pipeline is crucial for cost-effective and efficient vulnerability remediation.
• Secure the Pipeline Itself First: The CI/CD pipeline is a critical attack surface; ensure its configurations, third-party actions (pinning versions), and access controls are rigorously secured.
• Automate Scans Across All Stages: Implement automated secret, IaC, container, and runtime infrastructure scans to provide continuous security coverage throughout the SDLC.
• Open Source Tools Offer Robust Solutions: A wide array of open-source tools (e.g., GitLeaks, Trivy, Checkov, Grype, Prowler) can effectively secure pipelines without commercial lock-in.
• AI for Context, Not Primary Scanning (Yet): While promising, current AI/LLM-based scanning is best used for post-processing and contextualizing findings from deterministic tools, not as a primary vulnerability scanner.
• Prioritize Developer Empathy: Integrate security measures in a way that minimizes friction and avoids slowing down development, fostering adoption and a positive security culture.

About the Speaker(s)

Andoni Alonso Fernández and Paco Sanchez are experienced security professionals who previously worked together on CI/CD security initiatives. Paco Sanchez is currently affiliated with Progress. Their collective expertise lies in building and securing software development pipelines, advocating for "shift left" security principles, and leveraging open-source tools to achieve robust security postures. They are passionate about enabling developers to create secure products efficiently and are actively engaged in the security community, sharing their knowledge through workshops and talks like this one.