From Noise to Notes: Orchestrating SAST with Developers through AI-Driven Remediation

From Noise to Notes: Orchestrating SAST with Developers through AI-Driven Remediation

Speakers: Adrián Puente Z., Principal Security Engineer, Remountley

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=F9pltUu6bBc

Overview

In this insightful talk at BSides SF, Adrián Puente Z., a Principal Security Engineer at Remountley, presented a compelling framework for transforming the often-frustrating experience of Static Application Security Testing (SAST) into a harmonious and effective security program. Titled "From Noise to Notes," Puente’s presentation tackles the pervasive issue of overwhelming false positives and developer friction that plagues many SAST implementations, offering a practical, AI-driven remediation strategy. The core of his approach lies in deeply integrating security practices with developer workflows, prioritizing findings based on business risk, and leveraging artificial intelligence to automate and accelerate the remediation process.

Puente’s work addresses a critical challenge in modern DevSecOps: how to "shift left" security effectively without alienating engineering teams. He argues that the traditional SAST promise often devolves into a "cacophony" of unmanageable alerts, leading to developer apathy and a breakdown of trust. By focusing on high-confidence, high-impact findings and providing actionable, context-rich recommendations, his methodology aims to turn security from a bottleneck into an enabler. This talk is crucial for any organization struggling with SAST adoption, offering a tangible playbook for reducing security debt and fostering a culture of secure development, ultimately cutting remediation times from weeks or months down to mere hours or days.

Background

▶ Watch: The SAST problem: Noise, false positives, frustration. (0:30)

The concept of Static Application Security Testing (SAST) is foundational to modern application security. As Adrián Puente explained, SAST involves analyzing an application's source code, bytecode, or binary code without actually executing it. Its primary goal is to identify security vulnerabilities early in the Software Development Life Cycle (SDLC), a practice commonly known as "shift left." The promise of SAST is enticing: catch issues during development, empower developers to fix them before deployment, and reduce the cost and risk associated with vulnerabilities discovered later.

However, the reality of SAST often diverges sharply from this ideal. Puente vividly described the common scenario: organizations plug a SAST tool into their CI/CD pipeline or GitHub organization only to be deluged by thousands of findings. He cited an alarming statistic from his own experience: out of 5,000 initial findings, a staggering 70% were false positives. This "cacophony" of alerts quickly overwhelms developers, who, faced with irrelevant or unactionable issues, begin to ignore all security warnings, including critical ones. Security teams, instead of being seen as enablers, become a "nuisance," generating backlog and fostering resentment. Puente inherited a SAST program that was already "broken," a testament to the widespread struggle to make these tools effective. The initial chaos meant that the investment in SAST was making things worse, not better, failing to deliver on its core promise of early detection and developer empowerment. This necessitated a complete overhaul of the process, moving from a state of "noise" to a more structured "notes" approach, ultimately aiming for harmony between security and development teams.

Key Findings

▶ Watch: Talk Agenda: From Noise to Notes with AI. (3:00)

Puente's journey from SAST chaos to an orchestrated, AI-driven remediation process yielded several key findings and strategic shifts:

1. Initial Overwhelm and Developer Apathy: The rollout of SAST to 1,000 repositories initially resulted in approximately 4,000 findings, with a high percentage of false positives. This led to developers ignoring all alerts, including critical ones, highlighting a severe trust deficit and a lack of actionable signal. The core problem was that security was not addressing what developers cared about: "Is this real? Can I fix it? Will this block my deploy?"

1. Strategic Noise Reduction: The first critical step was to filter the overwhelming number of alerts. By focusing purely on critical and high-severity findings with high confidence (leveraging specialized rules from Sentry Pro), the team drastically reduced the alert surface. Furthermore, implementing Sentry memories allowed the team to contextually flag and ignore known false positive patterns, such as localhost usage during development. These measures collectively reduced the initial 5,000 findings to a more manageable 800.

1. Risk-Based Prioritization: Beyond severity, the team prioritized findings based on their business risk. As a fintech company, this meant focusing on systems handling PII (Personally Identifiable Information), regulatory data, payment processing, and those impacting customer experience or business availability. This approach, while covering only 40% of the organization's repositories, addressed 95% of the real business risk problems, proving that strategic focus trumps broad but unfocused coverage.

1. Integration into Developer Workflows: Security became an integral part of the development process. Findings were delivered directly in Pull Request (PR) comments with live feedback and fix recommendations. CI/CD checks blocked critical issues, but a "break glass" exception mechanism ensured emergency deployments were not unduly hampered. Integration with Slack and Jira provided real-time updates and facilitated issue ownership, fostering developer engagement and turning security into an enabler.

1. AI-Driven Remediation as a Workforce Multiplier: The most transformative finding was the successful implementation of AI-driven remediation, dubbed "bio security patching" or "slop security." By aggregating similar vulnerabilities, cloning affected code, and using Large Language Models (LLMs) with "laser-focused prompts" to generate specific patches, the team drastically reduced remediation time. This process cut the average time to remediate high-severity vulnerabilities from 130 days to just 17 days, with one security engineer able to address 78 high-severity vulnerabilities across three PRs in a single week. AI transformed security from merely identifying problems to actively proposing solutions, closing the last mile of remediation.

Technical Deep Dive

▶ Watch: Introducing AI-driven remediation for fast fixes. (3:50)

The journey from a chaotic SAST implementation to an efficient, AI-augmented security program involved several distinct technical and process-oriented phases.

Choosing the Right Tool and Initial Rollout

The first critical step was selecting a SAST tool that aligned with the organization's unique needs. The team opted for Sentry Pro due to its speed, flexibility, native CI/CD integration, and crucially, its customizable rules. This last feature proved invaluable, allowing the security team to create specific rules based on findings from their bug bounty program and pentesting team. By identifying common vulnerability patterns reported by external researchers, they could proactively scan their codebase for similar issues, reducing the financial outlay on bug bounties by addressing these patterns internally.

The rollout began with a cautious Proof of Concept (POC) involving 100 repositories. This initial phase demonstrated success, leading to an expansion to 1,000 repositories. However, this scale-up immediately exposed the inherent challenges of traditional SAST, generating approximately 4,000 findings and confirming the "noise" problem.

Orchestrating Noise Reduction and Prioritization

To convert this "noise" into actionable "signal," a multi-pronged strategy was deployed:

1. Severity Filtering: The team made a decisive move to stop chasing low-severity findings and even most medium ones. The primary focus shifted exclusively to critical and high-severity issues. This was bolstered by leveraging Sentry Pro's specialized rules, which provided a higher confidence score for true positives, a critical factor for a small security team needing to scale efficiently. This alone significantly reduced the attack surface requiring immediate attention.

1. Sentry Memories for False Positive Reduction: A key technical feature utilized was Sentry Memories. This functionality allowed the security team to add context to specific findings or entire repositories, enabling the tool to intelligently ignore known false positive patterns. A prime example cited was the prevalence of HTTP findings related to developers using localhost during development. By flagging this pattern in Sentry Memories, these irrelevant alerts were automatically suppressed, dramatically improving the signal-to-noise ratio.

1. Risk-Based Prioritization of Systems: Recognizing that not all code is created equal in terms of business impact, the team implemented a risk-based prioritization model for selecting which repositories to scan and prioritize findings within them. Given the organization's fintech nature, emphasis was placed on:

• Compliance-scoped systems
• Systems handling PII (Personally Identifiable Information) and regulatory data
• Payment processing systems
• Applications critical to customer experience
• Systems with high availability tier requirements.

This strategic focus meant that while only 40% of the entire code base was covered by SAST, 95% of the organization's actual business risk was effectively addressed. This was a "game-changer," allowing the small security team to make a disproportionately large impact.

Embedding Security into Developer Workflows

A crucial element of success was making security an integrated, developer-friendly part of the engineering process:

• Live Feedback in Pull Requests: SAST findings, complete with recommendations, were integrated directly into PR comments. This provided immediate, contextual feedback during code review, allowing developers to address issues before they merged.
• CI/CD Blocking with "Break Glass": Critical security findings were configured to block CI/CD pipelines. However, recognizing the realities of software development, a "break glass" exception mechanism was established. This allowed engineers to bypass blocks during emergencies (e.g., large-scale system outages) while ensuring these exceptions were tracked and addressed post-incident. This demonstrated a collaborative approach rather than an adversarial one.
• Real-time Updates via Slack and Jira: Integrations with Slack and Jira ensured real-time severity updates and facilitated the creation of structured tickets. This allowed for better tracking, ownership, and collaboration on security issues.
• Actionable Reporting: Findings were presented in a "developer-friendly" manner: clear, actionable messages that detailed the problem, its location, the associated risk, and specific recommendations for remediation. This transformed security reports from mere alerts into practical guidance. The speaker also mentioned using CVSS 4.0 in conjunction with LLMs to generate detailed risk reports, adding objective context to findings and reducing pushback from developers.

AI-Driven Remediation: "Bio Security Patching"

The most innovative and impactful technical contribution was the implementation of AI-driven remediation, which Puente referred to as "bio security patching" or "slop security." This initiative directly tackles the scalability problem of manual remediation.

The traditional remediation workflow is cumbersome: an engineer reads a finding, researches solutions (often via Google or Stack Overflow), understands the code context, writes a fix, tests it, and deploys. This process, even for a single vulnerability, can take hours and is not scalable across hundreds or thousands of findings.

The AI-driven workflow automates significant portions of this process:

1. Triaging and Aggregation: After initial noise reduction, high-priority findings are aggregated. Context switching is expensive for engineers, so grouping similar vulnerabilities (e.g., all SQL injection findings in a specific repository) into a single aggregated ticket streamlines the process.
2. Code Cloning and LLM Integration: The affected code is cloned from the repository. An LLM (Large Language Model) is then used to generate a patch. Puente mentioned using tools like Augment with Cloud (likely referring to cloud-based LLM services or platforms) and general-purpose LLMs such as GPT-3.5 or Opus.
3. Laser-Focused Prompting: A critical aspect is the use of "laser-focused prompts" when interacting with the LLM. Generic prompts can lead to the LLM making unwanted changes or even deleting unrelated code. The prompt must be highly specific, instructing the LLM to remediate only the identified vulnerability in a precise section of the code.
4. Pull Request Generation and Validation: The AI-generated patch is then used to create a commit and a Pull Request (PR). Crucially, the PR description explicitly states that it is a "security recommendation" and requires an engineer to validate that it does not break any existing functionality. This shifts the ultimate responsibility for code integrity back to the system owner, mitigating liability for the security team.
5. Workforce Multiplier: Puente described creating a "GPT companion" using OpenAI's ChatGPT (presumably with an enterprise license) to assist junior security engineers or contractors. This specialized agent guides users through the process of creating Jira tickets from Sentry findings, cloning code, and using specific prompts with other LLMs for patch generation. This transforms AI into a powerful "workforce multiplier," enabling less experienced personnel to perform tasks typically requiring staff-level expertise.

The results were dramatic: remediation time for high-severity vulnerabilities was reduced from an average of 130 days to 17 days. In one instance, a single security engineer, using this method, was able to remediate approximately 78 high-severity vulnerabilities across just three Pull Requests in a week. This enabled the team to clear critical and high-severity backlogs and move on to addressing medium-severity issues, demonstrating the profound impact of AI in scaling security operations.

Demo / Proof of Concept

▶ Watch: Choosing SAST tool and initial rollout challenges. (4:30)

While the talk did not feature a live, interactive demonstration of the SAST tools or the AI remediation process, the speaker provided a detailed conceptual walkthrough of the methodology and the achieved outcomes. The "Proof of Concept" was presented through the speaker's own real-world experience and the quantifiable results obtained by his team.

The workflow described for AI-driven remediation serves as a strong conceptual demonstration. It outlines a step-by-step process: from triaging and aggregating vulnerabilities, to cloning affected code, using an LLM with a "laser-focused prompt" to generate a patch, and finally creating a Pull Request (PR) for engineer validation. The speaker articulated how this process, despite not being shown live, translated into tangible improvements, such as remediating 78 high-severity vulnerabilities with just three PRs in a week, and reducing the average remediation time from 130 days to 17 days. The use of a "GPT companion" for junior engineers also illustrates a practical application of AI to streamline the workflow, even if the companion itself wasn't demonstrated in action.

Defensive Implications

▶ Watch: Building developer trust by prioritizing critical findings. (6:00)

The strategies and findings presented by Adrián Puente offer several critical defensive implications for organizations looking to enhance their application security posture:

1. Prioritize Signal Over Noise: Defenders must aggressively filter SAST findings. Ignoring low-severity, low-confidence, or known false positives is not just about convenience; it's about preserving developer trust and ensuring that real threats receive the attention they deserve. Implementing severity filtering and using contextual tools like Sentry Memories are essential first steps to create an actionable security queue.

1. Risk-Based Security is Paramount: Blanket SAST coverage is inefficient and often ineffective. Security teams should prioritize scanning and remediation efforts based on the business risk associated with specific applications and data. Identifying systems handling sensitive data (PII, payment info), critical business functions, or those with high regulatory compliance requirements ensures that resources are allocated where they can have the greatest impact on reducing actual organizational risk. This also provides clear metrics to communicate value to executives.

1. Embrace Developer Experience (DevEx): Security programs will fail without developer buy-in. Defenders must shift from an adversarial stance ("security yelling at developers") to a collaborative one. This means integrating security feedback directly into developer workflows (e.g., PR comments, CI/CD), providing clear, actionable recommendations with context and risk explanations, and offering "break glass" mechanisms for emergencies. Security should be an enabler, helping developers build secure code by default, rather than a blocker.

1. Leverage AI as a Workforce Multiplier for Remediation: The most significant defensive implication is the strategic adoption of AI for vulnerability remediation. Security teams are often small and overwhelmed. AI-driven "bio security patching" can drastically reduce the time and effort required to fix vulnerabilities, transforming security from a reporting function to an active remediation partner. Experimenting with LLMs for patch generation, guided by laser-focused prompts, can accelerate the elimination of security debt, freeing up human security engineers for more complex architectural and strategic tasks.

1. Measure Outcomes, Not Just Findings: The success of a SAST program should not be judged by the sheer number of findings, but by tangible outcomes like developer adoption rates, fix rates, and time to remediation. Tracking these metrics provides a clear indication of program effectiveness and allows for continuous improvement. Regularly auditing the "noise" in SAST reports and refining rules based on these metrics is crucial for sustained success.

1. Foster a Culture of Collaboration: Building bridges with engineering teams, understanding their workflows, hopes, and dreams, is non-negotiable. This involves finding technical debt teams, offering self-service SAST access where appropriate, and working together to integrate security into existing processes rather than imposing new ones. This collaborative approach builds trust and transforms security into a valued partner in the software development lifecycle.

Key Takeaways

• Developer Empathy is Non-Negotiable: SAST adoption hinges on working with developers, not against them. Prioritize positive developer experience, build trust, and integrate security into their existing workflows to foster collaboration.
• Reduce Noise to Find Signal: Drown out the "cacophony" of alerts by ruthlessly prioritizing. Focus only on high-confidence, high-severity findings that pose actual business risk. Use tools like Sentry Memories to filter out known false positives and consider objective risk assessment frameworks like CVSS 4.0 to provide clear, contextualized reports.
• Embed Security into the Workflow: Deliver security feedback where developers work – in PR comments, CI/CD checks (with "break glass" options), and integrated Slack/Jira channels. Actionable, context-rich recommendations are key to immediate remediation.
• AI is Your Workforce Multiplier: Leverage Large Language Models (LLMs) for AI-driven remediation ("bio security patching"). This dramatically reduces the time to fix vulnerabilities, transforming security from merely reporting issues to actively proposing and generating fixes. Use "laser-focused prompts" to ensure accuracy and create "GPT companions" to empower junior staff.
• Start Small, Build Credibility: When launching or overhauling a SAST program, begin with a small scope (e.g., 100 repositories). Demonstrate value and success, then expand incrementally. This approach builds internal credibility and justifies requests for more budget and resources.
• Measure Outcomes, Not Raw Counts: Shift metrics from the number of findings to the adoption rate, fix rate, and time to remediation. Continuously audit your SAST noise, re-evaluate rules, and turn off low-severity alerts to maintain a high-signal environment.

About the Speaker(s)

Adrián Puente Z. is a Principal Security Engineer at Remountley, based in the San Francisco Bay Area. With over 20 years of experience in cybersecurity, Adrián's career path is diverse and extensive. He began as a system administrator, a role that profoundly influenced his journey after experiencing a hack firsthand, which ignited his passion for security. He then transitioned into consulting, spending eight years as a pentester across three countries and two languages, gaining deep insights into offensive security. Currently, Adrián specializes in application security and DevSecOps, with a laser focus on fostering collaboration and making security work with engineering teams, rather than against them. Outside of his professional endeavors, Adrián enjoys sci-fi and gaming as his sanity checks. He is active on social media, where he can be found as @ch0ks.