From Idea to Open-Source: Building CNA-GURU, a Generative AI Assistant for Security Advisories

From Idea to Open-Source: Building CNA-GURU, a Generative AI Assistant for Security Advisories

Speakers: Ryan, Tech Lead for AWS Security Outreach, AWS

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=OYH8qbUueaI

Overview

In the dynamic and ever-expanding landscape of cybersecurity, the process of scoring vulnerabilities and drafting security advisories is a critical yet often challenging and time-consuming endeavor. Ryan, a Tech Lead for AWS Security Outreach, presented CNA-GURU (also known as Chat CVE), an innovative open-source generative AI assistant designed to streamline and democratize this complex task. Driven by a personal desire to improve efficiency and address the heightened demands of the 2024 CNA directives, Ryan embarked on a journey to build a tool that could effectively scale vulnerability assessment capabilities.

The talk highlighted the core problem: the difficulty of training human analysts in a niche skill set, exacerbated by the new regulations requiring more organizations to engage in Coordinated Vulnerability Disclosure (CVD) work. CNA-GURU emerges as a practical solution, aiming to lower the barrier to entry for new analysts and empower experienced ones by providing an intelligent, contextual assistant. This article delves into the project's evolution, technical underpinnings, key findings, and profound implications for security teams grappling with the escalating volume and complexity of vulnerabilities.

Ultimately, CNA-GURU is not positioned as a replacement for human expertise but rather as a powerful force multiplier. By automating repetitive tasks, providing immediate contextual information, and assisting in the drafting of critical security documentation, the tool allows analysts to focus on higher-value decision-making. Its open-source nature further underscores a commitment to community collaboration, offering a foundational framework that organizations can adapt and extend to meet their specific security advisory needs.

Background

▶ Watch: Introduction and motivation for building CNA-GURU. (0:00)

The genesis of CNA-GURU lies in a common pain point experienced by security teams globally: the arduous and time-intensive process of vulnerability assessment, specifically the determination of Common Weakness Enumeration (CWE) and Common Vulnerability Scoring System (CVSS) scores. This task requires a nuanced understanding of vulnerability characteristics, potential impact, and adherence to evolving industry standards. Compounding this challenge is the significant effort required to train new security analysts in these specialized skills. As Ryan eloquently put it, "scoring vulnerabilities is hard, and training humans is harder."

The urgency for a scalable solution was amplified by the new CNA (CVE Numbering Authority) directives that came into effect in 2024. These directives expanded the scope of organizations expected to perform CNA-type work, thrusting many companies and software builders into roles for which their teams were often unprepared, lacking the necessary training or skill sets. This regulatory shift created a pressing need for tools that could lower the bar to entry, enabling a broader range of personnel to contribute effectively to vulnerability disclosure processes.

The initial spark for CNA-GURU ignited at a previous VulnCon conference. Ryan, along with other industry peers like Sarah Evans, Casmir Schultz, and Christian Sver, shared common frustrations regarding the difficulty of translating and training engineers in vulnerability scoring. An hour of collaborative brainstorming cemented the idea, leading Ryan to immediately begin building a prototype. The objective was clear: create a portable, easy-to-deploy system that could process natural language vulnerability descriptions, leveraging a knowledge base to provide contextual answers and assist in critical decision-making, thereby empowering humans rather than replacing them.

Key Findings

▶ Watch: Origin of CNA-GURU: VulnCon brainstorming session. (2:08)

The development and deployment of CNA-GURU yielded several significant findings, demonstrating its potential to revolutionize vulnerability assessment workflows:

1. Dramatic Reduction in Analysis Time: One of the most impactful findings was the tool's ability to drastically cut down the time required for vulnerability analysis. Previously, a trained analyst, even one familiar with Google searches for CWEs, would spend approximately three hours to go from a vulnerability description to a complete CVE or GHSA submission (including CWE determination, CVSS scoring, and a justification write-up). With CNA-GURU, this process was reduced to a mere few minutes, allowing analysts to quickly draft initial responses and justifications.

1. High Accuracy and Consistency: Despite concerns about AI "hallucinations," CNA-GURU demonstrated remarkable accuracy. In a sample size of 300 randomly chosen CVEs from the NVD database, the tool achieved roughly 90% accuracy for both CVSS scoring (within a 0.2 delta of the official NVD score) and CWE determination. When the CWE did not match, it often provided a more niche or lower-lineage CWE, which could sometimes be more precise, though occasionally it was "blatantly wrong" due to poor verbiage in the original vulnerability description. This high level of accuracy instills confidence in its utility as an assistive tool.

1. Enhanced Analyst Training and Psychological Safety: The tool proved invaluable for training new analysts. It provides immediate, contextual answers to basic and follow-up questions, eliminating the need for repetitive explanations from senior staff. This fosters a sense of "psychological safety," as junior analysts are less hesitant to ask "dumb questions" to an AI assistant than to a human mentor, enabling them to build a better understanding and formulate more informed questions for their leads.

1. Cost-Effective Evolution of Architecture: The project's evolution highlighted the importance of balancing functionality with operational cost. The initial, feature-rich version (CNA-GURU v1) was effective but expensive. Subsequent iterations, CNA-GURU Jr. and CNA-GURU Esquire, successfully stripped down the architecture by moving from complex Amazon Bedrock knowledge base agents (which incurred costs for OpenSearch, S3, etc.) to a more streamlined local RAG (Retrieval Augmented Generation) system. This significantly reduced costs while maintaining similar accuracy and performance, making the solution more viable for widespread adoption.

1. Portability and Accessibility: The final iteration, CNA-GURU Esquire, running within a Jupyter Notebook, addressed a critical barrier to entry: deployment complexity. By removing the need for users to understand and deploy AWS CDK (Cloud Development Kit) infrastructure, the Esquire version made the tool exceptionally portable and accessible, allowing users to "pick it up, drop it on your computer, load it up, and start kicking off with it." This flexibility is key to its open-source adoption.

Technical Deep Dive

▶ Watch: Deep dive into the technical stack: Bedrock, Streamlit. (4:15)

The technical architecture of CNA-GURU underwent significant evolution, driven by the need to balance performance, accuracy, and cost-effectiveness. The speaker detailed three main iterations, each building upon the lessons learned from its predecessor.

CNA-GURU v1 (The "Island of Misfit Toys")

The initial version was rapidly prototyped by leveraging existing AWS Labs examples in CDK (Cloud Development Kit) and Python, specifically those interacting with chatbots and agent retrieval systems. This allowed Ryan to quickly assemble the desired user experience.

• Core LLM Interaction: The chatbot primarily interacted with Amazon Bedrock, AWS's fully managed service that provides access to foundation models.
• Frontend: A user-friendly, open-source Streamlit application served as the chat interface, making it accessible and familiar.
• Hosting: The application was containerized using Docker and deployed on Amazon ECS (Elastic Container Service), specifically utilizing AWS Fargate for serverless container execution. Fargate was chosen to meet specific feature requirements of ECS at the time of development.
• Knowledge Base: The foundational knowledge for vulnerability scoring was built entirely from publicly available data, including information from CVDs (Coordinated Vulnerability Disclosures), CWE (Common Weakness Enumeration), and NVD (National Vulnerability Database) guidance. This public dataset ensured replicability and transparency.
• Contextual Retrieval: The system used a customized prompt to search its knowledge base with context from user input. This involved an "agent retrieval" mechanism, where the user's natural language vulnerability description was used to match and extract relevant data from the knowledge base, providing the LLM with the necessary context for accurate scoring and explanation.

While effective and accurate, this initial architecture, particularly its reliance on Bedrock knowledge base agents (which involve components like an OpenSearch index and S3 bucket storage), proved to be expensive when scaled to a team of ten people for a month.

CNA-GURU Jr. (Cost Optimization)

To address the cost challenge, a stripped-down version, CNA-GURU Jr., was developed. The primary change was the removal of the expensive Bedrock knowledge base agents.

• Knowledge Base Redesign: Instead of Bedrock's managed knowledge base, CNA-GURU Jr. moved to a strategy that assumed the use of an LLM pre-trained on NVD data. For custom knowledge input, it adopted a local RAG (Retrieval Augmented Generation) system. This approach allowed for the injection of specific organizational or custom vulnerability information without the overhead of the full Bedrock knowledge base infrastructure.
• Prompt Engineering: The same customized prompt, refined through iterative testing, continued to be a critical component, guiding the LLM's responses and ensuring relevant outputs.
• Cost Reduction: This architectural shift significantly reduced operational costs, making the tool more financially viable for broader internal use.

CNA-GURU Esquire (Portability and Ease of Deployment)

The final iteration, CNA-GURU Esquire, aimed to overcome the hurdle of deployment complexity, specifically the need for users to understand and configure AWS CDK.

• Jupyter Notebook Integration: CNA-GURU Esquire runs entirely within a Jupyter Notebook. This design choice provides extreme portability, eliminating the need for users to deploy any additional infrastructure. Users can simply download the notebook and run it locally.
• Retained Core Logic: It maintains the same local RAG system and the refined prompt as CNA-GURU Jr., ensuring similar accuracy and pricing.
• Chatbot Experience: Despite being in a Jupyter Notebook, it still provides a contextual chatbot experience, allowing the AI to retain the entire conversation history for follow-up questions.
• LLM Choice: Ryan confirmed that the internal AWS-approved LLM solution used for this project was Claude, highlighting its effectiveness for vulnerability matching and composition.

Across all iterations, the core mechanism involves taking natural language input, using a customized prompt to query a knowledge base (whether Bedrock-managed or local RAG), and then leveraging the LLM to process this context and generate relevant, accurate responses for CVSS, CWE, and advisory drafting. The emphasis on "guard rails" and prompt engineering is crucial to mitigate AI "hallucinations" and guide the model's behavior, ensuring its responses are aligned with security best practices.

Demo / Proof of Concept

▶ Watch: Key benefit: drastically reducing repetitive training for CVE scoring. (5:55)

The talk included a compelling demonstration of CNA-GURU's capabilities, showcasing its utility for both novice and experienced security analysts. The examples illustrated how the tool facilitates a natural, conversational workflow for vulnerability assessment and advisory creation.

1. Simplistic Questions and General Knowledge:

The demo began by highlighting the tool's ability to answer basic, general knowledge questions related to CNA work. For example, an analyst could ask, "What is a CVE?" or "Explain what a CWE is?" This feature is particularly valuable for training new team members, providing immediate definitions and context without requiring them to consult wikis or interrupt mentors. Ryan emphasized how this eliminates the repetitive task of explaining fundamental concepts.

1. Contextual Follow-up Questions:

A key strength demonstrated was the assistant's capacity to maintain conversation context. When an analyst didn't agree with a CWE determination, they could ask a follow-up question like, "Why did you choose this CWE over another one?" The AI responded by explaining that specific phrases in the human-entered text mapped directly to that CWE. This capability mimics a mentor-mentee interaction, allowing analysts to delve deeper into the reasoning behind the AI's suggestions.

1. CVSS and CWE Determination:

The core functionality revolves around taking a vulnerability description and automatically calculating a CVSS score and determining relevant CWEs. The tool provides the calculated CVSS vector string and score. For CWEs, it offers the top three most relevant or related options based on the description, rather than just one. This approach acknowledges the potential for multiple valid classifications and allows the human analyst to make the final choice. The demo also showed how the tool quotes both the user's input and the knowledge base's relevant snippets to justify its recommendations, fostering transparency and trust.

1. Drafting Security Advisories (CVE and GHSA):

One of the most impressive aspects was CNA-GURU's ability to draft security advisories. Leveraging the conversational context—including previously scored CVSS, determined CWEs, and the original vulnerability description—the analyst could command, "Can you draft a CVE for me?" or "Generate a GHSA draft."

• If insufficient information was provided, the tool would proactively identify the missing attributes (e.g., "You don't have enough information to write the draft. These are the attributes you're missing."), guiding the user on what additional details are needed.
• When enough context was available, CNA-GURU could generate a comprehensive draft, including references, CWEs, mitigation steps, and impact statements. The speaker proudly displayed a screenshot of a full GHSA draft generated from the conversation leading up to that point in the presentation, highlighting its speed and thoroughness.

The demos collectively underscored that CNA-GURU functions as an intelligent assistant that not only provides answers but also guides the user through the complex process of vulnerability assessment and disclosure, making it more efficient and less intimidating.

Defensive Implications

▶ Watch: Example: LLM's CWE choice and follow-up explanation. (7:30)

CNA-GURU offers a multitude of benefits for security defenders, transforming how vulnerability management and disclosure are handled within organizations. Its implications extend beyond mere automation to impact training, quality assurance, and overall operational efficiency.

1. Accelerated Vulnerability Triage and Assessment:

The most immediate defensive implication is the significant reduction in the time required for initial vulnerability analysis. By cutting down the process from three hours to mere minutes, CNA-GURU allows security teams to triage and assess a higher volume of reported vulnerabilities more rapidly. This means faster initial responses, quicker understanding of potential impact, and a more agile security posture, especially critical for organizations dealing with a continuous stream of vulnerability reports.

1. Standardized and Scalable Analyst Training:

For organizations struggling to train new security analysts in the niche skill of vulnerability scoring, CNA-GURU is a game-changer. It provides an on-demand, consistent source of information for basic definitions (like "What is a CWE?") and complex scoring logic. This standardizes the training process, ensures new hires receive consistent guidance, and dramatically lowers the "barrier to entry" for individuals new to CNA work. The ability to ask "dumb questions" to an AI assistant without social pressure fosters a safer learning environment, empowering junior analysts to learn faster and contribute sooner.

1. Enhanced Quality Assurance and Second Opinions:

Even experienced analysts can benefit from CNA-GURU. It serves as an excellent quality assurance (QA) tool, providing a "second opinion" on CVSS scores and CWE determinations. An analyst can compare their manual assessment against the AI's suggestions, validating their findings or prompting further investigation if discrepancies arise. This helps catch potential errors, ensures consistency across assessments, and improves the overall quality of security advisories.

1. Streamlined Security Advisory Creation:

The ability to draft CVEs and GHSAs directly from conversation context is a powerful defensive capability. It alleviates the "blank page syndrome" that often paralyzes analysts when starting a write-up. By generating initial drafts, including references, CWEs, mitigations, and impact statements, CNA-GURU accelerates the entire disclosure process. It also proactively identifies missing information, guiding analysts to gather all necessary details for a complete and accurate advisory.

1. Customization for Organizational Context:

As an open-source project, CNA-GURU offers significant flexibility. Defensive teams can fork the repository and customize its prompts and even its knowledge base to align with their specific risk surfaces, internal policies, and unique operational contexts. This fine-tuning ensures that the AI's recommendations are always relevant to the organization's specific needs, maximizing its utility. For example, a company might adjust the prompt to prioritize certain types of vulnerabilities or risk factors that are more critical to their infrastructure.

1. Empowering Smaller Teams and New CNAs:

The 2024 CNA directives place new burdens on many organizations. CNA-GURU, especially its portable Jupyter Notebook version (Esquire), empowers smaller teams or those new to the CNA program to effectively manage vulnerability disclosures without needing extensive dedicated staff or complex infrastructure. It allows them to "punch above their weight class" by leveraging AI assistance for tasks that would otherwise require highly specialized and scarce human expertise.

In essence, CNA-GURU equips defenders with an intelligent, scalable, and adaptable tool that enhances efficiency, improves accuracy, and lowers the training overhead associated with modern vulnerability management.

Key Takeaways

• AI as an Assistant, Not a Replacement: CNA-GURU's primary goal is to empower and assist human security analysts, significantly reducing repetitive tasks and accelerating workflows, rather than replacing human judgment.
• Drastic Time Savings: The tool cuts vulnerability analysis time from approximately three hours to just minutes, enabling faster triage, scoring, and drafting of security advisories.
• High Accuracy: It achieves roughly 90% accuracy in CVSS scoring (within 0.2 delta) and CWE determination across a sample of 300 CVEs, providing reliable initial assessments.
• Cost-Effective Evolution: The project evolved from an expensive AWS Bedrock-centric architecture to a cost-efficient local RAG system, and finally to a highly portable Jupyter Notebook version (CNA-GURU Esquire), making it accessible to a wider audience.
• Enhanced Training and Psychological Safety: CNA-GURU improves analyst training by providing immediate, contextual answers and fostering a safe environment for asking questions, thereby lowering the barrier to entry for new CNA participants.
• Open-Source and Customizable: As an open-source project, it encourages community contribution, allowing organizations to fork, customize, and fine-tune the tool to align with their specific risk context and operational needs.

About the Speaker(s)

Ryan is the Tech Lead for AWS Security Outreach, a crucial team at AWS responsible for coordinated vulnerability disclosure (CVD). With a passion for improving quality of life through automation and scaling repetitive tasks, Ryan initiated the development of CNA-GURU. His work at AWS involves navigating the complexities of vulnerability management for a large, global organization. He is actively involved in the cloud security community and values collaborative problem-solving, as evidenced by the project's origins in a VulnCon brainstorming session.