The Pivotal Role of Large Language Models in Extracting Actionable TTP Attack Chains

The Pivotal Role of Large Language Models in Extracting Actionable TTP Attack Chains

Speakers: Lauri and Para, 360

Conference: Black Hat Asia

YouTube: https://www.youtube.com/watch?v=7S3OSvWXP0I

Overview

In an era where cybersecurity threats are constantly evolving, the ability to rapidly understand, extract, and operationalize adversary Tactics, Techniques, and Procedures (TTPs) is paramount for effective defense. This talk, delivered by Lauri and Para from 360 at Black Hat Asia, addresses the critical challenges in converting human-readable threat intelligence reports into actionable TTP attack chains. It introduces an innovative solution leveraging Large Language Models (LLMs) in conjunction with a Knowledge Graph-enhanced Retrieval Augmented Generation (KG-RAG) framework to automate this complex process.

The presentation highlights the shift from passive, signature-based defense to proactive, TTP-driven threat hunting and detection. While most valuable TTPs are buried within unstructured human-readable reports, their manual extraction and conversion into executable formats are time-consuming, prone to human error, and highly dependent on individual domain expertise. The proposed KG-RAG approach aims to overcome these limitations by providing a practical pipeline for high-precision TTP extraction, systematic chain enrichment, and the generation of diverse actionable intelligence artifacts, thus significantly enhancing the utility of threat intelligence for real-time defense and red/blue team operations.

Background

▶ Watch: Talk overview, agenda, and speaker introduction (0:00)

The foundation of modern cybersecurity defense increasingly relies on a deep understanding of adversary behaviors, often categorized using the MITRE ATT&CK® framework. This framework provides a standardized language for describing Tactics (the adversary's objective), Techniques (how they achieve the objective), and Procedures (specific implementations of techniques). For example, "Initial Access" is a Tactic, "Process Injection" is a Technique, and "APT29 used Mimikatz to exploit domain controllers via ZeroLogon vulnerability" is a Procedure. This granularity is crucial for developing robust detection models and shifting security operations centers (SOCs) from reactive, alert-based responses to proactive, intelligence-driven defense.

Despite the recognized significance of TTPs, their conversion into actionable intelligence faces several formidable challenges. Threat intelligence reports, while rich in detail, are primarily designed for human consumption. They lack structured TTP data, often contain abstract or subjective descriptions, and may selectively focus on novel or critical behaviors, potentially omitting crucial environmental or execution details. This unstructured nature makes automated extraction difficult. Furthermore, the subsequent step of converting these extracted TTPs into actionable formats—such as detection rules, red team scripts, or penetration testing scenarios—is predominantly a manual effort. This manual process is not only extremely time-consuming but also heavily dependent on the specialized domain knowledge and skills of individual analysts, leading to inconsistencies and bottlenecks in operationalizing intelligence. The need for a scalable, automated solution to bridge this gap is therefore pressing.

Key Findings

▶ Watch: Significance of converting TTP intelligence into actionable defense (4:00)

The core contribution of this talk is the introduction of a comprehensive solution pipeline designed to automate the extraction and actionable conversion of TTP attack chains from unstructured threat intelligence reports. This pipeline comprises four key modules: an Attack Path Generator, a TTP Chain Generator, a TTP Chain Enricher, and an Actionable TTP Generator. The speakers specifically focused on the latter three, emphasizing the critical role of KG-RAG in achieving superior performance.

A central finding is that while traditional machine learning (ML), deep learning (DL), and even fine-tuned pre-trained LLMs like BERT have advanced TTP extraction, they suffer from significant limitations. Traditional methods require extensive pre-processing and feature engineering, while fine-tuned models often lack current domain knowledge and can be biased by their training data. Generic Generative LLMs, despite their vast knowledge, struggle with out-of-date information regarding continuously updated TTPs and are prone to hallucination, generating fictional or irrelevant TTPs. For example, an LLM might correctly identify a technique but misclassify its associated tactic or completely miss newly released techniques.

To address these shortcomings, the speakers presented a KG-RAG approach that significantly outperforms previous methods. This hybrid approach integrates continuously updated domain knowledge from a Knowledge Graph into the LLM's retrieval and generation process, thereby enabling the LLM to identify new techniques and providing accurate context to drastically reduce hallucinations. The KG-RAG framework demonstrated the best performance in TTP extraction across different evaluation criteria, as shown in their comparative analysis. Furthermore, the talk highlighted the ability of KG reasoning to systematically enrich incomplete TTP chains by intelligently linking possible TTPs based on context and metadata, ensuring both completeness and realism of the attack chain. Finally, the solution successfully converts these structured TTPs into diverse actionable intelligence formats, including social engineering texts, lure generation scripts, and executable commands for penetration testing platforms like Metasploit Framework, and even the generation of new Metasploit modules when existing ones are insufficient.

Technical Deep Dive

▶ Watch: Proposed solution: Overview of four core modules (6:40)

The proposed solution is structured as a robust pipeline with various input and output capabilities, underpinned by a sophisticated architecture. Inputs can range from diverse reports (e.g., APT advisories, minor reports) to internet URLs, PDFs, Word documents, and plain text. The outputs are highly practical: social engineering phishing corpuses, lure generation scripts, and Metasploit Framework-based module execution commands. Crucially, the pipeline leverages a Knowledge Graph to store TTP data with designed relationships, featuring a schema similar to the STIX (Structured Threat Information eXpression) format, which ensures structured and interconnected intelligence.

The talk provided a detailed historical context of TTP extraction, tracing its evolution through three main stages:

1. Traditional Machine Learning/Deep Learning: Early approaches utilized models like Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTMs) for processing long sequences. However, these methods were constrained by their reliance on extensive pre-processing, manual feature engineering, and limited generalization capabilities.
2. Fine-tuned Pre-trained Models: The advent of models like BERT marked a significant leap, enabling deep contextual understanding of text. Fine-tuning these models for TTP extraction involved preparing high-quality training datasets. Despite their advancements, limitations persisted, notably a lack of up-to-date domain knowledge and inherent biases from their training data, making them struggle with new or evolving TTPs.
3. Generative LLMs (without RAG): These models possess vast knowledge of human language, code, and technical concepts, allowing them to parse complex attack behaviors and offer natural, adaptive output. Through proper prompt engineering, they can generate structured TTP information. However, their primary drawbacks are reliance on static training data (making them unaware of continuously updated MITRE ATT&CK techniques) and the tendency to hallucinate or generate fictional TTPs. An example cited was an LLM misclassifying a technique's tactic (e.g., "Hijacking Exchange Flow" incorrectly assigned to "Initial Access" instead of "Defense Evasion") or completely missing a recently released technique like mutual exclusion.

To overcome the inherent limitations of standalone LLMs, the speakers introduced their KG-RAG based TTP extraction approach. This method involves two primary tasks:

1. Generating Baseline Candidates: A fine-tuned pre-trained model (a BERT variant, optimized for the best results among options like RoBERTa) first generates an initial set of candidate TTPs. Improvements in this stage include distinguishing primary and secondary techniques to refine procedural differentiation and extracting TTPs from diverse artifacts such as tools and command-line arguments for more comprehensive coverage. The training data for this model focuses on critical techniques (out of 700+ in MITRE ATT&CK) and requires manual curation for accuracy.
2. Retrieving Similar TTPs and LLM-based Extraction/Re-ranking: This is the core of the KG-RAG process. The candidate TTPs from the fine-tuned model are combined with similar TTPs and related metadata retrieved from a vector store. This enriched input is then fed into an LLM via carefully crafted prompts, instructing it to extract TTPs and output them in a structured JSON format, using the candidate and retrieved TTPs as references. A subsequent re-ranking module provides additional context, such as detailed definitions of techniques and correct examples, to help the LLM assess the extracted TTPs and filter out irrelevant or incorrect ones. The Knowledge Graph plays a crucial role here by storing intricate relationships between tactics, techniques, procedures, artifacts (CVEs, tools, assets), which are then used to enrich the prompts and provide the necessary context to the LLM, ensuring the output is not only accurate but also rich in relational data.

Beyond initial extraction, the KG-RAG based TTP enrichment module addresses the challenge of incomplete or selectively reported TTP chains. Its workflow consists of three tasks:

1. Report Metadata Analysis: It analyzes report metadata (adversary information, attack path, attack vector, regions) to retrieve possible TTPs from the Knowledge Graph. This includes TTPs used by the same adversary, targeting similar vectors or regions, and popular TTPs relevant to the context.
2. Gap Identification: It identifies missing TTPs in the current chain using pre-defined "building blocks" (e.g., "Is there a description of initial access?", "How did the adversary gain higher permissions?").
3. Contextual Reasoning: Using the gathered context, the system reasons about the most appropriate TTPs to fill the gaps, considering factors like permission changes and asset changes. This reasoning engine, based on the Apache Jena reasoning engine, leverages a lightweight domain language and custom AI tools to achieve remarkable results in completing TTP chains.

Demo / Proof of Concept

▶ Watch: Solution architecture diagram and pipeline overview (8:00)

The speakers provided compelling demonstrations of the solution's capabilities across various actionable intelligence generation scenarios, showcasing its practical utility.

1. Social Engineering Text Generation: The system takes a human-readable chunk of a report describing a phishing attack. It first performs TTP instruction to extract structured TTPs, including descriptions, techniques used, specified artifacts, and original email information. From this structured data, the Actionable TTP Generator then produces multiple tailored social engineering phishing corpuses that align with the extracted TTPs, ready for use in security awareness training or red team engagements.

1. Lure Generation Script: Similar to the social engineering example, a report chunk describing a specific lure technique is processed. The system extracts the structured TTP details, and subsequently generates a lure generation script. This script, when executed, successfully creates a lure that precisely matches the TTPs identified from the original report, demonstrating the ability to automate the creation of malicious artifacts.

1. Metasploit Framework Use: This demonstration highlighted the system's ability to interface with penetration testing platforms.

• An original human-readable report chunk, detailing a CVE exploit and target assets, is processed. The KG-RAG system extracts comprehensive TTPs, including descriptions, artifacts, the specific CVE, target assets, and notably, permission change information—a critical detail for reasoning about TTP chain completion.
• Through prompt engineering, the LLM is instructed to provide keywords to search for existing modules within the Metasploit Framework console.
• The LLM then selects the most appropriate module based on its rank scores and descriptions.
• To ensure accuracy and executability, the LLM reads the selected module's documentation and generates the precise running commands.
• A particularly innovative aspect was demonstrated for scenarios where no suitable existing Metasploit module is found. In such cases, the LLM is capable of generating a new Metasploit module from scratch, adhering to Metasploit APIs, which can then be loaded and executed. This capability significantly extends the reach of automated actionable intelligence, allowing for rapid adaptation to novel vulnerabilities or techniques.

The speakers reported that all three types of generations—social engineering texts, lure generation scripts, and Metasploit framework commands/modules—achieved "good results," validating the practical effectiveness of their KG-RAG solution.

Defensive Implications

▶ Watch: Generative AI and LLMs for advanced TTP extraction (11:00)

The solution presented in this talk carries profound implications for enhancing cybersecurity defenses, enabling a more proactive, automated, and intelligent approach to threat management.

Firstly, by providing a practical pipeline to automatically convert human-readable reports into actionable TTP attack chains, security teams can dramatically accelerate their response times. Instead of manual analysis taking hours or days, the system can rapidly transform raw intelligence into formats suitable for real-time detection in SIEM (Security Information and Event Management) or XDR (Extended Detection and Response) solutions. This enables the detection of code-level attacks and rapid responses by SOC analysts.

Secondly, the KG-RAG paradigm, with its focus on integrating continuously updated domain knowledge and providing accurate context, significantly improves the precision and completeness of TTP extraction. This minimizes the risk of missing critical adversary behaviors due to outdated threat intelligence or LLM hallucinations. For defenders, this means more reliable and comprehensive TTPs to build robust detection rules and behavioral analytics.

Thirdly, the capability to systematically enrich TTP attack chains by linking possible, often-missed TTPs ensures that defensive strategies are built upon a complete understanding of adversary campaigns. By reasoning about permission changes and asset movements, the system helps defenders anticipate subsequent adversary actions, allowing for the deployment of layered defenses that cover the entire attack lifecycle, not just isolated events.

Finally, the generation of diverse actionable intelligence formats—from social engineering templates to executable Metasploit Framework commands and new modules—directly empowers both blue teams and red teams. Blue teams can use these outputs to simulate attacks, validate existing defenses, and develop new detection logic. Red teams can leverage these automated scripts and modules for penetration testing and adversary emulation, providing valuable improvement metrics for defense systems. This automation fosters a continuous improvement cycle, making defenses more resilient against evolving threats and enabling organizations to move closer to a truly proactive security posture.

Key Takeaways

• Automated Intelligence Pipeline: The solution offers a practical, automated pipeline for converting human-readable threat intelligence reports into actionable TTP attack chains, significantly enhancing the operational application of threat intelligence.
• KG-RAG for Precision TTP Extraction: It introduces a novel KG-RAG paradigm that combines lightweight pre-trained model predictions with a Knowledge Graph to ensure high-precision TTP extraction, effectively leveraging up-to-date data and mitigating LLM hallucinations.
• Systematic TTP Chain Enrichment: The approach proposes a method leveraging KG reasoning (powered by Apache Jena) to systematically enrich TTP attack chains by intelligently linking possible, often missed, TTPs, ensuring comprehensive and realistic attack scenarios.
• Diverse Actionable Intelligence Generation: The system, based on KG-RAG, converts structured TTPs into various actionable intelligence formats, including social engineering texts, lure generation scripts, and executable commands for penetration testing platforms like Metasploit Framework, demonstrating its versatility for red and blue team operations.
• Overcoming LLM Limitations: By integrating continuous domain knowledge and providing accurate context, the KG-RAG framework effectively addresses key limitations of standalone LLMs, such as outdated information and the tendency for hallucination, making them more reliable for cybersecurity applications.

About the Speaker(s)

Lauri and Para are researchers from the company 360. Their work focuses on leveraging advanced technologies like Large Language Models and Knowledge Graphs to enhance threat intelligence processing and make it more actionable for cybersecurity defense.