Redefining Purple Teaming for Max impact - A Pennington, S Marrone, L Proehl

Redefining Purple Teaming for Max impact

Speakers: Sydney Moroni (Principal Threat Hunter, Splunk); Lauren Pill (Global Head of Detection and Response, Marsh McLennon); Adam Pennington (Manager, MITRE ATT&CK, MITRE Corporation)

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=wU7xaXDupZo

Overview

In an insightful panel discussion at DEF CON, security experts Sydney Moroni, Lauren Pill, and Adam Pennington explored the evolving landscape of purple teaming and adversary emulation. The talk, titled "Redefining Purple Teaming for Max Impact," delved into the critical need for collaboration between red and blue teams, emphasizing a transparent, shared learning process over traditional adversarial engagements. This discussion, originating from the Adversary Village, challenged the common perception that Defcon is solely for offensive security, making a compelling case for the strategic importance of defense and proactive security measures.

The panelists, seasoned blue teamers with extensive experience in incident response, threat hunting, and detection, highlighted how purple teaming bridges the gap between offensive capabilities and defensive readiness. They argued that by aligning red team activities with real-world threat intelligence and adversary behaviors, organizations can achieve more effective security posture improvements. The conversation underscored that while "hacking as hard as you can" might be fun, truly impactful security testing requires a focus on practical, relevant threats that an organization is likely to face.

This talk is particularly significant for organizations seeking to mature their security operations, move beyond basic compliance, and optimize their defensive capabilities against sophisticated threats. It provides practical guidance on how to implement successful purple teaming initiatives, measure their impact, and communicate their value to executive leadership. By advocating for a collaborative and intelligence-driven approach, the speakers presented a roadmap for organizations to not only identify vulnerabilities but also to proactively enhance their ability to detect and respond to genuine attacks.

Background

▶ Watch: Panelist introductions: roles and experience (0:20)

The concept of purple teaming emerged as a necessary evolution in cybersecurity, addressing inherent limitations within traditional red team and blue team operations. Historically, red teams (attackers) and blue teams (defenders) often operated in silos. Red teams would conduct penetration tests or simulated attacks, and then "throw the report over the wall" to the blue team, who would then be tasked with deciphering findings and implementing defenses, often with limited context or direct engagement during the exercise. This lack of transparency and collaboration frequently led to missed learning opportunities, inefficient remediation, and a disconnect between the simulated threats and the blue team's actual detection capabilities.

Adam Pennington shared that even the MITRE ATT&CK® framework, which he now manages, originated from an internal purple teaming need at MITRE. They wanted their red team exercises to be observed by their Security Operations Center (SOC) and needed a common language for the two teams to communicate findings and observations. This internal tool eventually became the globally recognized ATT&CK framework, underscoring the long-standing challenge of effective red-blue communication.

The panelists stressed that simply engaging in "balls to the wall" red teaming, where the goal is solely to find the next big vulnerability or execute a "stunt," often leads to unrealistic scenarios. As Pennington noted, early red team exercises at MITRE sometimes found creative but impractical ways to breach systems, which didn't align with what real-world adversaries were actually doing. This meant organizations weren't testing their defenses against their most likely threats, leading to a misallocation of resources and a false sense of security. Lauren Pill further emphasized this, stating that a red teamer using a custom C2 to exfiltrate data when an organization lacks basic DNS logging for simpler exfiltration methods is not productive. The "cool factor" must be balanced with the "practicality factor" to ensure exercises benefit the organization's overall security posture.

Purple teaming, therefore, is about breaking down these silos, fostering transparency, and creating a continuous feedback loop. It's a process where red and blue teams plan together, execute together, and learn from each other in real-time. This collaborative approach not only improves the skills of individual team members but also raises the overall security maturity level of the entire organization. It ensures that the simulated attacks are relevant, realistic, and directly contribute to enhancing detection and response capabilities against the specific threats an organization faces.

Key Findings

▶ Watch: MITRE ATT&CK's origins in purple teaming (4:00)

The panel discussion illuminated several core findings and contributions to the understanding of effective purple teaming and adversary emulation:

1. Purple Teaming as Transparent Collaboration: The defining characteristic of purple teaming is its transparent, collaborative nature. Both red and blue teams actively work together throughout the entire exercise, from planning to execution, sharing information and learning from each other. This breaks down organizational silos and fosters a collective improvement in security posture.
2. Adversary Emulation Drives Realism: To maximize impact, purple team exercises must be grounded in adversary emulation, meaning they should mimic the tactics, techniques, and procedures (TTPs) of specific, relevant threat actors. This ensures that the testing directly addresses the actual threats an organization is most likely to encounter, rather than generic or unrealistic attack scenarios.
3. Threat Intelligence is Paramount: The selection of which adversaries or TTPs to emulate is critical and must be informed by Cyber Threat Intelligence (CTI). Organizations should leverage their CTI teams, vendor threat profiles, industry peers' experiences, and historical incidents to identify relevant threats. For those without dedicated CTI, resources like the ThaiCERT Threat Actor Encyclopedia can provide a starting point.
4. Metrics are Essential for Proving Value: Demonstrating the value of purple teaming to executive leadership requires robust metrics and narrative reporting. Beyond simple counts of exercises, valuable metrics include the number of new detections created, documentation updates, the range of threat actors covered, and improvements in Mean Time To Detect (MTTD) and Mean Time To Close (MTTC) for remediation efforts.
5. Beware of "100% ATT&CK Coverage" Claims: While MITRE ATT&CK is a valuable framework for measuring defensive coverage, claims of "100% ATT&CK coverage" by vendors are misleading and potentially detrimental. As Adam Pennington highlighted, some ATT&CK techniques are undesirable to defend against (e.g., environmental keying bypasses that could enable malware). A realistic goal is to achieve relevant and prioritized coverage, not complete coverage.
6. Readiness and Gradual Implementation: Organizations need to establish fundamental security capabilities (e.g., robust SOC, incident response plan, comprehensive logging) before embarking on advanced purple teaming or adversary emulation. For less mature organizations, starting small with a few techniques or in a test environment, building trust, and gradually expanding scope is crucial.
7. AI's Emerging Role: Artificial Intelligence is poised to significantly lower the barrier to entry for blue teamers engaging in adversary emulation, augmenting their capabilities by providing situational guidance and accelerating learning. However, current AI is seen as a tool for augmentation, not yet capable of independent or autonomous emulation.

Technical Deep Dive

▶ Watch: Purple teaming raises community skill levels (5:50)

The technical core of effective purple teaming, as articulated by the panelists, revolves around meticulous planning, intelligence-driven execution, and structured measurement. A key technical aspect is the selection and emulation of adversaries.

Adversary Selection and Emulation:

The first step is identifying the most relevant adversaries. The panelists strongly advocated for leaning on Cyber Threat Intelligence (CTI). This involves:

• Internal CTI: If available, leverage in-house threat intelligence teams to identify threat actors specific to the organization's industry, geographic location, and past incidents.
• Vendor Threat Profiles: Engage security vendors, as many offer threat profiles tailored to their clients' environments.
• Peer Intelligence: Observe what adversaries are targeting peer organizations or those in similar industries. As Lauren Pill noted, "if your peers are getting hit with something, there's a high likelihood you're going to get hit with it."
• Historical Events: Analyze past incidents within the organization. For instance, if an OT shop consistently faces attacks targeting Windows 7 environments (due to legacy systems), then emulating threat actors known to target Windows 7 becomes highly relevant.
• Open-Source Resources: For organizations without extensive CTI resources, Adam Pennington recommended the ThaiCERT Threat Actor Encyclopedia. This publicly available resource consolidates reporting on various threat actors, allowing users to search by industry and country to identify potentially relevant adversaries.

Once relevant adversaries are identified, the focus shifts to emulating their Tactics, Techniques, and Procedures (TTPs). This involves:

• Starting Small: Sydney Moroni advised beginning with just a "couple techniques" rather than attempting a "whole threat actor attack plan." This incremental approach allows for proving success early and building organizational confidence.
• MITRE ATT&CK Framework: The MITRE ATT&CK® framework serves as a critical blueprint for adversary emulation. It provides a comprehensive, globally accessible knowledge base of adversary TTPs, enabling red teams to accurately mimic real-world attacks and blue teams to map their detections and coverage. Lauren Pill, a self-proclaimed "MITRE fan girl," emphasized measuring "everything against MITRE," noting that covering "30% of techniques on MITRE" in a year is a valuable metric. However, the panel cautioned against the unrealistic and often misleading claim of "100% ATT&CK coverage" by vendors, as some techniques are not practical or desirable to defend against.

Execution Environment and Guard Rails:

The environment for running purple team exercises is crucial. While the ultimate goal is to test in production for realistic insights into logging and true defenses, initial exercises often begin elsewhere:

• Test/Dev Environments: Less mature organizations or those new to purple teaming might start in isolated test or development environments.
• Virtual Environments: Sydney Moroni shared that her team at Splunk initially spun up a dedicated virtual environment with EDR agents to demonstrate value before moving to production. This approach helps build trust with management.
• Production with Approvals: When moving to production, obtaining explicit approvals and establishing clear Rules of Engagement (RoE) with strict guard rails are non-negotiable. These guard rails define what actions are permissible, what systems are off-limits, and how to prevent unintended disruption. The panel acknowledged the challenge of convincing management to "attack production data," but stressed that starting small, proving value, and building trust are key to gradually expanding scope.

Logging and Detection Engineering:

A foundational technical requirement for effective purple teaming is robust logging. Lauren Pill highlighted the futility of advanced red team techniques when "basic logging for DNS" is missing. Purple teaming directly tests the efficacy of existing logging configurations and detection rules. If a simulated attack technique is executed but no logs are generated or no detection fires, it immediately exposes a gap. This feedback loop directly informs detection engineering, leading to the creation of new detections or refinement of existing ones.

Tools for Breach and Attack Simulation (BAS):

The discussion touched upon tools that facilitate adversary emulation. The question of "open source versus commercial solutions like SafeBreach versus Caldera" arose.

• Caldera: Maintained by MITRE, Caldera is an open-source adversary emulation platform that automates adversary behaviors and allows for mapping to ATT&CK TTPs.
• SafeBreach: A commercial Breach and Attack Simulation (BAS) platform designed to continuously validate security controls.

The panelists concluded that the choice between open source and commercial solutions largely depends on an organization's budget, internal staff capabilities (e.g., availability of developers for customization), and risk tolerance for open-source software. Both types of solutions have their merits, and the "color of your money" (capital vs. human budget) can often dictate the decision.

Balancing Historical and "Blue Sky" Attacks:

A nuanced technical point was the balance between emulating known, historical TTPs from CTI and exploring "blue sky" or conceptual attack paths. Adam Pennington emphasized the need for both. In domains with limited CTI (e.g., some OT/IC spaces), "blue sky" work is essential to anticipate unknown threats. For highly mature organizations or prime targets like Nvidia, bleeding-edge "blue sky" attacks are necessary to stay ahead. However, for most organizations, starting with historical, relevant TTPs provides the most immediate and practical security uplift. This balance allows red teamers some leeway to "think like the attacker" and pivot creatively while still operating within defined objectives.

Demo / Proof of Concept

▶ Watch: Purple teaming as a learning opportunity for all (6:45)

While the panel discussion itself did not feature a live technical demonstration or "proof of concept" in the traditional sense, the speakers provided concrete examples of how purple teaming initiatives are implemented and how their value is demonstrated.

Sydney Moroni, who runs the internal purple team service at Splunk, described their approach to proving value, particularly when an organization is initially hesitant to conduct exercises directly in production environments. She explained that when they first started the purple team service, they "spun up a virtual environment" and deployed their EDR agents on it. They then used this isolated, non-production environment to test their red team tools and methodologies. By successfully demonstrating findings and the efficacy of their approach in this controlled setting, they were able to "show value that way" and build the necessary trust to eventually conduct exercises in more realistic, production-like settings.

This example serves as a practical blueprint for organizations looking to initiate purple teaming. It highlights the importance of:

1. Controlled Environments: Utilizing virtual or test environments to minimize risk and gain initial approvals.
2. Tool Validation: Testing red team tools and techniques to ensure they function as expected and generate relevant security events.
3. Value Demonstration: Providing tangible evidence of gaps found or detections created, even in a simulated environment, to build a case for broader implementation.

Lauren Pill further reinforced the concept of continuous validation, advocating for integrating adversary emulation into the threat hunting process. She stated, "if you don't actually emulate, you're not actually going to see what it looks like." This implies that every threat hunt can, and perhaps should, include a mini-PoC where suspected adversary behaviors are emulated to confirm the effectiveness of detections or to discover new ones. This iterative approach to testing and validation ensures that defenses are continuously tuned and improved.

Though no direct "demo" was given, these real-world examples from the speakers' experiences illustrate the practical application of purple teaming principles and how organizations can effectively conduct and validate their efforts.

Defensive Implications

▶ Watch: Early red teaming misaligned with real-world adversaries (7:15)

The insights from this panel offer critical, actionable guidance for security defenders aiming to enhance their organization's posture against evolving threats. The core defensive implications can be summarized as follows:

1. Prioritize Foundational Security: Before embarking on advanced adversary emulation, defenders must ensure their fundamental security operations are robust. This includes having a mature Security Operations Center (SOC) and Incident Response (IR) team, comprehensive and effective logging across critical systems (e.g., DNS, endpoint activity), and well-defined incident response plans and SLAs. As Sydney Moroni stated, if the SOC and IR teams are not "flowing" and able to respond to incidents properly, an organization is "probably not ready" for purple teaming.
2. Integrate Cyber Threat Intelligence (CTI) Deeply: Defenders should actively engage with or develop CTI capabilities to inform their purple teaming efforts. By understanding the specific threat actors, TTPs, and campaigns relevant to their industry, geography, and historical incidents, blue teams can guide red teams to emulate the most impactful and realistic threats. This ensures that defensive resources are focused on detecting and responding to actual risks, rather than generic attack patterns. Defenders should look to resources like the ThaiCERT Threat Actor Encyclopedia if internal CTI is limited.
3. Embrace Collaboration and Transparency: The traditional "throw it over the wall" mentality between red and blue teams must be dismantled. Defenders should actively seek out and foster transparent collaboration with red teamers, participating in joint planning, live execution, and post-exercise analysis. This direct interaction facilitates immediate learning, improves mutual understanding of attack and defense methodologies, and ultimately strengthens the overall security team.
4. Develop Robust Metrics and Reporting: Defenders need to move beyond simply documenting vulnerabilities and instead focus on demonstrating the tangible value of purple teaming to leadership. This involves tracking metrics such as the number of new detections created, updates to documentation, the diversity of threat actors emulated, and improvements in Mean Time To Detect (MTTD) and Mean Time To Close (MTTC). Crucially, these metrics must be translated into a compelling narrative that explains the story of how security posture has incrementally improved, highlighting specific gaps closed and capabilities gained.
5. Utilize MITRE ATT&CK Strategically: The MITRE ATT&CK framework is an invaluable tool for mapping defensive coverage and identifying gaps. Defenders should use ATT&CK to measure their detection capabilities against specific TTPs. However, they must be wary of vendors claiming "100% ATT&CK coverage," as this is often unrealistic and may lead to misprioritization. Instead, focus on achieving comprehensive coverage for the ATT&CK techniques most relevant to their identified threat actors.
6. Adopt a Proactive Culture: Purple teaming and adversary emulation are inherently proactive security measures. Defenders should strive to integrate these activities into their regular security operations, ideally as part of a continuous threat hunting process. This continuous testing and validation help identify and remediate weaknesses before they can be exploited by real adversaries.
7. Consider External Expertise When Necessary: For smaller organizations or those with limited internal resources, seeking external assistance from vendors offering purple teaming, red teaming, or penetration testing services can be highly beneficial. These engagements can not only provide valuable security assessments but also serve as learning opportunities for internal teams to observe best practices and build foundational knowledge. Leveraging existing compliance requirements, such as annual penetration tests or incident response tabletops, to incorporate elements of adversary emulation is also a smart strategy.
8. Invest in Logging and Detection Engineering: Purple teaming will inevitably reveal gaps in logging and detection capabilities. Defenders should be prepared to advocate for increased logging infrastructure, improved log retention, and dedicated resources for detection engineering to translate purple team findings into hardened defenses. The ability to identify missing logs or ineffective detections is a direct, measurable output of these exercises.

Key Takeaways

• Purple teaming is a transparent, collaborative process that breaks down traditional red and blue team silos, fostering shared learning and collective improvement in security posture.
• Adversary emulation must be intelligence-driven, focusing on emulating the TTPs of specific, relevant threat actors to ensure security testing is realistic and impactful for the organization.
• Demonstrating value requires robust metrics and narrative reporting, moving beyond raw counts to communicate the story of incremental security improvements and resource needs to executive leadership.
• Foundational security capabilities are a prerequisite; organizations need a mature SOC, robust logging, and effective incident response before undertaking advanced purple teaming.
• Strategic tool selection and gradual implementation are crucial, with choices between open-source (e.g., Caldera) and commercial (e.g., SafeBreach) solutions depending on budget and maturity, and starting small to build trust.
• AI is an augmenting tool for defenders, lowering the barrier to entry for blue teamers in adversary emulation by providing situational guidance, though it's not yet ready for independent, autonomous emulation.

About the Speaker(s)

Sydney Moroni is a Principal Threat Hunter at Splunk, bringing over a decade of experience in blue team operations. Her expertise spans incident response, forensics, and threat hunting. Moroni is instrumental in running Splunk's internal purple team service, where she leads multiple exercises annually, focusing on collaborative security enhancements.

Lauren Pill serves as the Global Head of Detection and Response at Marsh McLennon, one of the largest companies in the insurance sector. With 10 years in the industry, she has a comprehensive background in various blue teaming roles. Pill is also a co-founder of Thor Collective, a resource for threat hunting content, and actively engages in purple teaming work, emphasizing its role in raising community skill levels.

Adam Pennington is a veteran of the MITRE Corporation, having been with the organization for 16 years. He is the manager of MITRE ATT&CK®, a globally recognized cybersecurity framework, a role he has held for the past five years. Pennington played a key role in the genesis of ATT&CK, which originated from MITRE's internal purple teaming efforts to facilitate communication between red and blue teams during exercises.