Defeating EDR Evading Malware with Memory Forensics

Defeating EDR Evading Malware with Memory Forensics

Speakers: Andrew Case, Director of Research, Volexity

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=PmqvBe1LSZc

Overview

In this DEF CON 32 presentation, Andrew Case, a core developer on the Volatility memory analysis project and Director of Research at Volexity, delves into the critical issue of sophisticated malware bypassing Endpoint Detection and Response (EDR) solutions. The talk addresses a prevalent and concerning disconnect observed in real-world incident response scenarios: EDRs are often deployed, configured, and reported as active on compromised machines, yet potent malware frequently operates undetected within the very processes these EDRs are supposed to monitor. This presentation outlines Volexity's research into understanding why this evasion occurs and, more importantly, how advanced memory forensics techniques, particularly leveraging the Volatility framework, can automatically detect such stealthy threats.

Case emphasizes that modern, advanced malware is increasingly designed to be "memory-only," minimizing its footprint on the file system and encrypting network communications to evade traditional security controls. When a compromised machine is shut down without a memory sample, crucial evidence is permanently lost. The research presented here focuses on developing scalable and automated methods to identify EDR evasion techniques in memory, thereby directly exposing the underlying malware that employs them. This approach is vital for security teams dealing with large volumes of memory samples, enabling them to efficiently pinpoint critical incidents amidst a sea of data and conduct targeted deep dives.

Background

▶ Watch: Introduction and the EDR evasion problem (0:00)

The landscape of cybersecurity defense has evolved significantly, with EDR solutions becoming a cornerstone of enterprise security. These tools are designed to monitor system activity, detect malicious behavior, and respond to threats in real-time. However, as Case highlights, a persistent challenge remains: despite the widespread deployment of EDRs, sophisticated threat actors continue to compromise systems and operate with impunity. Volexity's incident response experience repeatedly shows powerful malware residing in processes where EDRs are supposedly active, signaling a fundamental gap in detection capabilities.

This problem is exacerbated by the trend of advanced persistent threats (APTs) and sophisticated malware families adopting memory-only tactics. Such malware executes directly from memory, avoiding writing malicious artifacts to disk. Network communications are often encrypted, making traditional perimeter defenses less effective. As Case underscores, citing numerous industry reports including one from Microsoft detailing a spyware operation, the "real evasion" and "real payloads" of modern threats are predominantly found in memory. If a memory sample is not acquired from a compromised machine before it's powered off, critical forensic evidence—the very existence and capabilities of the malware—is irrevocably lost. This reality necessitates a robust memory forensics capability as an integral part of any comprehensive security strategy.

Volexity's research methodology involves deep dives into specific malware techniques, creating infected virtual machines to observe and analyze active evasion methods, and then developing automated detection capabilities using tools like Volatility. The goal is to move beyond manual analysis, which is impractical for the hundreds or thousands of memory samples often encountered in large-scale investigations, towards a scalable system that can accurately identify high-priority incidents with minimal false positives.

Key Findings

▶ Watch: Why memory forensics is vital for detection (2:20)

The research presented by Andrew Case and Volexity reveals several critical findings regarding EDR evasion and its detection through memory forensics:

1. Targeted Evasion: EDR bypasses are typically not system-wide. Instead, malware specifically targets the process or processes where it is running directly or into which it has injected code. This localized evasion means that detecting the bypass itself is a direct indicator of the presence of malware.
2. Scalable Detection: By focusing on the specific indicators of EDR evasion within memory, security teams can develop highly scalable detection mechanisms. This allows for efficient triage of a large number of memory samples, quickly identifying the few that warrant immediate, in-depth investigation. The objective is to produce a small, high-confidence set of alerts rather than a deluge of false positives.
3. Widespread EDR Monitoring Bypasses: Modern malware routinely bypasses common EDR monitoring interfaces. The talk categorizes these interfaces and details how they are circumvented:

• Kernel Callbacks: These older mechanisms, where EDRs register kernel drivers to be notified of process, thread, or DLL activity, are commonly disabled. Ransomware families, for instance, frequently leverage "bring your own vulnerable driver" (BYOVD) techniques to load their own kernel drivers, which then disable EDR/AV callbacks, effectively blinding the security solutions.
• Event Tracing for Windows (ETW): ETW provides a broad range of system activity data. However, as Case notes, "dozens and dozens of papers" describe methods to disable or tamper with ETW, making it a common target for evasion.
• Antimalware Scan Interface (AMSI): AMSI allows Windows Defender and other AV/EDR engines to scan scripting language code (e.g., PowerShell, macros) before execution. Like ETW, AMSI is a well-documented target for bypass techniques.

1. Evolution of System Call Monitoring Bypasses: The talk specifically highlights the evolution of techniques used to bypass EDRs that monitor system calls.

• Older Methods (Code Overwriting): Historically, malware would overwrite the code of legitimate system call handlers, often within ntdll.dll, to divert execution or prevent EDR hooks from firing. This approach is inherently risky; overwriting actively used code can lead to system instability or crashes. Furthermore, EDRs can easily detect this by periodically checking the integrity of their hooked functions.
• Newer Methods (Non-Code Overwriting): A significant finding is the emergence of more sophisticated bypasses that allow malware to make malicious system calls without overwriting any legitimate code. This makes detection much harder for EDRs that rely on code integrity checks or traditional hooking mechanisms. While the talk introduces the concept, it implies that understanding the underlying Windows system call mechanism is key to detecting these novel evasions.

These findings collectively underscore the growing sophistication of malware and the necessity for defenders to adopt advanced, memory-centric forensic approaches to keep pace.

Technical Deep Dive

▶ Watch: How EDRs monitor system activity explained (4:15)

To understand how EDRs are bypassed and how memory forensics can detect these evasions, it's crucial to grasp the underlying mechanisms EDRs use to monitor system activity, particularly around system calls. Andrew Case outlined four primary categories:

1. Kernel Callbacks: These are long-standing mechanisms where EDRs (or antivirus solutions) install a driver in the Windows kernel. This driver registers callbacks to be notified of specific system events, such as process creation, thread creation, or DLL loading. Upon notification, the EDR's kernel component can inspect the activity and potentially block it. However, as Case points out, this method is readily defeated by modern ransomware and other sophisticated malware. Attackers utilize "bring your own vulnerable driver" (BYOVD) exploits to load their own malicious kernel drivers. Once in the kernel, these drivers have the privileges necessary to unregister or disable the EDR's callbacks, effectively rendering the EDR blind to subsequent malicious activities like code injection or process creation.

1. Event Tracing for Windows (ETW): ETW is a high-performance, general-purpose event tracing facility built into Windows. It allows kernel drivers and user-mode applications to subscribe to a wide array of system events generated by the operating system and other applications. EDRs leverage ETW to gain visibility into processes, network activity, file operations, and more. While powerful, ETW is not immune to evasion. Numerous publicly documented techniques exist for disabling ETW providers or otherwise interfering with its logging mechanisms, making it a common target for attackers seeking to reduce their telemetry footprint.

1. Antimalware Scan Interface (AMSI): AMSI is a generic interface that allows applications and services to integrate with any antimalware product present on a machine. It's particularly relevant for scripting engines (like PowerShell, VBScript, and JavaScript) and macro-enabled documents. Before a script or macro executes, AMSI allows registered antivirus and EDR engines to scan its content. If deemed malicious, the execution can be blocked. AMSI has been a significant hurdle for attackers, but like ETW, it has been the subject of extensive research, leading to "dozens and dozens of papers" detailing methods to bypass or patch AMSI in memory, allowing malicious scripts to run undetected.

1. System Call Monitoring: This is the primary focus of Case's talk, representing a deeper layer of EDR interaction with the operating system. System calls are the interface through which user-mode applications request services from the Windows kernel (e.g., creating files, allocating memory, creating processes). EDRs monitor these calls to detect suspicious behavior.

Case elaborates on the evolution of system call bypasses:

• Code Overwriting (Older, Unsafe): Earlier evasion techniques involved directly overwriting portions of the ntdll.dll library in memory. ntdll.dll is a critical system library that acts as a proxy between user-mode applications and the kernel, containing the user-mode stubs for system calls. By overwriting the code of specific system call functions within ntdll, malware could either remove EDR hooks or redirect execution to its own malicious code. However, this method is problematic:
• Unsafe: If an EDR or another legitimate thread is actively executing the overwritten function at the moment of modification, it can lead to system crashes or instability.
• Detectable: EDRs can periodically verify the integrity of their hooks or the original ntdll code. Any deviation would immediately signal tampering.

• Non-Code Overwriting (Newer, Stealthier): The talk highlights the emergence of more advanced system call bypasses that do not rely on overwriting code. This represents a significant challenge for EDRs that depend on detecting code modifications. While the exact mechanisms of these non-overwriting bypasses are not fully detailed in the provided transcript, Case explains the fundamental structure of 64-bit Windows system calls, which is crucial for understanding how such bypasses might operate:
• Nt vs. Zw Prefixes: Windows system call functions often have two versions: Nt (e.g., NtCreateFile) and Zw (e.g., ZwCreateFile). The Nt prefixed functions are typically the user-mode entry points in ntdll.dll, while Zw prefixed functions are their kernel-mode counterparts. From a functional perspective, they represent the same underlying system service.
• 64-bit Calling Conventions: In 64-bit Windows, parameters for functions are passed in specific registers. For regular C/C++ function calls, the first four parameters are typically passed in RCX, RDX, R8, and R9. However, for system calls, there's a specific alteration: the second parameter is passed in R10 instead of RDX. This means the calling stub in ntdll must adjust the register usage before initiating the kernel transition.
• System Call Number: Crucially, the EAX (or RAX in 64-bit) register must hold the system call number. Each system call (e.g., NtCreateFile, NtAllocateVirtualMemory) has a unique identifier for a given Windows version. This number serves as an index into the kernel's System Service Descriptor Table (SSDT), which maps system call numbers to their corresponding kernel functions. The assembly instructions for all system call handlers in ntdll look almost identical, with the only variation being the specific system call number loaded into EAX.

The implication of these non-overwriting bypasses is that attackers can craft their own system call stubs in memory, carefully replicating the legitimate calling conventions and providing the correct system call number, effectively "calling" the kernel directly without traversing the EDR-hooked ntdll functions. Detecting such sophisticated manipulation requires a deep understanding of memory structures and the ability to analyze memory regions for unusual code patterns or execution flows that deviate from legitimate system call paths. This is where tools like Volatility become indispensable.

Demo / Proof of Concept

▶ Watch: Talk's focus: system call monitoring (5:45)

While the provided transcript does not detail a live demonstration or specific proof-of-concept tools shown during the talk, Andrew Case explicitly outlines Volexity's rigorous research methodology which includes practical implementation and validation. The team's process involves:

1. Deep Dives: Performing targeted, specific deep dives on malware techniques observed in the wild or reported by other vendors.
2. Documentation: Documenting all identified techniques, including both private tools from threat groups and open-source implementations.
3. Infected Virtual Machines: Creating "infected virtual machines" where these evasion techniques are actively deployed. This allows the researchers to observe, analyze, and collect memory samples from environments where EDRs are purportedly active but malware is successfully evading detection.
4. Memory Forensics Research: Applying the "deepest memory forensics research" using Volatility to understand how these evasions manifest in memory.
5. Public Disclosure: Publishing these findings through talks like this one.

Therefore, while a direct demo was not described, the entire research project serves as a proof-of-concept for the efficacy of memory forensics in detecting these advanced evasions. The emphasis is on developing Volatility-based detection capabilities that can be applied to real-world memory samples, moving beyond theoretical understanding to practical, automated detection.

Defensive Implications

▶ Watch: Common EDR bypass techniques: callbacks, ETW, AMSI (6:00)

The insights presented in "Defeating EDR Evading Malware with Memory Forensics" carry profound implications for cybersecurity defenders, necessitating a strategic shift in how organizations approach endpoint security and incident response:

1. Memory Forensics is Non-Negotiable: The most critical implication is the undeniable necessity of integrating memory forensics into every organization's security stack and incident response plan. Relying solely on file system analysis, network logs, or even advanced EDRs that miss memory-resident threats is no longer sufficient. If a memory sample is not acquired from a compromised machine before it's powered down, critical evidence is lost forever. Defenders must have robust processes and tools (like Volatility) for memory acquisition and analysis.

1. Evolve EDR Strategies: EDR solutions need to evolve beyond traditional hooking mechanisms and signature-based detection. They must develop capabilities to detect the subtle indicators of system call manipulation, even when code is not directly overwritten. This might involve more sophisticated behavioral analysis, hardware-assisted memory protection, or leveraging kernel-level telemetry that is harder to bypass. EDR vendors should also explore integrating memory analysis capabilities directly or providing better interfaces for external memory forensic tools.

1. Focus on Evasion Detection: Instead of solely chasing known malware signatures, defenders should prioritize detecting the techniques of evasion themselves. As Case highlights, if EDR evasion is detected in a process, it directly indicates the presence of malware. This shift in focus allows for a more resilient defense against novel or polymorphic threats.

1. Automated and Scalable Analysis: Given the volume of data in enterprise environments, manual memory analysis is impractical. Organizations must invest in automated, scalable memory analysis frameworks. Tools like Volatility, with their extensible plugin architecture, are crucial for developing custom detection routines that can rapidly triage hundreds or thousands of memory samples, identifying the most critical ones for human analysts. This helps minimize false positives and maximizes the efficiency of incident response teams.

1. Understanding Low-Level OS Internals: Security professionals, especially incident responders and threat hunters, need a deeper understanding of operating system internals, particularly how Windows handles processes, threads, DLLs, and system calls. This knowledge is vital for interpreting memory forensic artifacts and understanding the sophisticated techniques used by attackers.

1. Regular Threat Emulation and Testing: Organizations should regularly test their EDR solutions against known evasion techniques, including BYOVD for kernel callbacks, ETW/AMSI bypasses, and the newer non-code-overwriting system call manipulation methods. This proactive testing can reveal gaps in detection before a real attack exploits them.

By adopting these defensive strategies, organizations can significantly enhance their ability to detect and respond to the most advanced and stealthy threats that currently bypass conventional security controls.

Key Takeaways

• Memory-Only Malware is Prevalent: Modern, sophisticated malware predominantly operates in memory, avoiding file system artifacts and encrypting network traffic to evade traditional EDRs and security tools.
• Memory Forensics is Indispensable: Without memory acquisition and analysis, critical evidence of advanced threats is lost upon system shutdown, making memory forensics (e.g., with Volatility) a vital component of incident response.
• EDR Evasion is Targeted: Malware typically bypasses EDRs only in specific processes where it runs or injects, meaning detection of evasion directly correlates to the presence of malware.
• Traditional EDR Monitoring is Bypassed: Common EDR monitoring mechanisms like kernel callbacks (via BYOVD), ETW, and AMSI are routinely circumvented by advanced threat actors.
• Evolving System Call Bypasses: Attackers are moving beyond unsafe code-overwriting techniques for system call evasion to more stealthy, non-code-overwriting methods that are harder for EDRs to detect.
• Scalable, Automated Detection is Key: To handle the vast number of memory samples in enterprise investigations, automated and scalable memory analysis techniques are crucial for efficiently identifying high-priority incidents with minimal false positives.

About the Speaker(s)

Andrew Case is a highly respected figure in the field of cybersecurity, particularly known for his contributions to memory forensics. He is a core developer on the Volatility memory analysis project, an open-source framework widely used for extracting digital artifacts from volatile memory (RAM) samples. In addition to his work on Volatility, Andrew Case serves as the Director of Research at Volexity, a leading provider of incident response and threat intelligence services. His role at Volexity involves conducting deep dives into novel malware techniques and developing advanced detection capabilities to combat sophisticated threats seen in real-world attacks.