eAUDIT: A Fast, Scalable and Deployable Audit Data Collection System

eAUDIT: A Fast, Scalable and Deployable Audit Data Collection System

Speakers: R. Sekar; Hanke Kimm; Rohit Aich

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=9lYeqYLIMq4

Overview

In this IEEE S&P presentation, Hungi Kim from the Secure Systems Lab at Stony Brook University introduces eAUDIT, a novel system designed to overcome critical limitations in existing audit data collection mechanisms. The talk highlights how current logging solutions are fundamentally flawed, exhibiting high overhead, significant data loss, and vulnerabilities to log tampering, which severely hinder their utility in detecting and analyzing advanced persistent threats (APTs). eAUDIT proposes a robust solution built on eBPF (extended Berkeley Packet Filter) that dramatically improves performance, reliability, and security of audit data collection.

The core problem eAUDIT addresses is the inadequacy of current audit systems to cope with the immense volume and rate of system calls generated in modern computing environments. These systems often drop over 90% of events under intense loads and maintain large in-memory buffers that can be wiped by attackers with root access. By leveraging eBPF and introducing a series of innovative optimizations, eAUDIT aims to provide a deployable, high-fidelity audit system that can capture full provenance data without compromising system performance or security.

This work is particularly significant for enterprise security, digital forensics, and incident response teams. The ability to collect comprehensive, tamper-proof system call logs with minimal overhead is paramount for understanding complex attack campaigns, reconstructing timelines, and developing effective post-breach detection strategies. eAUDIT's advancements represent a crucial step towards building more resilient and observable computing infrastructures.

Background

▶ Watch: Drawbacks: high overhead, large data volume in existing loggers (0:45)

The landscape of modern cyber threats is dominated by Advanced Persistent Threats (APTs), sophisticated attack campaigns designed to evade traditional preventative security measures and remain undetected within enterprise systems for extended periods. Effective response to APTs relies heavily on post-attack detection and forensic analysis, a process that demands high-fidelity audit logs. System call logs, in particular, are considered the "gold standard" due to their ability to provide full provenance, linking all stages and components of an attack campaign.

However, existing audit data collection systems, including widely used solutions like Linux auditd and various commercial and experimental loggers, suffer from severe drawbacks. Firstly, they impose high overhead, with experiments showing Linux auditd slowing down workloads by as much as eightfold. This high overhead isn't just a performance bottleneck; it's a direct indicator that these systems cannot keep pace with the excessive rates of system calls, leading to the second major issue: data loss. Benchmarks using workloads like Postmark, a file system benchmark, reveal that existing systems often drop a majority of events even at moderate loads, escalating to over 90% data loss under more intense conditions. This means critical attack evidence can be entirely missed or obscured by the sheer volume of benign activity.

The third critical vulnerability is log tampering. Existing loggers maintain a "window" of records in memory before they are transmitted to a secure server. If an attacker gains root access, they can exploit this window, which can range from 50,000 to 500,000 audit records, to erase evidence of privilege escalation or other malicious activities. This renders the logs unreliable for forensic analysis. The challenges in building a resilient system are compounded by the difficulty of developing and maintaining kernel code, and the necessity to handle system call rates that can reach tens of millions per second. These fundamental weaknesses in current audit logging solutions underscore the urgent need for a new approach like eAUDIT.

Key Findings

▶ Watch: Introducing eBPF for safe kernel extensions and syscall logging (4:30)

The research presented in the eAUDIT talk uncovers several critical findings regarding the state of audit data collection and proposes innovative solutions. The primary findings are:

1. Systemic Flaws in Existing Loggers: Current commercial and experimental audit logging systems exhibit profound shortcomings, including:

• Excessive Data Loss: Even at moderate loads, these systems lose a majority of audit events, with losses exceeding 90% under high system call rates. This makes them practically useless for comprehensive threat detection.
• High Overhead: Traditional methods introduce significant performance degradation, exemplified by Linux auditd's eightfold slowdown, making them impractical for production environments.
• Large Log Tamper Windows: In-memory buffers can store tens to hundreds of thousands of records, creating a critical vulnerability where attackers with root access can erase their tracks before logs are secured.

1. eBPF Alone is Insufficient: While eBPF offers a powerful, safe, and kernel-level mechanism for intercepting system calls without modifying kernel source code, a straightforward eBPF-based logging architecture still experiences substantial data loss (e.g., up to 20% even with initial improvements), indicating that eBPF capabilities alone cannot solve the data loss problem under intense loads.

1. Three Pillars of Optimization: eAUDIT's success stems from three synergistic optimizations:

• Compact Encoding Scheme: By logging only essential system call arguments, eAUDIT drastically reduces the average record size from over 170 bytes to approximately 17 bytes. This significantly lowers data volume and helps reduce initial data loss by 10-20%.
• Two-Level Buffering (Per-CPU Caches): The core issue of ring buffer contention is addressed by introducing per-CPU caches. These caches aggregate multiple system call events (parameter P) before flushing them as a single block to the eBPF ring buffer. This reduces the rate of ring buffer accesses by a factor of P, leading to a dramatic reduction in overhead from 100% to single-digit percentages (e.g., below 5% with P=100). This optimization effectively eliminates data loss.
• Analytical Performance Model: To balance the trade-off between low overhead (achieved by larger P values) and a small log tamper window (requiring lower latency), eAUDIT introduces a performance model. This model, which tunes parameters P (message cache size) and W (user-level wake-up interval), minimizes the product of overhead and latency, ensuring optimal system configuration.

1. Superior Performance and Security: The combined effect of these optimizations allows eAUDIT to achieve an average overhead of just 3% across all benchmarks and loads, virtually eliminate data loss, and maintain a log tamper window that is hundreds of times smaller than the next best performing system, Sysdig. For instance, eAUDIT typically buffers only a couple of hundred records, compared to Sysdig's 35,000 to 200,000 records.

These findings collectively demonstrate that a practical, high-performance, and secure audit data collection system is achievable by strategically combining modern kernel technologies like eBPF with intelligent buffering and analytical tuning.

Technical Deep Dive

▶ Watch: eAUDIT base design with eBPF and initial data loss (5:30)

eAUDIT's technical prowess lies in its intelligent application of eBPF combined with several critical optimizations to address the inherent challenges of high-volume, low-latency audit logging.

The foundation of eAUDIT is eBPF, a virtual machine integrated into the Linux kernel. eBPF allows for the compilation and loading of safe kernel extensions without requiring modifications to the kernel source code. This capability makes it ideal for system call logging, as eBPF programs can be attached to kprobes or tracepoints to intercept every system call entry and exit. Crucially, while eBPF programs typically have read-only access to kernel data, this is sufficient for logging purposes, as the goal is to capture system call arguments and metadata.

A naive eBPF-based design for audit logging would involve:

1. An eBPF program capturing approximately 80 provenance-related system calls.
2. Immediately sending each intercepted system call event to user-level memory via an eBPF ring buffer.
3. A user-level process then reading these events and writing them to disk.

Initial experiments with this naive design, using a record size comparable to previous systems (around 200 bytes), showed substantial improvement over traditional loggers, reducing data loss from 60%+ to less than 20% for the Postmark workload. However, more intense loads still pushed data loss to unacceptably high levels, demonstrating that eBPF alone isn't a silver bullet. The core problem was identified not as the size of individual records, but the high rate of messages being enqueued onto the ring buffer, leading to contention.

To overcome these limitations, eAUDIT introduces three key optimizations:

1. Compact Encoding Scheme: The first optimization focuses on reducing the data volume per event. Instead of logging extensive metadata, eAUDIT logs only the essential system call identifier and its arguments. This compact encoding scheme dramatically reduces the average record size from over 170 bytes to a lean 17 bytes. This reduction directly translates to lower data volume and, initially, a 10-20% decrease in data loss across most benchmarks, freeing up valuable bandwidth for event transmission.

1. Two-Level Buffering Design (Per-CPU Caches): Addressing the ring buffer contention, eAUDIT implements a two-level buffering strategy. The eBPF ring buffer is efficient for transmitting large volumes of data, but high-frequency, small-message writes cause contention. eAUDIT mitigates this by introducing per-CPU caches. Each CPU maintains a dedicated cache that accumulates P system call events. Once a cache is full, its entire contents are flushed as a single, larger message to the central eBPF ring buffer. This mechanism effectively reduces the rate of ring buffer accesses by a factor of P. For example, setting P to 100 means that 100 system call events are buffered on-CPU before a single ring buffer write occurs. This dramatically reduces overhead, bringing it down from over 100% (relative to baseline) to single-digit percentages (e.g., below 5% for all benchmarks when P=100), and crucially, solves the data loss problem entirely.

1. Analytical Performance Model for Balancing Overhead and Latency: While a larger P value improves throughput and reduces overhead, it simultaneously increases latency, meaning a larger number of records are buffered in the per-CPU caches before being transmitted. This directly impacts the log tamper window. To find the optimal balance, eAUDIT develops a performance model that tunes two key parameters:

• P: The message cache size in terms of the number of system call events.
• W: The interval (in milliseconds) at which the user-level process is woken up to read messages from the ring buffer.

The model's objective is to minimize the product of Overhead (O) and Latency (L) (i.e., O * L). By differentiating this product with respect to W and setting it to zero, an optimal formula for W can be derived. Experimental validation of this model shows a strong correlation between predicted and measured overheads. For instance, the model suggests an optimal W value of 5 to 8 milliseconds. This analytical approach ensures that eAUDIT can be configured to achieve minimal overhead while maintaining a log tamper window that is hundreds of times smaller than competing systems. The combined optimizations result in an 18-fold decrease in overhead compared to the unoptimized eBPF design, achieving a consistent 3% overhead across diverse workloads.

Demo / Proof of Concept

▶ Watch: Performance model for balancing overhead and log tampering latency (8:00)

While the talk did not feature a live, interactive demonstration of eAUDIT, the speakers extensively presented the results of rigorous experimental validation and benchmarking that serve as a robust proof of concept for the system's effectiveness. These validations were conducted on experimental platforms to measure eAUDIT's performance against existing commercial and experimental audit loggers, including a specific comparison with Sysdig, often considered one of the better-performing systems.

The core of the proof of concept revolved around demonstrating eAUDIT's ability to:

1. Eliminate Data Loss: Using the Postmark file system benchmark, the researchers showed that while existing systems experienced 60% to over 90% data loss, eAUDIT, with its compact encoding and two-level buffering, achieved near-zero data loss even under intense loads. This directly validates the claim that eAUDIT can capture essentially all events, a critical requirement for forensic analysis.
2. Achieve Minimal Overhead: The impact on system performance was quantified, with eAUDIT demonstrating an average overhead of 3% across all benchmarks and loads. This is a dramatic improvement compared to the eightfold slowdown observed with Linux auditd or the 100%+ overhead of an unoptimized eBPF approach. The talk specifically highlighted that with a per-CPU cache size P set to 100, overhead consistently dipped below 5%.
3. Maintain a Small Log Tamper Window: A crucial security metric, the log tamper window, was rigorously compared. eAUDIT consistently produced windows of only a couple of hundred records, even at maximum benchmark loads, peaking at around 1,000 records. In stark contrast, Sysdig, the "next best performing system," exhibited tamper windows ranging from 35,000 to 200,000 records. This demonstrates eAUDIT's superior resilience against evidence destruction by an attacker who gains root access.
4. Validate the Performance Model: The analytical model for balancing overhead and latency was experimentally validated. A chart showing measured overhead against predicted overhead demonstrated that data points fell closely around a diagonal line, indicating a strong match between the model's predictions and actual system behavior. This confirmed the model's ability to identify optimal W values (e.g., 5-8 milliseconds) for minimizing the overhead-latency product.

These comprehensive experimental results provide compelling evidence that eAUDIT successfully addresses the major bottlenecks in existing audit collection systems and delivers on its promise of a fast, scalable, and deployable solution.

Defensive Implications

▶ Watch: Deriving optimal parameters for the overhead-latency trade-off (9:40)

The advancements presented by eAUDIT have profound implications for security defenders, offering a path to significantly enhance an organization's ability to detect, investigate, and respond to sophisticated cyber threats.

1. Prioritize High-Fidelity, Low-Overhead Logging: Defenders should recognize that traditional audit logging systems are fundamentally inadequate. The high data loss and performance overhead of systems like Linux auditd mean that critical evidence is likely being missed, and production systems are being slowed unnecessarily. Adopting or developing solutions with eAUDIT's characteristics (near-zero data loss, ~3% overhead) should be a top priority for effective APT detection and forensic analysis.

1. Minimize Log Tamper Windows: The vulnerability of large in-memory log buffers is a critical security flaw. Defenders must ensure that their chosen audit solution minimizes the window during which an attacker with root privileges can erase evidence. eAUDIT's ability to reduce this window by hundreds of factors compared to competitors like Sysdig is a significant defensive advantage, making it much harder for attackers to hide their tracks.

1. Leverage eBPF for Kernel-Level Observability: eBPF is a game-changer for kernel-level security monitoring. Defenders should actively explore and adopt eBPF-based solutions for audit data collection and other security functions. Its safety (no kernel source modification needed) and power (full system call interception) make it an ideal foundation for robust security tools. Organizations should invest in understanding and implementing eBPF-driven security controls.

1. Understand and Tune Logging Parameters: The trade-off between logging throughput (overhead) and latency (tamper window) is critical. Defenders need to move beyond "set and forget" logging configurations. Solutions like eAUDIT's analytical performance model highlight the importance of understanding how parameters like P (cache size) and W (flush interval) impact these metrics. Security teams should strive to use systems that allow for intelligent tuning to optimize for their specific operational and security requirements.

1. Enhance Forensic Capabilities: With eAUDIT's promise of full provenance and complete data capture, forensic investigations can become significantly more effective. Incident responders will have a more reliable and complete dataset to reconstruct attack timelines, identify compromised assets, and understand attack methodologies, leading to more thorough remediation and improved future defenses.

In essence, eAUDIT provides a blueprint for building next-generation audit infrastructure that is resilient, performant, and secure, enabling defenders to stay ahead in the continuous battle against advanced threats.

Key Takeaways

• Existing audit data collection systems suffer from severe data loss (often >90%), high performance overhead, and large log tampering windows (tens to hundreds of thousands of records), making them ineffective against advanced persistent threats.
• While eBPF provides a strong foundation for kernel-level system call logging, a basic eBPF implementation alone cannot fully resolve data loss issues due to contention on the ring buffer.
• eAUDIT introduces three critical optimizations: a compact encoding scheme (reducing record size from 170+ to ~17 bytes), a two-level buffering design using per-CPU caches, and an analytical performance model to optimally balance overhead and latency.
• The combined optimizations enable eAUDIT to achieve near-zero data loss, a remarkably low average overhead of 3%, and a log tamper window that is hundreds of times smaller than leading alternative systems like Sysdig.
• The analytical performance model effectively tunes parameters (like P for cache size and W for user-level wake-up interval) to minimize the product of overhead and latency, ensuring optimal system configuration for both performance and security.
• eAUDIT represents a significant leap forward in audit data collection, offering a fast, scalable, and deployable system that provides high-fidelity, tamper-resistant system call logs crucial for modern cybersecurity defenses.

About the Speaker(s)

The eAUDIT project and presentation come from the Secure Systems Lab at Stony Brook University.

Hungi Kim presented this work, representing the research efforts of the lab. His involvement indicates a focus on developing practical solutions for security challenges within complex computing environments.

R. Sekar and Rohit Aich are also credited as contributors to this work, underscoring the collaborative research environment at the Secure Systems Lab. Their collective work is dedicated to the investigation of advanced persistent threats and the development of robust systems to detect and analyze them, with a particular emphasis on addressing the limitations of existing security tools.