Clash, Burn, and Exploit Manipulate Filters to Pwn kernelCTF

Clash, Burn, and Exploit Manipulate Filters to Pwn kernelCTF

Speakers: HexRabbit Chen, Security Researcher, Devcore

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=_1DTkkaNfM

Overview

In this highly technical talk, HexRabbit Chen, a security researcher from Devcore, dissects the intricacies of NS tables, the Linux kernel's modern packet filtering framework, revealing critical vulnerabilities that allowed him to compromise Google's demanding kernelCTF challenge. The presentation, titled "Clash, Burn, and Exploit: Manipulate Filters to Pwn kernelCTF," offers a deep dive into NS tables' internal architecture, its batch processing mechanism, and the subtle flaws in its object lifecycle management that can lead to severe kernel vulnerabilities.

Chen's research focuses on the Netfilter infrastructure, a crucial component of the Linux networking stack. By meticulously analyzing the differences in how NS tables manages object states during batch operations, he uncovered a fundamental logic error that could be leveraged for kernel exploitation. This talk is particularly relevant for kernel developers, security researchers, and anyone interested in understanding the complexities of kernel-level packet filtering and the sophisticated techniques required to find and exploit vulnerabilities in such critical components.

The significance of this work extends beyond academic interest, as kernelCTF offers substantial bounties—exceeding $50,000 for a zero-day LTS exploit—making it a prime target for security researchers. Chen's success in this highly competitive environment underscores the depth of his expertise in binary exploitation and Linux kernel internals, providing valuable insights into the methodologies used to identify and leverage kernel vulnerabilities.

Background

▶ Watch: Talk Introduction and Speaker Background (0:00)

The journey into kernel exploitation often begins with identifying a target, and for HexRabbit Chen, kernelCTF proved to be an irresistible challenge. As part of the Google Vulnerability Reward Program (VRP), kernelCTF presents participants with an unprivileged shell on a Linux system, tasking them with exploiting the kernel to retrieve a flag and earn a bounty. The challenge operates on a bi-weekly release cycle, featuring new kernel versions and different configurations: LTS and COS instances (newer kernels) and a "mitigation" instance (older kernel with custom patches, accepting more one-day exploits). The financial incentives are significant, with a zero-day exploit for an LTS kernel potentially yielding over $50,000 USD, and the same vulnerability applicable to mitigation and COS instances adding up to $21,000 each.

Chen, as a self-proclaimed kernelCTF newbie, recognized NS tables as a promising attack surface. Its configuration is enabled in kernelCTF, and a history of recently discovered vulnerabilities (multiple after kernelCTF's announcement) signaled that it was an active and potentially fruitful area for research.

NS tables is a modern, flexible packet filtering framework integrated into the Linux kernel. It was designed to supersede older, disparate filtering systems like IP tables, ARP tables, and EB tables, consolidating them into a single, unified entry point. At its core, NS tables employs a simple virtual machine (VM) design to implement its functionality, offering a new, more expressive command-line interface (CLI) syntax that allows for configuring multiple network filtering rules simultaneously, a capability derived from its unique batch processing architecture.

The framework's structure is hierarchical and tree-like, composed of five fundamental modules:

• Table: The highest-level container, used for managing related filtering objects. Each table is associated with a specific network family (e.g., IPv4, IPv6), and all objects within it adhere to that family.
• Chain: Hooks into predefined points within the Netfilter infrastructure (e.g., PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING). When a packet traverses a hooking point, the chain executes the rules it manages to decide whether to accept, block, or modify the packet.
• Rule: Can be thought of as a "small function" for the NS tables VM. Each rule contains one or more expressions that define the actual filtering logic.
• Expression: These are the "instructions" for the NS tables VM. Different types of expressions exist: some manipulate VM registers, while others load, read, or write data within network packets.
• Set: Used to store groups of data, such as IP addresses, port numbers, or arbitrary binary data. Sets can also function as maps, associating one type of data with another. Expressions can query these sets during packet filtering to make decisions.

This sophisticated architecture, while powerful, introduces significant complexity, particularly in managing the lifecycle of its various objects, making it a fertile ground for subtle vulnerabilities.

Key Findings

▶ Watch: NS tables Packet Filtering Framework Overview (2:20)

HexRabbit Chen's initial foray into the NS tables codebase, despite his admitted unfamiliarity at the time, quickly yielded a suspicious pattern related to object deletion. He observed three distinct functions responsible for deleting NS tables objects:

1. NS table delete rule deactivate: Called during rule deletion.
2. NFT flush: Called during a full configuration flush.
3. NFT set element catch all deactivate: Called during the deletion of a set element.

The crucial observation was a discrepancy in how these functions checked the "active" status of an object before deactivation. The first two functions, NS table delete rule deactivate and NFT flush, consistently employed is active next to determine if an object was active. In contrast, NFT set element catch all deactivate used NFT is active. This seemingly minor difference in an is_active check function proved to be a critical indicator of a potential vulnerability, specifically a use-after-free (UAF) or double-free scenario, stemming from a misunderstanding or misimplementation of NS tables' sophisticated object lifetime management.

To fully grasp the implications of this finding, Chen delved into the batch request mechanism and the generation-based object lifetime management of NS tables. Unlike simple query requests, which are handled locklessly by Netlink receive SKB, operations that modify the control plane (e.g., creating a new table or deleting a chain) are processed via NFT Netlink receive batch. This batch mechanism allows for multiple operations to be processed sequentially within a single request, a design choice that enables the CLI's ability to configure many elements at once.

The complexity arises when an operation within a batch request fails. To ensure atomicity and consistency, NS tables divides batch request handling into three distinct phases:

1. Prepare Phase: In this phase, each operation within the batch is processed. Any modifications to the control plane are only applied to the next generation state of the affected objects. The current generation state, which dictates runtime behavior, remains untouched. This temporary state allows for a staged modification.
2. Commit Phase: If all operations in the prepare phase complete successfully without errors, the system transitions to the commit phase. Here, the current generation state of all modified objects is updated to reflect their next generation state. The next generation state is then reset to a clean slate, making the changes live and available for both runtime packet filtering and subsequent query operations.
3. Abort Phase: If any operation fails during the prepare phase, the system enters the abort phase after all operations have been attempted. In this phase, all changes made to the next generation state during the prepare phase are meticulously reverted in reverse order. The goal is to restore the system to its state before the batch request began, effectively undoing any partial modifications.

The identified vulnerability lies precisely within the abort phase's interaction with the NFT set element catch all deactivate function. Because this function checks NFT is active (referring to the current generation's active state) instead of is active next (referring to the temporary, next generation state modified during the prepare phase), a critical mismatch occurs. If a set element was marked for deletion in the prepare phase, its next generation state would reflect deactivation. However, if the batch then aborts, NFT set element catch all deactivate might incorrectly conclude that the element is not active (based on the current generation state, which was never committed) and thus fail to properly revert its deactivation or re-add it to the hash table. This could lead to a scenario where the memory associated with the set element is freed during the abort, but the kernel's internal tracking (based on the current generation state) might still consider it active, leading to a use-after-free when subsequent operations attempt to access it. Alternatively, if the element was added and then deleted within the same aborting batch, incorrect state management could lead to a double-free.

Technical Deep Dive

▶ Watch: Initial Vulnerability Discovery: Deactivation Mismatch (4:40)

The core of the identified vulnerability resides in the subtle yet critical distinction between two state-checking functions, NFT is active and is active next, within the context of NS tables' multi-phase batch processing and its object lifetime management. To fully appreciate this, a deeper understanding of the current generation and next generation state variables is essential.

Every object within NS tables (tables, chains, rules, expressions, sets, and set elements) possesses these two state variables.

• current generation: This variable reflects the object's active state in the live, operational configuration. Only objects marked active in the current generation influence the runtime behavior of packet filtering and are visible to standard query operations.
• next generation: This variable serves as a temporary, staging area for proposed changes. Any operation that modifies the control plane, such as adding a new rule or deleting a set, exclusively manipulates the next generation state during the prepare phase of a batch request. This ensures that changes are not immediately applied to the live configuration, preventing inconsistent states if a batch operation fails midway.

Let's walk through the batch request mechanism with a focus on these generation states:

1. Batch Request Reception and Prepare Phase:

When a user-space utility (like nft) sends a batch request containing multiple operations (e.g., delete set C, add set D), the kernel's NFT Netlink receive batch handler initiates the prepare phase.

• For an operation like delete set C: The kernel locates set C. Instead of immediately freeing or deactivating it in the current generation, its next generation state is marked as deactivated. Crucially, the current generation state of set C remains activated, meaning it continues to function in the live filtering path.
• For an operation like add set D: A new set D object is allocated, and its next generation state is marked as activated. It's typically added to a temporary hash table or list associated with the next generation. Again, it does not immediately become part of the current generation live configuration.

1. Commit Phase (Successful Batch):

If all operations within the batch request successfully complete their processing in the prepare phase, the system transitions to the commit phase.

• The primary action here is to synchronize the current generation with the next generation. For every object whose next generation state was modified, its current generation state is updated.
• In our example: Set D's current generation would be set to activated, making it available for runtime and queries. Set C's current generation would be set to deactivated, and its associated memory would then be freed.
• Finally, the next generation states of all objects are reset to a clean, inactive state, ready for the next batch request.

1. Abort Phase (Failed Batch):

This is where the vulnerability becomes apparent. If any operation within the batch request encounters an error during the prepare phase (e.g., invalid parameters, out-of-memory), the entire batch is aborted. The system then enters the abort phase to revert all changes made during the prepare phase.

• The abort phase processes operations in reverse order of their application in the prepare phase. The goal is to undo only the next generation changes, restoring the system to its prior current generation state.
• For an operation like add set D (which was processed in the prepare phase): The set D object, which only existed in the next generation state, would be properly cleaned up and freed, as it never reached the current generation.
• For an operation like delete set C: The intention is to undo the deactivated state set in its next generation. This means set C, which was still active in the current generation, should remain active and its memory should not be freed.

The bug lies in the NFT set element catch all deactivate function. When an abort occurs, this function is called to clean up set elements that were potentially marked for deactivation in the next generation. However, instead of checking is active next (which would correctly reflect the temporary state within the aborting batch), it checks NFT is active. NFT is active queries the object's current generation state.

Consider a scenario:

1. A set element E is currently active in the current generation.
2. A batch request includes an operation to delete E. In the prepare phase, E's next generation state is set to deactivated.
3. Another operation in the same batch fails, triggering the abort phase.
4. During the abort phase, NFT set element catch all deactivate is called for E. It checks NFT is active. Since E's current generation state was never committed and remains activated, NFT is active returns true.
5. This could lead to a situation where the abort logic incorrectly believes E is still active in the current configuration and thus might not properly revert its next generation deactivation, or it might attempt to re-add an element that was already considered for removal.

Conversely, if an element was added and then deleted within the same aborting batch:

1. A batch request includes add set element F then delete set element F.
2. In the prepare phase, F is created and its next generation is set to activated, then deactivated.
3. The batch aborts. During the abort, the delete set element F operation is undone. If NFT set element catch all deactivate uses NFT is active, it might see that F is not active in the current generation (because it was never committed). This could lead to premature freeing or incorrect state handling during the undo process.

The most critical impact of this logic error is a Use-After-Free (UAF). If NFT set element catch all deactivate (or a related function using NFT is active during abort) incorrectly triggers the freeing of a set element's memory based on its next generation state (which should only be temporary), while the current generation state still considers the object active, subsequent access to that "active" object would become a UAF. This allows an attacker to manipulate freed memory, potentially leading to arbitrary read/write primitives in the kernel or even direct code execution, which is the ultimate goal in kernel exploitation challenges like kernelCTF.

Demo / Proof of Concept

▶ Watch: NS tables Object Lifetime Management and Generations (6:00)

The provided transcript focuses heavily on the background, the intricate design of NS tables, and the detailed explanation of the first vulnerability's root cause within the batch processing mechanism. While HexRabbit Chen mentions discovering three vulnerabilities in total and alludes to his "kernelCTF journey," the transcript does not include a description of a specific demo or a full proof-of-concept exploit chain for this particular flaw. The talk likely progresses to demonstrate the exploit in subsequent sections not covered in this excerpt.

Defensive Implications

▶ Watch: Example: How Batch Request Phases Work (8:00)

The vulnerabilities highlighted in HexRabbit Chen's talk, particularly the object lifetime management flaw in NS tables, carry significant defensive implications for kernel developers, system administrators, and security practitioners.

1. Rigorous State Management in Kernel Subsystems: The core issue stems from an inconsistent check (NFT is active vs. is active next) during complex multi-phase operations. This underscores the critical importance of meticulous state management in kernel subsystems, especially those dealing with dynamic object creation and deletion. Developers must ensure that functions interacting with objects during temporary states (like batch "prepare" or "abort" phases) consistently use the correct state variables (next generation in this case) rather than the current generation state, which represents the live configuration. This requires careful auditing of code paths that handle object lifecycle within transactional frameworks.

1. Enhanced Code Review and Static Analysis for Generation-Based Systems: Systems employing generation-based object management (where objects have "current" and "next" states for transactional integrity) are inherently complex. Standard code review processes might miss subtle discrepancies like the one identified. Enhanced code review practices, potentially incorporating specialized static analysis tools capable of tracking object state transitions across different operational phases, could help identify such logical flaws. Fuzzing tools specifically designed to trigger abort conditions in batch processing APIs would also be highly beneficial.

1. Importance of Netfilter/NS tables Configuration Hardening: For system administrators, the talk reinforces that kernel components like Netfilter and NS tables, while essential for network security, can also be a source of vulnerabilities if not properly secured. While the identified bug is a kernel-level flaw, understanding the attack surface and minimizing exposure is crucial. Regular patching of the Linux kernel is paramount to address such vulnerabilities as they are discovered and fixed. If possible, restricting unprivileged user access to Netlink APIs that manipulate NS tables configuration could also limit the exploitability surface, though this might not always be practical in containerized or multi-tenant environments.

1. Impact of Use-After-Free (UAF) Vulnerabilities: The potential for a Use-After-Free (UAF) due to this bug is a severe concern. UAFs are a classic class of kernel vulnerabilities that can lead to arbitrary memory corruption, privilege escalation, or even remote code execution. Defenders should prioritize patching UAF vulnerabilities quickly. Kernel developers should explore memory safety mechanisms (e.g., Kernel Address Space Layout Randomization (KASLR), Slab Quarantine, KFENCE, and other hardening features) that can mitigate the impact or detect UAFs early in development or testing. While these don't prevent the bug, they make exploitation significantly harder.

1. Understanding KernelCTF as a Threat Model: The existence and success of exploits against kernelCTF highlight that even highly scrutinized and well-mitigated kernel environments can be compromised. This serves as a reminder that sophisticated attackers are actively seeking and exploiting subtle logic flaws in complex kernel subsystems. Organizations should consider adopting similar rigorous security testing methodologies, including internal CTF-like challenges or dedicated red-teaming efforts against their critical systems, to proactively uncover such vulnerabilities.

In essence, Chen's research is a call for greater vigilance in designing, implementing, and securing complex kernel components, emphasizing that even minor logical inconsistencies in state management can have profound security implications.

Key Takeaways

• NS tables is a powerful, yet complex, packet filtering framework in the Linux kernel, consolidating legacy systems with a VM design and batch processing.
• KernelCTF provides significant bounties for kernel exploits, making it a high-stakes environment for security researchers targeting components like NS tables.
• Sophisticated object lifetime management in NS tables uses current generation and next generation states to ensure atomicity during batch operations.
• A critical vulnerability was identified due to an inconsistent state check (NFT is active vs. is active next) during the abort phase of batch requests for set element deletions.
• This inconsistency can lead to Use-After-Free (UAF) vulnerabilities, allowing an attacker to manipulate freed memory and potentially achieve kernel privilege escalation.
• Defensive strategies include rigorous code review, static analysis, prompt patching, and leveraging kernel memory safety mitigations to counter such complex logical flaws.

About the Speaker(s)

HexRabbit Chen is a security researcher based in Taiwan, currently working at Devcore. His specialization lies in binary exploitation, with a particular focus on Linux kernel exploitation. He has a notable track record of discovering vulnerabilities in various Linux kernel components, including io_uring, KSMBD, and NS tables, among others. His presentation at DEF CON 32 showcases his expertise in identifying subtle, yet critical, flaws in complex kernel subsystems and leveraging them for exploitation.