ICSQuartz: Scan Cycle-Aware and Vendor-Agnostic Fuzzing for Industrial Control Systems

ICSQuartz: Scan Cycle-Aware and Vendor-Agnostic Fuzzing for Industrial Control Systems

Speakers: Corban Villa, Undergraduate, moment lab at NYU Abu Dhabi

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=RI8M5G1mBSA

Overview

In this compelling talk at the NDSS Symposium, Corban Villa from the moment lab at NYU Abu Dhabi presented "ICSQuartz," a novel fuzzing framework designed to enhance the security of Industrial Control Systems (ICS). The research addresses the critical need for robust vulnerability detection in the Programmable Logic Controllers (PLCs) that form the backbone of essential infrastructure like power plants and water treatment facilities. With the increasing convergence of operational technology (OT) and information technology (IT), these systems are now exposed to a broader spectrum of sophisticated cyber threats, necessitating advanced security mechanisms.

ICSQuartz distinguishes itself by introducing scan cycle awareness and a vendor-agnostic approach to fuzzing, overcoming significant limitations of existing state-of-the-art ICS fuzzers. The framework leverages an open-source compiler to achieve whitebox-like instrumentation, incorporating crucial features like code coverage and address sanitizers. This allows for more precise and efficient identification of vulnerabilities, including memory corruption issues that might otherwise go undetected. The talk underscored the framework's superior performance and its ability to uncover real-world vulnerabilities, marking a significant step forward in securing critical infrastructure against evolving cyber threats.

The importance of this work cannot be overstated, particularly in light of the escalating geopolitical tensions and the constant barrage of attacks targeting ICS environments, from the infamous Stuxnet to recent incidents affecting water and wastewater systems and infrastructure in Ukraine. By providing a more effective and adaptable method for discovering vulnerabilities before they are exploited, ICSQuartz offers a proactive defense mechanism crucial for maintaining the integrity and resilience of global critical infrastructure.

Background

▶ Watch: Introduction to ICS and critical infrastructure security (0:00)

Industrial Control Systems (ICS) and critical infrastructure, encompassing everything from power grids to water treatment plants, are increasingly under threat from cyberattacks. Historically, these systems were isolated and relied on physical security, but the modern landscape has drastically changed. The merging of Operational Technology (OT), which manages these physical processes, with Information Technology (IT) has introduced a new paradigm of vulnerabilities. PLCs, once simple, isolated boards, now frequently run full-fledged operating systems, complete with complex memory architectures susceptible to the same types of software supply chain attacks and memory vulnerabilities found in traditional IT systems. Furthermore, the push for remote monitoring and control capabilities means many of these devices are now connected to the internet, expanding their attack surface from local to global. Compounding these issues is the inherent fallibility of human-written software, leading to a proliferation of bugs and potential exploits.

Detecting these vulnerabilities before they can be exploited is paramount. Software fuzzing has emerged as one of the most effective techniques for this purpose. Fuzzing involves feeding a program with a vast array of malformed or unexpected inputs to identify crashes, errors, or unexpected behaviors that can indicate vulnerabilities. In the context of ICS, these inputs often represent sensor values, and the program under test is the control logic running on a PLC. The goal is to maximize code coverage (exploring as many execution paths as possible) and identify crashes, which are often prime candidates for exploits.

However, existing state-of-the-art ICS fuzzers, such as IC Fuzz and Field Fuzz, face significant limitations. Primarily, they operate as blackbox fuzzers. This means they interact with proprietary compilers and executables, making it difficult to gain deep insights into program execution. Traditional fuzzers often instrument binaries (e.g., ELF binaries) compiled with standard tools to add whitebox functionality like detailed code coverage tracking. In contrast, proprietary ICS compilers prevent such instrumentation, leading to vendor lock-in and making it challenging to develop universal fuzzing solutions.

Another major drawback of these blackbox fuzzers is their imprecision. They often lack essential tools like an address sanitizer, which is designed to detect memory errors such as out-of-bounds reads or writes. Without an address sanitizer, these fuzzers typically rely on the operating system catching severe memory violations that result in a segmentation fault. However, many memory vulnerabilities, particularly subtle out-of-bounds reads, might not cause a segfault but still represent critical security flaws.

Crucially, existing fuzzers also lack scan cycle awareness. ICS programs are inherently stateful. A PLC receives an input, processes it, and modifies its internal state. The next input is then processed based on this modified state, and this cycle continues, gradually building up the system's overall state. Simply testing all permutations of inputs in isolation is insufficient for comprehensive vulnerability detection. To truly assess an ICS program's security, fuzzers must test all possible states in conjunction with all input permutations, a challenge that blackbox and traditional fuzzing approaches struggle to address.

Key Findings

▶ Watch: Limitations of state-of-the-art ICS fuzzers (3:30)

The ICSQuartz framework introduces significant advancements in ICS vulnerability detection, yielding several key findings and contributions that address the limitations of prior work:

1. Superior Performance: ICSQuartz demonstrably outperforms state-of-the-art blackbox fuzzers, including IC Fuzz and Field Fuzz, by over an order of magnitude across critical metrics. This includes execution speed, time to first crash, and inputs to first crash. This dramatic improvement is largely attributed to ICSQuartz's ability to detach the fuzzing process from proprietary vendor runtimes and small, underpowered PLCs, allowing it to leverage powerful server hardware for faster execution.

1. Enhanced Precision and Coverage: By integrating code coverage instrumentation and an address sanitizer directly into a modified open-source compiler (Rusty ICS), ICSQuartz provides significantly increased precision for both code coverage analysis and vulnerability detection. This capability allows it to identify subtle memory vulnerabilities, such as out-of-bounds reads, that do not trigger segmentation faults and are therefore missed by fuzzers lacking an address sanitizer.

1. Novel Scan Cycle Mutation Strategies: The introduction of ICS-specific scan cycle mutation strategies is a core innovation. These strategies enable ICSQuartz to effectively explore the complex stateful nature of ICS programs. Through rigorous benchmarking against stateful programs intentionally designed with vulnerabilities, ICSQuartz consistently identified flaws in 10 out of 10 trials, while other fuzzers (AFL++, Field Fuzz, and IC Fuzz) failed to detect any. This confirms the critical importance and effectiveness of state-aware fuzzing for ICS.

1. Discovery of Real-World Vulnerabilities and CVE: ICSQuartz was successfully deployed against 61 ICS binaries from an open-source library, leading to the discovery of significant vulnerabilities. Notably, it uncovered a memory-related vulnerability in a month_to_string function, specifically an out-of-bounds read that was local to the program and did not cause a segmentation fault. This vulnerability was exclusively found by ICSQuartz and not by Field Fuzz or IC Fuzz, highlighting its unique precision. This discovery was reported and resulted in the disclosure of the first known Structured Text CVE.

1. Identification of Compiler Bugs: Beyond application-level vulnerabilities, ICSQuartz also identified a compiler bug within the open-source Rusty ICS compiler. This finding underscores the framework's ability to scrutinize the entire software supply chain for ICS, from the compiler itself to the generated control logic. The identified bug has since been patched, improving the overall security of the open-source ICS development ecosystem.

1. Community Contributions: The project contributes significantly to the security research community by releasing the developed benchmarks, enabling further research and comparative analysis of ICS fuzzing techniques. The framework's modularity also allows for the integration of other LLVM-based fuzzers, fostering collaboration and continuous improvement.

In essence, ICSQuartz establishes a new benchmark for ICS fuzzing by delivering a native, whitebox-like solution that not only outperforms existing blackbox methods but also introduces critical capabilities for stateful vulnerability detection and precise memory error identification, leading to the discovery of practical, high-impact security flaws.

Technical Deep Dive

▶ Watch: Addressing the critical issue of scan cycle awareness (4:40)

The ICSQuartz framework is meticulously designed to overcome the inherent challenges of fuzzing proprietary and stateful Industrial Control Systems. Its architecture can be broadly divided into a compilation pipeline and a fuzzing stage, with key contributions (highlighted in yellow in the original presentation) integrated throughout.

The foundation of ICSQuartz lies in its approach to the compilation pipeline. It primarily targets Structured Text, one of the IEC 61131-3 standard languages for programming PLCs, which resembles traditional C-like programming languages, supporting pointers and arrays. To facilitate broader adoption and ease of use, ICSQuartz incorporates compatibility layers to make the input process more plug-and-play.

A pivotal component is the use of Rusty ICS, an open-source ICS compiler. Unlike proprietary compilers that prevent introspection, Rusty ICS was modified by the ICSQuartz team. These modifications are critical:

1. Code Coverage Instrumentation: The compiler is instrumented to insert code that tracks which parts of the program's logic are executed by a given input. This code coverage feedback is essential for guiding the fuzzer towards unexplored execution paths, a cornerstone of effective fuzzing.
2. Address Sanitizer: A robust address sanitizer runtime is integrated. This powerful tool is designed to detect various memory safety errors, such as out-of-bounds reads/writes, use-after-free, and double-free, even if they don't immediately cause a program crash (segmentation fault). This significantly enhances the precision of vulnerability detection, enabling ICSQuartz to find subtle memory corruption bugs that blackbox fuzzers would miss.

The modified compiler then outputs an Intermediate Representation (IR). This IR is subsequently compiled with a custom backend that maps in the necessary runtime components for both the code coverage and address sanitizer. This entire process allows ICSQuartz to transform proprietary-like Structured Text logic into a fuzzer-friendly, instrumented binary that can be executed and analyzed with high fidelity.

Moving to the fuzzing stage, the generated IR is used to create a fuzz target. To simplify the process for operators, ICSQuartz automatically generates a harness. A harness is a small piece of code that wraps the target program, taking inputs from the fuzzer and feeding them to the program in the correct format. This automation eliminates the need for manual harness development, making the framework more accessible.

The core of the fuzzing stage is the ICSQuartz fuzzer itself. This fuzzer is developed based on the advanced fuzzing strategies found in AFL++ (American Fuzzy Lop Plus Plus), inheriting its efficiency and mutation capabilities. A key design principle of ICSQuartz is its modularity: while it provides its own fuzzer, the framework is designed to be compatible with other LLVM-based fuzzers such as AFL, libFuzzer, and Honggfuzz. This allows researchers and operators to leverage their preferred fuzzing engines or integrate future advancements in fuzzing techniques.

The output of the fuzzing process includes a fuzzing corpus (a collection of interesting inputs that achieved new code coverage or caused crashes) and fuzzing solutions (detailed reports on coverage achieved and identified crashes). These outputs can then be fed into standard security analysis tools like GDB (GNU Debugger) for dynamic analysis or Ghidra for reverse engineering and static analysis.

A critical aspect of the ICSQuartz methodology is cross-validation. Given the complexities of ICS environments and the potential for differences between compiled code and actual PLC behavior, findings are cross-validated. For instance, a vulnerability identified through fuzzing might be a bug in the compiler itself rather than the target program. By testing identified vulnerabilities on a development board or actual vendor PLCs, researchers can ensure that the discovered flaws are genuine and exploitable in real-world deployments. This step is crucial for preventing false positives and ensuring the practical relevance of the findings.

The framework's ability to detach runtime and vendor requirements from the fuzzing process is a major performance enabler. Instead of being confined to the limited computational resources of small PLCs, ICSQuartz allows fuzzing to occur on regular server hardware with the fastest available CPUs. This drastically increases execution speed, which is a primary driver of fuzzing effectiveness.

Finally, the most innovative technical contribution is the scan cycle awareness. ICS programs are inherently stateful: an input is processed, modifying the program's internal state, and subsequent inputs act upon this updated state. Traditional fuzzers often treat each input in isolation, failing to explore the vast state space that accumulates over multiple scan cycles. ICSQuartz introduces specific mutation strategies that are designed to investigate and manipulate this scan cycle mechanism. By intelligently mutating inputs across multiple program executions, ICSQuartz can explore how different input sequences, and the resulting state changes, can lead to vulnerabilities that would be missed by stateless fuzzing. This unique capability is what allowed ICSQuartz to consistently identify vulnerabilities in stateful benchmarks where other leading fuzzers failed entirely.

Demo / Proof of Concept

▶ Watch: ICSQuartz contributions: instrumentation and precise vulnerability detection (5:50)

While the talk did not feature a live, interactive demonstration, the presentation provided compelling evidence of ICSQuartz's effectiveness through its real-world vulnerability discoveries and comparative performance benchmarks. The framework's ability to unearth previously unknown flaws in established ICS software and tools serves as a robust proof of concept for its design principles and capabilities.

A primary highlight was the discovery of a memory-related vulnerability in a commonly used month_to_string function within ICS binaries. This vulnerability was an out-of-bounds read that occurred locally within the program's memory space and, critically, did not cause a segmentation fault. This type of subtle memory corruption is notoriously difficult for traditional blackbox fuzzers to detect, as they often rely on the operating system's crash detection mechanisms. ICSQuartz, armed with its integrated address sanitizer, was uniquely positioned to identify this precise error. The successful reporting of this flaw led to the disclosure of the first known Structured Text CVE, validating the framework's precision and real-world impact.

Further demonstrating its capabilities, ICSQuartz was run against a collection of 61 ICS binaries sourced from an open-source library. During this extensive testing, the framework also identified a significant compiler bug within the Rusty ICS compiler. This finding, which was subsequently patched, underscores ICSQuartz's ability to scrutinize the entire ICS software development ecosystem, from the low-level compiler to the final control logic.

The presentation also included quantitative results comparing ICSQuartz against state-of-the-art fuzzers like IC Fuzz and Field Fuzz. These benchmarks showed ICSQuartz outperforming them by "over an order of magnitude" in execution speed, time to first crash, and inputs to first crash. More critically, in benchmarks specifically designed to introduce vulnerabilities into stateful programs, ICSQuartz consistently detected flaws across 10 trials, while the other fuzzers failed to identify any. This consistent success in detecting vulnerabilities missed by others, particularly in stateful contexts, serves as a powerful demonstration of the practical advantages of ICSQuartz's scan cycle-aware and precision-enhanced approach. The combination of these specific vulnerability discoveries and the overwhelming performance and detection rate improvements constitute a strong proof of concept for the ICSQuartz methodology.

Defensive Implications

▶ Watch: ICSQuartz outperforms state-of-the-art fuzzers (8:00)

The findings and capabilities of ICSQuartz offer crucial insights and actionable strategies for defenders tasked with securing Industrial Control Systems. The framework highlights critical areas where current defensive postures may be insufficient and provides a roadmap for enhancing the resilience of critical infrastructure.

1. Embrace Whitebox Fuzzing and Instrumentation: The most significant implication is the urgent need to move beyond blackbox testing for ICS software. Defenders should advocate for and adopt development practices that allow for code coverage instrumentation and the integration of address sanitizers during the compilation of ICS logic. This means pushing for more open-source compilers or demanding vendor transparency and tooling that enables such deep analysis. Integrating these tools into the software development lifecycle (SDLC) can catch vulnerabilities much earlier.

1. Prioritize Memory Safety: The discovery of an out-of-bounds read that did not cause a segmentation fault underscores the prevalence and danger of subtle memory safety bugs. Defenders must assume that ICS software, like any complex software, contains such vulnerabilities. Implementing memory sanitizers as part of testing and deployment, even if it incurs a performance overhead in non-production environments, is essential for identifying these elusive flaws. Developers should adopt secure coding practices that minimize memory-related errors.

1. Implement Stateful and Scan Cycle-Aware Testing: The unique stateful nature of ICS programs necessitates a paradigm shift in testing methodologies. Traditional fuzzing that treats inputs in isolation is insufficient. Defenders should ensure that their testing strategies, whether manual or automated, incorporate scan cycle awareness. This involves testing sequences of inputs that build up and manipulate the program's state over time, rather than just individual inputs. Developing or adopting tools like ICSQuartz that can model and mutate inputs based on the evolving system state is crucial.

1. Cross-Validation is Paramount: The importance of cross-validating identified vulnerabilities on actual hardware or vendor-specific emulators cannot be overstated. Bugs found in development environments or through compiler instrumentation might not manifest identically on deployed PLCs. Defenders should establish rigorous validation processes to confirm the real-world exploitability and impact of any discovered vulnerability, ensuring that remediation efforts are focused on genuine threats.

1. Scrutinize the Entire Software Supply Chain: The discovery of a compiler bug in Rusty ICS highlights that vulnerabilities can exist at any layer of the ICS software supply chain, not just in the application logic. Defenders need to assess the security of the compilers, libraries, and development tools used to build ICS software. Where open-source components are used, contributing to their security and staying updated on patches is vital. For proprietary tools, demanding security assurances and vulnerability disclosure from vendors becomes critical.

1. Leverage Open-Source Tools and Benchmarks: ICSQuartz's release of its framework and benchmarks to the community provides valuable resources. Defenders and security teams should explore integrating such open-source tools into their security assessments. Utilizing standardized benchmarks allows for consistent evaluation of different security solutions and helps in identifying the most effective approaches for their specific ICS environments.

By integrating these defensive implications, organizations can move towards a more proactive, precise, and comprehensive approach to securing their critical Industrial Control Systems against the complex and evolving threat landscape.

Key Takeaways

• ICSQuartz is the first native, state-of-the-art fuzzer for Industrial Control Systems (ICS), designed to overcome the limitations of existing blackbox solutions.
• The framework significantly outperforms prior state-of-the-art ICS fuzzers (IC Fuzz, Field Fuzz) in execution speed, time to first crash, and inputs to first crash by over an order of magnitude, primarily by detaching fuzzing from proprietary vendor runtimes.
• Novel ICS-specific scan cycle mutation strategies enable ICSQuartz to effectively detect vulnerabilities in stateful ICS programs that are consistently missed by other fuzzers.
• ICSQuartz provides increased precision for code coverage and vulnerability detection through the integration of an address sanitizer and instrumentation into the open-source Rusty ICS compiler.
• The research led to the discovery of a memory-related CVE (an out-of-bounds read in a month_to_string function, the first known Structured Text CVE) and a compiler bug in the Rusty ICS compiler, both of which have been reported and patched.
• The framework is modular and vendor-agnostic, allowing for the use of other LLVM-based fuzzers (e.g., AFL, libFuzzer) and the release of new benchmarks to the community.

About the Speaker(s)

Corban Villa is an undergraduate researcher associated with the moment lab at NYU Abu Dhabi. His work, as presented in the ICSQuartz talk, focuses on advancing the security of Industrial Control Systems, particularly through innovative fuzzing techniques.