There and Back Again: Detecting OT Devices Across Protocol Gateways

There and Back Again: Detecting OT Devices Across Protocol Gateways

Speakers: Rob King

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=YBPYYk8FIkc

Overview

In the realm of Operational Technology (OT) and Industrial Control Systems (ICS), maintaining a comprehensive inventory of devices is paramount for security and operational integrity. Rob King's DEF CON talk, "There and Back Again: Detecting OT Devices Across Protocol Gateways," delves into the intricate challenges of discovering these critical assets, particularly when they reside behind complex protocol gateways. King, an expert in device discovery and fingerprinting, highlights that while the convergence of OT with IP networks offers undeniable advantages in terms of cost and integration, it simultaneously introduces significant security blind spots and complicates traditional discovery methods.

The presentation systematically unpacks how legacy industrial protocols, not originally designed for modern network discovery, are adapted to IP networks, often by simply encapsulating their entire message structures within TCP/IP. This adaptation, while functional, creates hurdles for security practitioners attempting to gain visibility into their OT environments. King's talk provides a crucial technical deep dive into the specifics of Modbus, DNP3, and Ethernet/IP, demonstrating how to leverage specific protocol features and sometimes counter-intuitive behaviors to pull detailed device identification information from behind these gateways, ultimately enabling more robust asset management and defensive postures in industrial settings.

The significance of this work lies in its direct applicability to critical infrastructure protection. As industrial environments increasingly connect to broader enterprise networks and even the internet, understanding exactly what devices are present, their capabilities, and their configurations becomes a foundational security requirement. King's research offers practical methodologies to achieve this granular visibility, moving beyond superficial network scans to uncover hidden OT assets and their specific attributes, thereby empowering defenders to secure systems that control real-world physical processes.

Background

▶ Watch: Introduction to discovering OT devices across gateways (0:00)

The evolution of industrial control technology has been a journey from manual operation to highly sophisticated automated systems. Initially, control involved human operators directly interacting with machinery via buttons and levers. This progressed to simple electronic logic, allowing for basic automation, but still requiring physical presence to interact with controllers. The concept of Distributed Control Systems (DCS) emerged, centralizing control and monitoring into single rooms, extending control wires to provide a holistic view of factory operations. However, this still entailed vast amounts of dedicated wiring, leading to complex and often problematic physical infrastructures, as dramatically illustrated by incidents like the Three Mile Island near-meltdown, where a stuck mechanical gauge played a role.

The advent of computer intelligence brought about the idea of bus architectures, significantly reducing cabling complexity. Early examples like the Hewlett-Packard Instrument Bus (HPIB), adapted even for consumer devices like the Commodore 64, allowed multiple devices to communicate over a single set of wires, addressing devices by unique identifiers. While HPIB was suitable for laboratory settings, it lacked the robustness for industrial environments. This led to the development of protocols specifically for industrial use, with Modicon Modbus (introduced in 1978) being a seminal example. Modbus, now an open standard, offered a simple bus architecture for industrial controllers, primarily on serial lines.

As technology advanced, the undeniable advantages of IP and Ethernet convergence began to transform OT networks. Ethernet, being a commodity technology, offered lower costs, easier integration, and higher bandwidth. This led to industrial protocols being adapted to run over TCP/IP, creating "industrial Ethernet" variants. While this convergence streamlined operations and integration, it also introduced significant security risks, as industrial systems could now be directly exposed to the internet, a practice King explicitly cautions against. The core problem for security practitioners in this converged landscape is the difficulty of device discovery. Older protocols like Modbus were not designed with extensive self-describing metadata or discovery mechanisms, assuming operators had intimate knowledge of their fixed factory layouts. In today's dynamic, often sprawling industrial environments, including the rise of "shadow OT," this assumption no longer holds, making robust discovery essential.

Key Findings

▶ Watch: Distributed Control Systems (DCS) for centralized monitoring (2:00)

Rob King's research reveals specific, actionable techniques for discovering OT devices across protocol gateways, addressing the limitations of older industrial protocols when exposed to IP networks. His key findings center on exploiting protocol-specific features to extract detailed identification information.

For Modbus, King highlights the protocol's inherent simplicity and lack of rich metadata in its original design. While basic Modbus can report a Server ID (function code 0x11), this provides minimal information, typically just a unit ID. The significant breakthrough for Modbus discovery came with Modbus 1.1 in 2004, which introduced the Encapsulated Interface Transport (MEI). This allows for more complex, object-oriented operations to be "shoved" within the Modbus payload. Crucially, the Read Device Identification MEI type (0x0E) enables devices to return comprehensive details such as vendor name, product code, major/minor revision, and often a vendor URL and product name. This feature transforms Modbus discovery from a basic address check into a rich information-gathering process.

DNP3, a more sophisticated protocol primarily used in utility distribution, presents a different set of discovery challenges. When adapted to TCP/IP, the entire DNP3 protocol stack is encapsulated. A key finding for DNP3 is the reliance on unsolicited responses. Many DNP3 devices, upon connection, will send an unsolicited message containing a full DNP3 transport layer, including the device's own address (source) and its designated primary device's address (destination). This allows a discovering system to learn the device's DNP3 address and, critically, the address it expects to communicate with, which must be spoofed to query the device further. If unsolicited responses are not sent (which can vary by configuration), the fallback is to enumerate the DNP3 address space, typically focusing on the lower range of addresses where most devices are configured. Once an address is determined and the primary spoofed, specific DNP3 attributes like software version, hardware version, and hostname can be queried.

The most potent discovery technique discussed is for Ethernet/IP, an adaptation of the Common Industrial Protocol (SIP) for IP networks. While Ethernet/IP offers a straightforward List Identity command (0x63) that can be sent via UDP broadcast or TCP to identify the adapter itself, King's core finding is its capability for recursive discovery. By connecting to an Ethernet/IP adapter and leveraging SIP's object model (specifically the connection manager object), it's possible to establish connections to other devices residing behind the adapter on its backplane or sub-networks. This is achieved by constructing a source route path (e.g., specifying a slot number like slot 12) and forwarding Get Attributes All requests to the identity object of these internal devices. This allows a single query to an Ethernet/IP adapter to enumerate and gather detailed identification attributes from an entire rack of connected devices, potentially spanning different underlying protocols (e.g., serial) that the adapter bridges. This recursive capability provides unparalleled visibility into complex, multi-device OT systems.

Technical Deep Dive

▶ Watch: The advent and importance of Modbus (4:00)

The technical core of Rob King's presentation lies in dissecting how industrial protocols are adapted for IP networks and how their inherent features can be exploited for discovery.

For Modbus, the adaptation to TCP/IP is remarkably simple. Modbus TCP literally takes the core Modbus Application Protocol Data Unit (APDU) and prefixes it with a 7-byte Modbus Application Protocol (MBAP) header. This header provides transaction identifiers, protocol identifiers (always 0x0000 for Modbus TCP), and the length of the subsequent Modbus PDU. The entire Modbus PDU is then "shoved" into the TCP data segment. The original Modbus serial protocol, developed in 1978, was minimalist, designed for efficiency over low-bandwidth serial lines. Its function codes, such as Read Holding Registers (0x03) or Write Single Register (0x06), primarily deal with reading and writing numerical values or coil states. The Report Server ID (0x11) function provides only a unit ID and some basic status. This inherent simplicity means standard Modbus TCP offers very little in the way of device self-identification.

The critical enhancement for Modbus discovery is the Encapsulated Interface Transport (MEI), introduced in Modbus 1.1. This mechanism uses function code 0x2B (or 43) to wrap more complex, higher-level operations within a standard Modbus frame. Within MEI, the specific sub-function 0x0E corresponds to Read Device Identification. This allows a client to query a Modbus device for structured identification data. The response is object-oriented, returning a series of objects, each with a specific ID (e.g., 0x00 for vendor name, 0x01 for product code, 0x02 for major/minor revision). Devices supporting MEI are mandated to provide at least these core identification objects, with many also offering additional details like vendor URL (0x03) and product name (0x04). This transforms a blind Modbus connection into a source of rich, descriptive metadata.

DNP3 (Distributed Network Protocol 3) is considerably more complex than Modbus. Designed later, it incorporates its own robust framing, checksumming, transport layer, and a more sophisticated addressing scheme. When adapted for IP networks, the entire DNP3 protocol stack, from application to pseudo-transport layers, is encapsulated directly over TCP. This design choice, while simplifying code reuse across physical media, makes discovery tricky. A DNP3 device on an IP network has both an IP address and a DNP3 address. Crucially, DNP3 devices are typically configured to only communicate with a designated "primary" device, identified by its DNP3 address. An unsolicited connection from an unknown source will often yield no information. To query a DNP3 device, a discovering system must not only know the device's DNP3 address but also spoof the DNP3 address of its primary. This is where unsolicited responses become invaluable. Many DNP3 devices, when connected to, will spontaneously send a DNP3 message (e.g., status updates, clock sync requests). This message, being a full DNP3 frame, contains both the device's own DNP3 source address and its configured primary's DNP3 destination address. By capturing this, an attacker or defender can learn the necessary addresses to construct valid DNP3 queries for attributes like software version, hardware version, and hostname. Without an unsolicited response, the only recourse is to enumerate the entire 65,535 DNP3 address space while spoofing every possible primary address, a time-consuming process often mitigated by the tendency for networks to use addresses in the lower ranges (e.g., 0-1000).

Ethernet/IP is an adaptation of the Common Industrial Protocol (SIP) for IP networks. It's important to note the "IP" in Ethernet/IP stands for "industrial protocol," not "Internet Protocol," leading to potential confusion when talking about "Ethernet/IP on top of IP." SIP is an abstract, object-oriented protocol that defines a standard set of objects, messages, and attributes. Ethernet/IP is the specific mapping of SIP over TCP/IP. The protocol offers a built-in discovery mechanism called List Identity (command 0x63). This command can be sent via UDP broadcast or TCP to an Ethernet/IP device (typically an adapter or controller) and will return basic identification information about that specific device, such as its vendor, product type, and revision.

However, the true power of Ethernet/IP for discovery lies in its ability to facilitate recursive enumeration of devices on an internal backplane or sub-network. An Ethernet/IP adapter often acts as a gateway to multiple other devices (PLCs, motor controllers, I/O modules) connected via a proprietary backplane or another serial protocol. By establishing a SIP session with the adapter, a client can interact with specific SIP objects. The connection manager object within SIP allows a client to request a connection to another device that the adapter manages. This process involves constructing a source route path, which specifies how to reach the target device through the adapter (e.g., Port Segment, Address, where the address might be a slot number on a backplane). Once a connection path is established, the client can then forward a Get Attributes All request (specifically for instance 1 of the identity object) through the adapter to the target device. The adapter acts as a transparent proxy, relaying the request and the response. This recursive capability means that a single Ethernet/IP connection can be used to discover and fingerprint every device within an entire rack or even cascaded racks, regardless of the underlying backplane protocol. This provides an unprecedented level of granular visibility into complex, multi-component OT systems.

Demo / Proof of Concept

▶ Watch: Convergence of OT systems with IP and Ethernet (5:30)

While Rob King's talk did not feature a live, interactive demonstration, he presented compelling evidence and results from his lab setups, which served as a proof of concept for each discovery technique. He explicitly noted the arduous process of setting up the DNP3 lab environment, stating, "I hated my life, but I got it working," underscoring the practical challenges involved in working with these specialized protocols.

For Modbus, King illustrated the output of a successful Read Device Identification request using the MEI (0x2B function code, 0x0E MEI type). The presented results clearly showed the retrieval of structured data, including the vendor name, product code, major revision, and minor revision. This demonstrated that the Modbus 1.1 extension effectively provides the richer metadata necessary for detailed asset identification, moving beyond simple register values.

For DNP3, the proof of concept focused on the critical role of unsolicited responses. King showed how, upon connecting to a DNP3 device, an unsolicited message could be captured. This message, as described, contained the full DNP3 transport layer, revealing both the device's own DNP3 source address and the DNP3 address of its designated primary. This capture is fundamental because it provides the necessary parameters (device address and primary address to spoof) to then send legitimate queries. Subsequently, King presented output demonstrating the successful retrieval of device attributes such as software version, hardware version, and hostname once the correct DNP3 addresses were identified and spoofed.

The most elaborate proof of concept related to Ethernet/IP's recursive discovery. King presented a diagram illustrating a scenario where an Ethernet/IP adapter acts as a gateway to a rack containing multiple devices, such as a PLC, a motor controller, a thermal monitor, and a "spline reticulator." He then showed the command structure used to query devices deeper within this hierarchy. The key was the construction of a source route path, such as specifying a port segment and an address (e.g., slot 12 on the backplane) to target a specific device behind the adapter. The resulting output demonstrated that, by forwarding Get Attributes All requests to the identity object of these internal devices through the Ethernet/IP adapter, it was possible to enumerate and retrieve detailed identity attributes for each device in the rack, not just the adapter itself. This effectively proved that a single network connection to the gateway could yield a comprehensive inventory of all connected components, even if they communicate over different internal protocols (like RS232 or a proprietary backplane).

These proofs of concept, though presented as captured outputs rather than live demos, effectively validated the speaker's claims about the feasibility and effectiveness of these specialized discovery techniques across different OT protocols.

Defensive Implications

▶ Watch: The challenge of discovering connected OT devices (6:50)

The techniques presented by Rob King have profound implications for OT network defenders, highlighting both vulnerabilities and opportunities for improved security posture. The core defensive takeaway is the absolute necessity of comprehensive asset visibility in OT environments. The existence of "shadow OT" and the inherent opaqueness of devices behind protocol gateways mean that many organizations likely have critical assets operating without proper inventory or monitoring.

Defenders must actively implement and utilize specialized discovery mechanisms, such as those detailed in the talk, to identify all devices on their OT networks, including those nested behind gateways. Relying solely on IT-centric scanning tools will miss a significant portion of the OT attack surface. This includes:

• Leveraging Modbus 1.1 MEI for rich device identification: This allows for more precise inventorying of Modbus devices, moving beyond generic "Modbus device" entries to specific vendor, product, and revision details. This information is crucial for vulnerability management and patching strategies.
• Implementing DNP3 primary spoofing and unsolicited response monitoring: Defenders should proactively connect to DNP3 devices to capture unsolicited responses, thereby learning their DNP3 addresses and expected primary relationships. This knowledge is vital for legitimate querying and for detecting anomalous DNP3 communications. If unsolicited responses are disabled or infrequent, targeted enumeration of common DNP3 address ranges should be performed.
• Utilizing Ethernet/IP's recursive discovery for deep inventory: This is perhaps the most powerful technique for gaining visibility into complex OT racks. Defenders should employ tools that can construct source route paths and query internal SIP objects to enumerate every component within an Ethernet/IP connected system. This reveals devices that would otherwise be completely hidden behind an adapter, providing a critical understanding of the full system's composition.

Beyond discovery, the talk implicitly highlights several other defensive considerations:

• Network Segmentation and Gateway Security: The example of publicly exposed OT devices underscores the critical need for strict network segmentation. Protocol gateways, while enabling IT/OT integration, become crucial chokepoints. They must be hardened, regularly patched, and their configurations carefully reviewed to limit exposure and control access.
• Configuration Management: Understanding how DNP3 devices are configured regarding unsolicited responses (e.g., frequency, conditions for sending) is important. While these responses aid discovery, they also leak internal network information. A careful balance must be struck between operational needs and security implications.
• Supply Chain Security and Bill of Materials (BOM): The ability to discover specific vendor, product, and revision information directly from devices aids in validating components against expected BOMs and identifying unauthorized or vulnerable equipment introduced into the environment.
• Anomaly Detection: Once a baseline of device identities and configurations is established through deep discovery, defenders can better detect anomalies. For instance, an unexpected device appearing behind an Ethernet/IP gateway or a DNP3 device communicating with an unfamiliar primary address could indicate compromise or misconfiguration.

In essence, King's research empowers defenders to overcome the historical opacity of OT networks. By understanding and implementing these protocol-specific discovery methods, organizations can build a truly comprehensive asset inventory, a foundational element for any robust cybersecurity program in industrial environments.

Key Takeaways

• OT Device Discovery is Inherently Complex: Traditional IT scanning tools are often insufficient for accurately identifying and inventorying devices in Operational Technology (OT) environments due to the use of specialized, often legacy, industrial protocols and the prevalence of protocol gateways.
• Modbus 1.1 MEI is Crucial for Rich Identification: While basic Modbus offers limited metadata, the Encapsulated Interface Transport (MEI) introduced in Modbus 1.1 (function code 0x2B, MEI type 0x0E) enables the retrieval of detailed vendor, product, and revision information, significantly enhancing Modbus device inventory.
• DNP3 Discovery Relies on Unsolicited Responses and Spoofing: Identifying DNP3 devices and querying their attributes often requires capturing unsolicited responses to learn their internal DNP3 address and the address of their designated primary, which then needs to be spoofed for further communication. Without this, extensive address enumeration is necessary.
• Ethernet/IP Offers Powerful Recursive Discovery: By leveraging the Common Industrial Protocol (SIP) and its connection manager object within an Ethernet/IP session, it's possible to construct source route paths to enumerate and gather detailed identity attributes from multiple devices residing behind an Ethernet/IP adapter on its backplane or sub-networks.
• Protocol Gateways Obscure Visibility: While essential for IT/OT convergence, protocol gateways can create significant blind spots for defenders, hiding numerous critical assets unless specialized, protocol-aware discovery techniques are employed to look "through" the gateway.
• Comprehensive Asset Inventory is Foundational for OT Security: Gaining granular visibility into all OT devices, including those deeply embedded behind gateways, is paramount for effective vulnerability management, configuration control, anomaly detection, and overall cybersecurity posture in industrial control systems.

About the Speaker(s)

Rob King is a security researcher with a clear passion for network discovery and device fingerprinting. Throughout his career, he has dedicated significant effort to identifying and understanding devices on networks, a task he describes as "super fun." This enthusiasm for uncovering hidden assets and understanding their unique characteristics underpins his expertise in dissecting industrial protocols and developing novel methods for OT device identification. His work focuses on practical, technical solutions to complex visibility challenges in critical infrastructure environments.