Visibility Metrics

Visibility Metrics: Defining the Intangible in ICS Security

Speakers: Oscar Delgado

Conference: S4

YouTube: https://www.youtube.com/watch?v=wYSLFLsHeHg

Overview

In the complex and often opaque world of Industrial Control Systems (ICS) and Operational Technology (OT) security, the concept of "visibility" is frequently invoked yet rarely precisely defined. Oscar Delgado’s S4 talk, "Visibility Metrics," tackles this fundamental challenge head-on, arguing that while visibility is universally acknowledged as critical, it remains a subjective, intangible, and unstandardized metric. This talk introduces a structured, rational, and repeatable methodology for organizations to define, measure, and improve their visibility in a way that directly supports their overarching business objectives.

Delgado draws from his experience as an industrial consultant and later as a detection engineer, illustrating how a lack of a common framework for visibility can lead to significant disagreements and inefficient resource allocation. He proposes a pragmatic approach inspired by how healthcare professionals assess "overall health" – an equally intangible concept – by relying on measurable proxies or vitals. The core message is that visibility should not be pursued for its own sake but as a targeted tool, with its scope and depth dictated by specific organizational needs and use cases. This talk offers a crucial framework for ICS/OT security practitioners to move beyond vague notions of "more visibility" towards a strategic, cost-effective, and actionable plan.

The importance of this discussion for the ICS community cannot be overstated. As cyber threats against critical infrastructure continue to escalate, understanding what is happening within OT environments is paramount for effective defense. However, the unique characteristics of these environments – legacy systems, proprietary protocols, and the imperative of uptime – make comprehensive data collection both technically challenging and prohibitively expensive if not approached strategically. Delgado's methodology provides a much-needed guide for organizations to prioritize their visibility investments, ensuring they gather the right information to address their most pressing security and operational concerns, rather than drowning in irrelevant data.

Background

▶ Watch: Realizing the lack of a standard definition for visibility (0:00)

The genesis of Oscar Delgado’s exploration into visibility metrics stems from a direct professional challenge. As an industrial consultant, he once presented a client with a report stating they had "low visibility." The client vehemently disagreed, asserting they had "medium visibility." This seemingly minor semantic dispute highlighted a profound underlying problem: the absence of a standardized, objective definition for visibility. Both parties were looking at the same evidence, yet drawing different conclusions because their individual concepts of visibility were shaped by different backgrounds and perspectives. This incident underscored the critical need for a common language and a measurable framework.

The client’s subsequent question further solidified this need: assuming low visibility, how could they prioritize smart investments to move towards medium visibility? This question demanded a system that transcended subjective opinion, one that was rational, repeatable, and traceable. Rationality ensures that the approach is based on sound reasoning; repeatability allows for consistent measurement over time to track progress; and traceability provides a clear audit trail for decision-making.

Delgado realized this wasn't a problem unique to ICS. He observed how other fields successfully measure intangible concepts. A particularly compelling analogy came from healthcare: how is a patient's "overall health" assessed? It's an intangible concept, but doctors don't measure "health" directly. Instead, they examine vitals – temperature, blood pressure, heart rate, etc. These vitals are proxies; individually, they provide specific information, but collectively, they paint a comprehensive picture of the patient's health. Crucially, these proxies are not random; they are chosen because they answer specific diagnostic questions and relate directly to the target state (health). This insight became the cornerstone of Delgado’s proposed methodology for visibility.

The traditional, often default, approach to visibility is to "collect everything." Delgado strongly cautions against this. While seemingly comprehensive, it is both dangerous and inefficient. A data "swamp" or "lake" full of undifferentiated information makes it exceedingly difficult for security and operations teams to extract valuable insights. Essential information becomes buried under layers of irrelevant data, forcing analysts to spend inordinate amounts of time digging. Furthermore, data collection is expensive. Storage, transfer, and processing of vast quantities of data require significant infrastructure investments, potentially necessitating upgrades to existing assets. This "collect everything" mentality, therefore, often leads to diminishing returns, increased operational overhead, and a false sense of security. The underlying problem persists: without a clear purpose, even abundant data fails to provide meaningful visibility.

Key Findings

▶ Watch: Defining requirements: rational, repeatable, and traceable visibility (2:00)

Oscar Delgado's talk distills several critical findings regarding visibility in ICS/OT environments, offering a pragmatic pathway to address its inherent subjectivity and intangibility.

Firstly, the talk firmly establishes that visibility is subjective and intangible. There is no universal scale or standard definition for what constitutes "low," "medium," or "high" visibility. This inherent subjectivity is a primary driver of miscommunication and misaligned expectations within organizations. To overcome this, Delgado asserts that visibility can be measured, but only if we leverage proxies. These proxies are specific, measurable pieces of information that, when collected and analyzed, provide insights into the broader, intangible concept of visibility. Much like medical vitals informing overall health, these data points serve as indicators for the state of security or operations.

Secondly, the most crucial prerequisite for defining and measuring visibility is to define the purpose first. Visibility should never be an end in itself; it is a means to achieve specific organizational goals. As Delgado highlights, a refinery manager's primary goal is to produce gasoline, diesel, and jet fuel, while a water treatment plant director aims to provide clean water. Visibility, in this context, is a tool that various teams (IT, OT, security) use to support these core business objectives. Without a clear understanding of why certain information is needed and how it will be used, any data collection effort risks being wasteful and ineffective. This "purpose-first" approach ensures that resources are directed towards gathering data that directly contributes to risk reduction, operational efficiency, or compliance.

Thirdly, Delgado introduces a foundational framework for any robust visibility system: it must be rational, repeatable, and traceable.

• Rationality ensures that the data being collected and the methods used to interpret it are logically sound and aligned with stated objectives.
• Repeatability means that the process of measuring visibility can be consistently applied over time, allowing organizations to track progress, evaluate the impact of changes, and benchmark their posture.
• Traceability provides an audit trail, enabling teams to understand why specific decisions were made regarding visibility, how the data was gathered, and how it informed subsequent actions. This is crucial for accountability and continuous improvement.

Finally, the talk emphasizes a three-step validation process for identifying and prioritizing effective proxies:

1. Is someone going to use this information? This first question filters out irrelevant data. If no team or individual has a legitimate need for a particular piece of information, it should not be collected.
2. How are they going to use it? This question is paramount as it precisely defines what specific data points are required. For instance, if a team needs to identify vulnerabilities, knowing the model, vendor, firmware version, and series of a hardware asset is crucial. Knowing its color, however, is typically irrelevant unless a highly specific use case (e.g., physical security inventory based on color-coding) can be articulated. This step encourages a "greedy" approach to data collection – only gathering exactly what is needed, no more.
3. The process for defining and implementing visibility must be flexible and subject to regular review. Organizational needs, threat landscapes, and technological capabilities are constantly evolving. What constitutes valuable visibility today might change tomorrow. Therefore, the system should not be considered "perfect" or static. Regular feedback loops and scheduled reviews are essential to adjust the design, incorporate new insights, and ensure ongoing alignment with evolving business and security requirements. Documentation of the process is critical to facilitate these reviews and maintain traceability.

These findings collectively offer a powerful paradigm shift for organizations struggling with visibility, particularly in complex ICS/OT environments. By focusing on purpose, leveraging carefully selected proxies, and adhering to a rational, repeatable, and traceable methodology, companies can transform visibility from an elusive ideal into a measurable and actionable strategic asset.

Technical Deep Dive

▶ Watch: Patient health analogy for measuring intangible concepts (4:00)

While Oscar Delgado's talk does not delve into specific code implementations or network protocols in the traditional sense of a "technical deep dive," it presents a robust methodological framework that is inherently technical in its approach to problem-solving. This framework provides the engineering principles necessary to construct a practical, data-driven visibility system for ICS/OT environments. The "technical" aspect here lies in the structured, systematic design of data collection and interpretation.

The core technical challenge addressed is how to operationalize the measurement of an intangible concept like "visibility." Delgado's solution is rooted in the strategic identification and utilization of proxies. A proxy, in this context, is a specific, discrete, and measurable data point that serves as an indicator for a broader state of visibility. For example, if the desired visibility is "vulnerability posture of hardware assets," technical proxies would include:

• Hardware Model Number: e.g., Siemens S7-1500 CPU 1516-3 PN/DP
• Vendor: e.g., Siemens
• Firmware Version: e.g., V2.9.4
• Serial Number/Series: e.g., 6ES7516-3AN02-0AB0

These specific data points are technically derivable and directly inform the vulnerability management team about potential exposures (e.g., CVEs associated with V2.9.4 firmware on a specific Siemens model). Conversely, information like the physical color of the equipment, while technically observable, is typically not a relevant proxy for cybersecurity visibility unless a very specific and documented use case dictates otherwise.

The proposed system for visibility must embody three fundamental engineering principles: rationality, repeatability, and traceability.

1. Rationality: This implies that the selection of proxies and the logic for their aggregation must be sound and justifiable. For instance, collecting Modbus TCP traffic on a specific segment is rational if the goal is to identify unauthorized commands or changes to process values. It would be irrational if the segment exclusively carries OPC UA traffic and the detection rules are designed only for Modbus. The technical implementation must reflect this logical alignment.
2. Repeatability: Any measurement of visibility must produce consistent results under the same conditions. This necessitates well-defined data collection mechanisms, standardized data formats, and consistent processing pipelines. If an organization measures the number of known vulnerabilities in its OT environment, the method of asset discovery, vulnerability scanning, and reporting must be repeatable to allow for tracking progress over time. This often involves automated tools, clear configuration management, and documented procedures.
3. Traceability: Every piece of information collected, every proxy defined, and every visibility metric derived must be traceable back to its source and its purpose. This means documenting data collection points (e.g., network taps, host agents, PLC logs), the transformations applied to the data, and the specific organizational need it fulfills. In a technical sense, this translates to robust logging, metadata management, and potentially data lineage tools to understand the provenance and journey of visibility data.

Delgado emphasizes that the definition of visibility is not static; it must be adaptive and periodically reviewed. This requires a feedback loop mechanism. Technically, this could involve:

• Regular reporting and dashboarding: Presenting visibility metrics to relevant stakeholders.
• Automated alerts and notifications: Triggering when visibility gaps are identified or when new information is required.
• Configuration management: Storing and versioning the definitions of proxies, data collection policies, and visibility metrics, allowing for easy updates and rollbacks.
• Performance monitoring: Tracking the efficiency and cost-effectiveness of data collection pipelines to ensure they remain optimized.

The talk also implicitly addresses the technical implications of data cost. When advocating against collecting "everything," Delgado points to the expense of storage, transfer, and processing. In an ICS context, this is particularly acute:

• Storage: Raw packet captures from high-bandwidth OT networks can quickly consume massive storage, requiring expensive enterprise solutions.
• Transfer: Moving large volumes of data from remote plant sites to centralized security operations centers (SOCs) can strain limited network bandwidth, especially in environments with constrained infrastructure.
• Processing: Analyzing vast datasets for anomalies or specific events demands significant computational resources, often requiring specialized Big Data platforms or high-performance analytics engines.

By adopting a "greedy" approach – collecting only the information needed to serve a defined purpose – organizations can technically optimize their data infrastructure, reduce operational costs, and improve the signal-to-noise ratio for their security analysts. This means carefully configuring sensors, logging levels, and data aggregation points to capture only relevant fields or events, rather than indiscriminately ingesting all available data. For example, instead of streaming all Modbus packets, one might configure an IDS/IPS to log only specific function codes or anomalous requests that align with a known threat model.

In essence, Delgado provides a technical blueprint for designing an intelligent, purpose-driven data acquisition strategy for ICS security. It's about applying sound engineering principles to an abstract problem, ensuring that every byte of collected data serves a specific, documented, and reviewed purpose, thereby maximizing its value for defense and operations.

Demo / Proof of Concept

▶ Watch: Understanding 'proxies' and the importance of defining purpose (5:14)

Oscar Delgado's talk, "Visibility Metrics," focuses on a conceptual and methodological framework rather than demonstrating a specific tool, exploit, or technical proof of concept. The presentation outlines a strategic approach to defining and measuring visibility within an organization, particularly in the context of Industrial Control Systems (ICS) and Operational Technology (OT). Therefore, there was no live demo or technical proof of concept presented during the talk. The speaker elaborated on theoretical models and practical considerations for implementing a rational, repeatable, and traceable visibility system.

Defensive Implications

▶ Watch: The dangers of trying to 'collect everything' for visibility (6:00)

Oscar Delgado's framework for defining and measuring visibility has profound implications for defensive strategies in ICS/OT environments. By advocating for a purpose-driven, rational, repeatable, and traceable approach, the talk empowers defenders to build more effective, efficient, and sustainable security programs.

Firstly, the "purpose-first" principle directly informs threat detection and incident response. Instead of attempting to monitor every possible data point, defenders can identify specific threats relevant to their critical assets and operations (e.g., unauthorized PLC programming changes, denial-of-service attacks on HMI, data exfiltration of intellectual property). The "how are you going to use it?" question then guides the selection of proxies. For detecting unauthorized PLC changes, proxies might include system logs indicating program uploads, network traffic showing unusual protocol commands (e.g., Modbus function code 0x17 Write Multiple Registers), or integrity checks on PLC logic files. For an article, the focus shifts to ensuring that the collected data directly supports the detection of these specific TTPs (Tactics, Techniques, and Procedures) outlined in frameworks like MITRE ATT&CK for ICS. This targeted approach prevents data overload, improves alert fidelity, and reduces the time to detect and respond to actual threats.

Secondly, the emphasis on a rational, repeatable, and traceable system is critical for vulnerability management and asset inventory. Defenders often struggle with incomplete or outdated asset inventories, making it impossible to assess their vulnerability posture accurately. Delgado's framework suggests that asset visibility should be driven by the need to manage vulnerabilities. The proxies (model, vendor, firmware, series) are precisely the technical details required to cross-reference with vulnerability databases (e.g., CISA advisories, NVD). By systematically collecting and maintaining this information, defenders can:

• Accurately identify vulnerable assets: Pinpoint exactly which devices are susceptible to known CVEs.
• Prioritize patching and remediation: Focus resources on the most critical vulnerabilities affecting essential systems.
• Track remediation efforts: Use the repeatable measurement process to demonstrate improvement over time.
• Justify security investments: Traceability allows defenders to show how visibility data informed decisions and improved the security posture, making a strong case for budget allocation.

Thirdly, the caution against collecting "everything" has significant implications for resource optimization and operational resilience. ICS environments often have bandwidth constraints, limited storage on edge devices, and strict uptime requirements. Indiscriminate data collection can degrade network performance, overwhelm data storage systems, and even impact control system operations. By being "greedy" and collecting only necessary proxies, defenders can:

• Reduce operational overhead: Minimize storage, processing, and network bandwidth costs.
• Improve data quality: Focus on clean, relevant data, making it easier for security tools and analysts to derive insights.
• Preserve system performance: Avoid impacting critical OT network and device functionality with excessive data extraction.
• Extend the lifespan of legacy equipment: Minimize the burden on older, less powerful devices that might struggle with extensive logging or data transmission.

Finally, the need for a flexible and continuously reviewed process is vital for maintaining an effective defensive posture against an evolving threat landscape. The ICS threat environment is dynamic, with new attack vectors and adversary techniques emerging regularly. Defenders must be prepared to:

• Adapt visibility requirements: As new threats emerge (e.g., novel ransomware variants targeting specific industrial protocols), the proxies collected might need to change to ensure detection capabilities.
• Incorporate feedback: Regularly solicit input from SOC analysts, OT engineers, and incident responders to refine what information is most useful during an investigation or for proactive defense.
• Document changes: Maintain clear records of how visibility definitions and data collection strategies have evolved, ensuring institutional knowledge is preserved and the system remains traceable.

In essence, Delgado provides a blueprint for a mature ICS security program that moves beyond reactive measures. By applying his principles, defenders can establish a proactive, intelligence-driven approach to visibility, ensuring they have the right information at the right time to protect critical infrastructure, manage risks effectively, and ultimately support the core mission of the organization.

Key Takeaways

• Visibility is Subjective and Intangible: There is no universal standard for defining or measuring visibility, leading to miscommunication and inefficient efforts.
• Measure Visibility through Proxies: Intangible concepts like "visibility" can be objectively measured by identifying and collecting specific, relevant data points, similar to how "overall health" is assessed via medical vitals.
• Define Purpose First: Visibility should always be a tool that serves specific organizational goals (e.g., vulnerability management, threat detection), not an end in itself. Ask "how will this information be used?"
• Avoid Collecting Everything: Indiscriminate data collection is inefficient, expensive (storage, transfer, processing), and can obscure valuable insights, creating data "swamps" rather than clarity.
• Implement a Rational, Repeatable, and Traceable System: A robust visibility framework must be logically sound, consistently applicable over time to track progress, and provide clear audit trails for decision-making.
• Embrace Flexibility and Continuous Review: Organizational needs and the threat landscape evolve, so visibility systems must be adaptable, documented, and regularly reviewed and adjusted based on feedback.

About the Speaker(s)

Oscar Delgado is a detection engineer, whose professional journey includes significant experience as an industrial consultant. This background provided him with firsthand insights into the challenges organizations face in gathering and interpreting information from complex industrial environments. His work with clients, including water treatment plants, highlighted the critical need for a structured approach to defining and measuring visibility, particularly given that OT technicians often manage IT, OT, and security responsibilities. His current role as a detection engineer further underscores his focus on actionable intelligence and effective security monitoring within operational technology.