A SBOM'd Substation

A SBOM'd Substation

Speakers: Matt Wyckhouse

Conference: S4

YouTube: https://www.youtube.com/watch?v=JiI7NYEWC_s

Overview

This talk, officially titled "A SBOM'd Substation" and playfully presented as "The Quest for S-bombs and the Legend of the S-bomb Substation" at S4x24, chronicles a pioneering real-world endeavor to generate and operationalize Software Bill of Materials (S-bombs) for an entire industrial control system (ICS) substation. Led by Matt Wyckhouse, with significant contributions from Alex (implied to be from the asset owner, Southern Company), the presentation uses an engaging video game analogy to illustrate the formidable challenges and unexpected discoveries encountered during this ambitious project. The core mission, born from a roundtable discussion at S4x23, was to move beyond theoretical discussions of S-bombs and demonstrate their practical application in a critical infrastructure setting.

The significance of this undertaking cannot be overstated. As digital components proliferate within operational technology (OT) environments, understanding the software supply chain becomes paramount for cybersecurity. S-bombs offer a crucial mechanism for achieving this transparency, yet their implementation in complex, multi-vendor OT landscapes has remained largely an aspiration. This talk provides invaluable firsthand insights into the logistical, technical, and human hurdles involved, offering a pragmatic roadmap for other asset owners grappling with similar challenges. It highlights not just the technical complexities of S-bomb generation and analysis, but also the critical role of vendor engagement, internal collaboration, and persistent effort in advancing OT supply chain security.

Background

▶ Watch: Introduction to the S-bomb substation mission (0:00)

The genesis of this "quest" can be traced back to S4x23, where a pivotal roundtable discussion brought together a diverse group of stakeholders: asset owners, OT equipment suppliers, security vendors, and government representatives. The central question posed was whether, despite extensive discussions surrounding S-bombs, anyone was truly operationalizing them effectively within critical infrastructure. The consensus was that practical implementation remained elusive, leading to a direct challenge issued to Alex: "We need to S-bomb a substation." The target was a specific substation in Mississippi, with the initial understanding being a straightforward process of identifying devices, requesting S-bombs, and then leveraging them for security insights.

The initial idealized vision of this mission was depicted as a simple, almost utopian, "perfect asset inventory" leading directly to vendors providing S-bombs "on a silver platter." This was to be followed by a "magic machine" for interpretation and a "magic vulnerability eraser," culminating in harmonious collaboration between vendors and asset owners. However, the reality, as the speakers vividly described, proved to be far more akin to "Jumanji" than "Candyland." The journey quickly devolved into a "foreboding" landscape marked by "Vendor Valley," the "inventory jungle," and "internal politics point," highlighting the stark contrast between theoretical S-bomb benefits and the arduous path to achieving them in a live OT environment. This foundational struggle underscores why S-bomb adoption has been slow in OT, revealing the deep-seated complexities beyond mere technical specifications.

Key Findings

▶ Watch: Navigating Inventory Jungle: challenges of detailed software inventory (4:00)

The journey to S-bomb a substation uncovered a multitude of significant findings, challenging preconceived notions and exposing the intricate realities of OT supply chain security. Foremost among these was the profound difficulty of establishing a comprehensive and accurate asset inventory, particularly concerning software versions and firmware versions. The initial assumption of a readily available, perfect inventory quickly dissolved, revealing a fragmented landscape managed by multiple business partners and units across various network segments (control network, OT monitor, OT fault, physical security, cyber security, telecommunications). The physical walkthrough, involving photography and engagement with disparate teams, was crucial to identifying all 38 devices from 16 different vendors.

Another critical discovery lay within "Vendor Valley," which encapsulated the diverse and often challenging interactions required to obtain S-bombs. While some vendors were receptive, others were shrouded in "legal lair," necessitating extensive, persistent communication—sometimes up to 12 emails or meetings per vendor. This highlighted that S-bomb sharing remains "clunky" and heavily reliant on human interaction and relationship building. Furthermore, the project revealed significant inconsistencies in S-bomb quality and completeness. Through independent verification, particularly by analyzing unencrypted firmware, it was found that vendor-provided S-bombs often lacked crucial dependencies. One instance showed an average of 72 fewer components in the S-bomb compared to the firmware analysis, indicating that S-bombs generated at different stages of the build process (e.g., early code vs. post-build binary analysis) can vary significantly, potentially omitting critical software components.

Finally, the project underscored the complexity of S-bomb analysis and the nascent state of S-bomb maturity. Even when S-bombs were obtained, their quality often fell short of NTIA minimum requirements, lacking essential component and dependency data. The existence of multiple formats (CycloneDX, SPDX) and versions, coupled with subjective or erroneous software identifiers, created further hurdles for automated analysis. This necessitated significant "manual analysis" and scripted preprocessing before S-bombs could be effectively ingested into platforms like Finite State. The overarching finding was that successfully operationalizing S-bombs in OT is not merely a technical exercise but a complex socio-technical challenge requiring robust inventory practices, persistent vendor engagement, rigorous quality verification, and a collaborative "coalition of stakeholders" across organizational boundaries.

Technical Deep Dive

▶ Watch: Inventory Jungle outcome: 38 devices, 16 vendors identified (6:00)

The technical execution of S-bombing the substation began with a rigorous asset inventory process, far more detailed than traditional IT asset management. Leveraging existing OT Network visibility tools like Dragos, along with internal asset inventory databases, provided a foundational understanding. However, these tools proved insufficient for capturing the granular detail required for S-bombs, specifically software versions and firmware versions. This necessitated a manual, on-site "walkdown" of the substation. This physical inspection involved taking photographs of devices, noting model numbers, serial numbers, and any visible firmware information. Crucially, the team engaged with various internal business units responsible for different aspects of the substation—physical security, condition-based maintenance (CBM), and telecommunications—to piece together a complete picture. This meticulous effort ultimately identified 38 distinct devices across 16 different vendors, segmented across control, OT monitor, OT fault, physical security, cyber security, and telecommunications networks. The control network alone comprised 18 devices, underscoring the complexity of the environment.

Once the comprehensive inventory was established, the next technical challenge was S-bomb collection from the identified vendors. This process was far from automated, often requiring direct, persistent interaction. The speakers noted that the "S-bomb sharing is clunky," involving numerous emails and meetings (up to 12 per vendor) to navigate legal departments and internal processes. This highlights a significant technical-organizational interdependency: even with a clear technical requirement, the delivery mechanism is heavily reliant on human-centric, often bureaucratic, vendor engagement processes.

A critical technical phase involved S-bomb quality assessment and verification. The talk emphasized that simply receiving an S-bomb is not enough; its correctness and completeness must be validated. The team utilized Finite State as a platform for analysis, but also performed independent verification. A key example cited was the analysis of unencrypted firmware obtained from one vendor. By running this firmware through Finite State's binary analysis capabilities, they compared the derived component list against the vendor-provided S-bomb. This revealed significant discrepancies, with the S-bomb containing an average of 72 fewer components than identified by the independent firmware analysis. This finding underscores the importance of understanding the different points at which S-bombs can be generated—early in the code development, at build time, or post-build via binary analysis—and that each method can yield different levels of dependency coverage. The goal is to ensure all components are present, not just a subset. The speakers also noted that a Linux-based system's S-bomb could easily contain over 1800 different components, illustrating the immense scale of data involved and the "needles in a stack of needles" challenge for vulnerability management.

Finally, S-bomb analysis and operationalization presented its own set of technical hurdles. The two major S-bomb formats, CycloneDX and SPDX, while standardized, have multiple versions, and the quality of the data within the S-bomb is paramount. Issues such as incorrect software identifiers (e.g., Package URLs or CPEs) or typos could render automated analysis ineffective. The team performed "scripted analysis" to preprocess S-bombs, ensuring they contained the necessary component and dependency data before uploading them to the Finite State platform. They also utilized an "S-bomb quality score" (likely based on NTIA minimum requirements) to assess incoming S-bombs, finding that many "barely passing" or failed to meet these basic criteria. This highlights the nascent maturity of S-bomb generation practices among many vendors and the ongoing need for tooling and community-driven projects (like those from CISA) to improve data quality and enable effective, automated security insights. The technical journey was therefore a cyclical process of inventory, collection, rigorous verification, and iterative analysis, constantly pushing for higher data fidelity and operational utility.

Demo / Proof of Concept

▶ Watch: Level 2: Vendor Valley and clunky S-bomb sharing (7:00)

While the talk did not feature a live, interactive software demonstration in the traditional sense, the entire presentation served as a comprehensive proof of concept for the feasibility and challenges of operationalizing S-bombs in a real-world critical infrastructure setting. The "gameplay" narrative effectively demonstrated the practical steps, tools, and human interactions required to achieve the mission of S-bombing a substation.

The demonstration of concept began with the practical implementation of an enhanced asset inventory. This involved combining existing Dragos OT network visibility data with manual, on-site walkthroughs, photography, and cross-functional engagement within the asset owner's organization. This process, which went beyond typical asset discovery to pinpoint specific firmware versions, was a critical foundational "demo" of how to achieve the granular detail necessary for S-bomb correlation.

The subsequent "levels" of the game narrative then demonstrated the process of vendor engagement and S-bomb acquisition. This wasn't a technical demo of a tool, but rather a practical demonstration of the social engineering and organizational persistence required. The speakers detailed the iterative communication, the negotiation of legal hurdles, and the sheer volume of effort (up to 12 communications per S-bomb) needed to extract these crucial documents from 16 different vendors. This implicitly "demonstrated" that S-bomb collection is currently a highly manual, relationship-driven process, not a simple API call.

Furthermore, the project showcased the application of S-bomb analysis tools and techniques. While not a live UI demo, the speakers explicitly mentioned using the Finite State platform for both binary analysis of unencrypted firmware and for ingesting and analyzing collected S-bombs. They "demonstrated" the findings from this analysis, such as the discrepancy of 72 fewer components in vendor S-bombs compared to independent firmware analysis. This highlighted the practical utility of such platforms in verifying S-bomb quality and identifying missing dependencies. The use of "scripted analysis" for preprocessing S-bombs and assessing them against an "S-bomb quality score" further illustrated the practical, albeit challenging, steps required for effective S-bomb operationalization. In essence, the talk itself was a detailed case study, serving as a powerful proof of concept for the entire S-bomb lifecycle in a complex OT environment, demonstrating both the "how-to" and the "what-to-expect" for asset owners.

Defensive Implications

▶ Watch: Level 4: Firmware Forest - verifying S-bomb component accuracy (8:00)

The findings from S-bombing a substation carry profound defensive implications for critical infrastructure asset owners and cybersecurity professionals. The primary takeaway is the urgent need to move beyond theoretical discussions of S-bombs and actively pursue their practical implementation, acknowledging the significant challenges involved.

Firstly, asset owners must prioritize and significantly enhance their asset inventory processes. Traditional OT network visibility tools provide a good starting point (e.g., Dragos), but they are often insufficient for capturing the granular software and firmware version information essential for S-bomb utility. Defenders should implement rigorous physical walkdowns, engage cross-functional teams (e.g., physical security, CBM), and leverage photographic evidence to create a comprehensive inventory that includes precise versioning for every device. Without this detailed foundation, S-bombs cannot be accurately mapped or operationalized.

Secondly, proactive and persistent vendor engagement is critical. Defenders should establish clear communication channels with their OT vendors, setting expectations for S-bomb delivery and quality. This requires building relationships, understanding vendor internal processes, and being prepared for "clunky" and time-consuming interactions. The experience of 12 emails/meetings per S-bomb highlights that this is a significant resource commitment, but one that is necessary to gain transparency into the supply chain. Asset owners should also push for vendors to improve their S-bomb generation capabilities, emphasizing the importance of including all transitive dependencies.

Thirdly, S-bomb quality verification cannot be overlooked. Simply receiving an S-bomb is not enough. Defenders must develop or acquire capabilities to independently assess the completeness and accuracy of vendor-provided S-bombs. This could involve using binary analysis tools (like Finite State) to compare S-bombs against actual firmware or device images, looking for missing components (e.g., the 72 fewer components identified in the talk). Establishing an "S-bomb quality score" based on NTIA minimum requirements can provide a standardized metric for evaluating vendor submissions and driving improvements. This verification step is crucial to avoid a false sense of security based on incomplete data.

Fourthly, internal coalition building is essential. The project's success was heavily reliant on engaging diverse internal stakeholders, from engineering to legal to vendor risk management. Defenders need to champion S-bomb initiatives across their organization, educating different departments on the benefits and responsibilities. This collaborative approach helps overcome internal politics and streamlines the complex processes involved in inventory, vendor interaction, and data utilization.

Finally, leverage S-bomb analysis platforms for vulnerability management and risk assessment. Once high-quality S-bombs are acquired, they should be ingested into specialized platforms that can automatically correlate components with known vulnerabilities (CVEs), identify outdated software, and flag insecure dependencies. This moves beyond reactive vulnerability patching to a more proactive, supply-chain-aware security posture. The sheer volume of components in modern OT devices (e.g., 1800+ in a Linux system) necessitates automated analysis to make S-bombs actionable. By implementing these defensive strategies, asset owners can transform S-bombs from a regulatory checkbox into a powerful tool for enhancing the security and resilience of their critical OT infrastructure.

Key Takeaways

• Comprehensive Inventory is Foundational: Accurate and granular asset inventory, including precise software and firmware versions, is a non-negotiable prerequisite for effective S-bomb operationalization in OT environments.
• Vendor Engagement Requires Persistence: Obtaining S-bombs from OT vendors is currently a "clunky" and resource-intensive process, demanding significant human interaction, relationship building, and persistence to navigate legal and organizational hurdles (e.g., up to 12 emails/meetings per S-bomb).
• S-bomb Quality Varies Wildly: Vendor-provided S-bombs often lack completeness, particularly regarding transitive dependencies, and may not meet basic quality standards like NTIA minimum requirements, necessitating independent verification and analysis.
• Binary Analysis is Key for Verification: Independent analysis of device firmware (e.g., using platforms like Finite State) is crucial for validating the accuracy and completeness of S-bombs and uncovering missing components (e.g., the 72 fewer components observed).
• Internal Collaboration is Paramount: Successful S-bomb implementation in complex OT environments requires building a "coalition of stakeholders" across engineering, IT, legal, and risk management teams to overcome organizational barriers.
• S-bombs Enable Proactive Defense: Despite the challenges, operationalized S-bombs provide critical transparency into the software supply chain, enabling asset owners to move towards a more proactive and informed cybersecurity posture for their OT assets.

About the Speaker(s)

Matt Wyckhouse, the host and narrator of this S4x24 talk, guided the audience through the intricate journey of S-bombing a substation. His technical insights and expertise were evident throughout the presentation, particularly in discussions surrounding S-bomb generation processes, binary analysis, and the nuances of component identification. While his specific company affiliation wasn't explicitly stated in the transcript, the frequent mention and use of the Finite State platform strongly suggest his connection to that organization.

Alex, presented as the "player" in this "game," provided the firsthand account of the practical challenges and "real-life intricacies" of the S-bomb mission. His perspective, including lamenting the "daunting task" and discussing the involvement of "the people at Southern that were involved in getting the S-bombs," clearly indicates his role as an asset owner representative, likely from Southern Company, who undertook the mission of S-bombing their substation. Together, Matt and Alex offered a comprehensive view combining technical expertise with practical, operational experience.