Anatomy of Smart Building Ransomware Attacks

Anatomy of Smart Building Ransomware Attacks

Speakers: Anthony Forde

Conference: S4

YouTube: https://www.youtube.com/watch?v=ovghcbt8oBI

Overview

Anthony Forde's talk, "Anatomy of Smart Building Ransomware Attacks," delivers a sobering deep dive into a real-world ransomware incident that crippled a healthcare facility. The presentation meticulously details how a seemingly isolated IT problem escalated to impact critical operational technology (OT) systems within a smart building, highlighting the severe consequences of IT/OT convergence vulnerabilities. Forde, an incident response expert, walks through the complex investigation, the numerous challenges encountered, and the ultimate recovery efforts, providing invaluable lessons for organizations managing converged environments.

This talk is crucial for anyone involved in cybersecurity, particularly those responsible for critical infrastructure, healthcare, or smart building management. It underscores the evolving threat landscape where ransomware actors increasingly target OT for maximum disruption and leverage the interconnectedness of modern facilities. Forde's narrative serves as a stark reminder of the importance of robust incident response planning, vendor security oversight, and comprehensive visibility across both IT and OT domains to mitigate the growing risks posed by sophisticated cyber adversaries.

Background

▶ Watch: Initial ransomware attack on healthcare facility (0:00)

The convergence of information technology (IT) and operational technology (OT) has been a double-edged sword for many organizations, particularly in critical sectors like healthcare. While it offers efficiencies and enhanced capabilities for smart buildings and modern facilities, it also introduces a vast new attack surface. Traditionally, OT environments were air-gapped or isolated, relying on physical security and obscurity for protection. However, the drive for remote management, data analytics, and integrated systems has led to increased connectivity, often blurring the lines between IT and OT networks. This interconnectedness means that a compromise in one domain can rapidly propagate to the other, leading to severe operational disruptions, safety concerns, and potential loss of life.

The specific incident discussed by Forde exemplifies these challenges. A healthcare facility, leasing its smart building, found itself in a precarious position where property management was responsible for OT systems, while the healthcare provider managed IT. This split responsibility often leads to fragmented security practices, lack of unified visibility, and significant coordination hurdles during an incident. Many OT environments, especially those managed by third-party vendors, suffer from a lack of basic cybersecurity hygiene: outdated legacy systems, absence of Endpoint Detection and Response (EDR) or antivirus (AV) solutions, poor logging, and inadequate asset inventories. These vulnerabilities make OT systems attractive targets for ransomware groups like Vice Society, known for targeting healthcare organizations due to their critical nature and high likelihood of paying ransoms to restore essential services. The incident also highlights common pitfalls in incident response, such as inadequate backup strategies, vendor resistance to security assessments, and the inadvertent destruction of critical forensic evidence by security tools.

Key Findings

▶ Watch: SIM vendor suffering its own ransomware attack (4:00)

The incident began at 4:00 a.m. when a healthcare facility experienced widespread IT operational issues, including patient portal outages and failing legacy healthcare equipment. The IT team quickly discovered their electronic healthcare records database servers were encrypted, with ransom notes attributing the attack to Vice Society ransomware. Within 24 hours, ambulances were diverted, and patients redirected. A third-party incident response (IR) team was brought in within 48 hours.

The initial containment efforts included disabling internet access, remote access, and performing two rounds of full global password resets, including the Active Directory (AD) Kerberos account (KRBTGT), to invalidate any potential golden tickets. An EDR tool was deployed across IT endpoints and servers, replacing the facility's existing solution.

Key findings and challenges during the incident response included:

• OT System Involvement: Property management, responsible for the building's OT systems, became concerned. The speaker's team was brought in to assess if OT systems were compromised and to provide guidance on threat actor negotiations.
• Vendor Resistance: OT system vendors pushed back on IR teams running triage scripts or taking systems offline, citing warranty concerns and a lack of environmental visibility (no asset inventories, network topologies, or cybersecurity tooling).
• Logging Failures: The shared internet connection between IT and OT systems relied on a single firewall managed by the healthcare facility. While the firewall pushed logs to a Security Information and Event Management (SIM) system with a year's worth of data, the SIM provider itself was undergoing its own ransomware incident, making the logs inaccessible.
• Backup Catastrophe: The healthcare facility's on-premises backups had been wiped by the threat actor. There were no offsite copies, and the backup authentication mechanism was tied to Active Directory, which the adversary controlled.
• EDR Malfunction: Crucially, the newly deployed EDR tool, in its aggressive malware detection mode, flagged and deleted all encrypted files and ransom notes at the beginning of the incident. This critical error cost the facility two weeks to recover the deleted data using specialized forensic tools, leading to increased ransom demands and agitation from the threat actor.
• Data Exfiltration: Forensic analysis revealed 4 terabytes (TB) of data had been exfiltrated from the IT side of the house. This data was sent to an IP address in the Netherlands over TCP port 445 (SMB), indicating the threat actor mapped a share to a public-facing malicious server. The exfiltration originated from a legacy Windows server lacking EDR or AV.
• Successful Decryption: Despite the setbacks, threat actor negotiations and decryption were successful. While a 75% success rate is typical for decryptors, in this scenario, two rounds of the decryptor fully restored systems.
• Effective Recovery Strategy: A three-stage VLAN approach was used: leaving the compromised network isolated, re-imaging and securing systems on a staging VLAN, and then migrating them to a clean network. This, combined with EDR and network-level visibility, allowed the team to systematically clean and restore operations over seven weeks.
• Root Cause: OT System Compromise: The investigation, heavily relying on NetFlow logs, traced the initial malicious activity to an HVAC server within the OT environment. This server was found port scanning the environment. The vendor eventually admitted that one of their engineers fell victim to a phishing email attack, leading to the compromise of their email account. This account contained a spreadsheet of credentials for TeamViewer, a remote access tool, which the threat actor then used to access the HVAC server.
• Attacker TTPs: Once on the HVAC server, the adversary used legitimate system administration tools for discovery and execution. They leveraged AD Recon and NLTest to map the domain and used WMI and PsExec to stage and execute malware and ransomware remotely. Interestingly, the threat actors used cartoon movie references (e.g., Toy Story) for account names and malware, providing a potential clue to their age or operational style.

Technical Deep Dive

▶ Watch: First IR team's EDR deleted all ransom notes (5:50)

The attack vector highlights a critical vulnerability at the IT/OT intersection: a seemingly innocuous OT component serving as the initial entry point for a widespread ransomware campaign. The HVAC server, a legacy Windows system without EDR or AV, became the beachhead. This initial access was gained not through a direct exploit of the HVAC system itself, but through a human element – a phishing attack against a vendor engineer. The compromise of the engineer's email account yielded TeamViewer credentials, which provided the adversary with remote access to the vulnerable HVAC server. This underscores the importance of securing remote access pathways and the human factor in the attack chain.

Once inside the OT network via the HVAC server, the threat actor demonstrated a sophisticated understanding of network reconnaissance and lateral movement using readily available, legitimate tools. The HVAC server was observed port scanning the environment, indicating the adversary's efforts to map the internal network and identify other potential targets. For discovery within the Active Directory domain, the attackers utilized tools like AD Recon and NLTest. These tools are commonly used by legitimate system administrators to gather information about users, groups, computers, and trust relationships within an AD environment, but in the hands of an attacker, they provide a blueprint for further compromise.

For execution of malware and ransomware, the adversaries leveraged Windows Management Instrumentation (WMI) and PsExec. Both are powerful Windows command-line tools that allow for remote command execution and service deployment. By using these legitimate system administration tools, the attackers could blend their malicious activity with normal network traffic, making detection more challenging. They staged their binaries on various servers and then executed them remotely, demonstrating a methodical approach to deploying their payload across the network. The choice of cartoon movie references for account names and malware further illustrates how attackers sometimes attempt to obfuscate their activities or simply reveal aspects of their operational persona.

The data exfiltration phase was significant, with 4TB of sensitive healthcare data stolen. This exfiltration occurred over TCP port 445 (SMB) to an IP address in the Netherlands. The forensics confirmed that the threat actor mapped a share from a compromised legacy Windows server within the healthcare facility to a public-facing malicious server. This method of exfiltration is common, leveraging a standard file-sharing protocol to move large volumes of data. The fact that the originating server lacked EDR or AV highlights the blind spots that often exist within an enterprise's asset inventory and security coverage, especially for older or specialized systems.

The incident response itself faced numerous technical hurdles. The shared internet connection and single firewall for both IT and OT, while seemingly efficient, created a single point of failure and a shared attack surface. The SIM provider's own ransomware incident rendered crucial firewall logs inaccessible, severely hampering the forensic investigation. The most critical technical misstep was the EDR tool's aggressive deletion of encrypted files and ransom notes. While EDRs are designed to remediate threats, the lack of a proper configuration or understanding of the incident context led to the destruction of vital evidence and the very data needed for recovery. This highlights a need for sophisticated EDR policies that can differentiate between malicious files and evidence, or at least quarantine rather than delete.

The recovery process, while protracted, demonstrated effective technical strategies. The three-stage VLAN approach (compromised, staging, clean) allowed for methodical cleaning and restoration, minimizing the risk of re-infection. NetFlow logs emerged as an indispensable tool, providing network flow data that allowed the IR team to baseline normal OT communication, identify the initial port scanning activity from the HVAC server, and trace the root cause. This network-level visibility was critical in an environment where host-level visibility (EDR/AV) was non-existent on OT assets.

Demo / Proof of Concept

▶ Watch: NetFlow logs provide critical OT network visibility (6:40)

This talk focused on a detailed post-mortem analysis of a real-world ransomware incident rather than demonstrating a specific proof of concept or live exploit. The speaker, Anthony Forde, presented a comprehensive case study of an attack and the subsequent incident response efforts.

Defensive Implications

▶ Watch: Discovery of 4TB data exfiltration via SMB (7:00)

The "Anatomy of Smart Building Ransomware Attacks" talk provides a wealth of defensive implications for organizations grappling with IT/OT convergence and the persistent threat of ransomware.

1. Unified IT/OT Security Strategy: The incident starkly highlights the dangers of disparate IT and OT security management. Organizations must adopt a holistic security strategy that accounts for the interconnectedness of these environments. This includes shared risk assessments, incident response planning, and consistent security policies. The single firewall serving both IT and OT in this case created a critical common point of failure.
2. Robust Vendor Risk Management: OT vendors often have privileged access to critical systems and can introduce significant risk. Organizations must implement rigorous vendor security programs that include:

• Contractual Obligations: Mandate security requirements, including adherence to cybersecurity standards, provision of asset inventories, network topologies, and access to logs for auditing and incident response.
• Security Assessments: Conduct regular security assessments of vendor practices, especially concerning remote access tools like TeamViewer and credential management.
• Phishing Training: Ensure vendor personnel with access to client systems also undergo comprehensive phishing awareness training.

1. Comprehensive OT Visibility and Control: The lack of visibility into OT systems was a major impediment. Defenders must:

• Asset Inventory: Maintain a precise and up-to-date inventory of all OT assets, including their operating systems, patch status, and security tooling compatibility.
• Network Segmentation: Implement strong network segmentation between IT and OT networks, and within OT networks themselves, to limit lateral movement.
• Logging: Deploy robust logging solutions across the OT environment, including NetFlow logs and firewall logs, to establish baselines of normal behavior and detect anomalies. Ensure logs are centralized, immutable, and accessible, ideally off-premises or to a highly secured SIM/SIEM.
• Endpoint Security: Where compatible, deploy EDR/AV solutions on OT endpoints. For legacy systems that cannot support modern EDR, implement compensating controls such as strict network segmentation, application whitelisting, and enhanced monitoring.

1. Strengthen Incident Response Capabilities:

• Clear Roles and Responsibilities: Establish clear incident response roles and communication protocols between IT, OT, property management, and third-party vendors before an incident occurs.
• Backup Strategy Reinforcement: Implement a 3-2-1 backup strategy (three copies of data, on two different media, with one copy off-site and isolated). Ensure backups are immutable, regularly tested, and their authentication mechanisms are entirely separate from the primary Active Directory domain to prevent their compromise during a domain-wide attack.
• EDR Deployment Best Practices: Configure EDR tools carefully during an incident to quarantine or isolate rather than delete potentially critical evidence like encrypted files and ransom notes. Forensic preservation should be prioritized.
• Negotiation Strategy: Develop a pre-defined strategy for ransomware negotiations, including communication protocols, cryptocurrency procurement, and decryptor testing procedures.
• Isolation and Recovery: Plan for a phased recovery using segmented environments (e.g., the three-stage VLAN approach) to systematically clean and restore systems, minimizing the risk of re-infection.

1. Address Legacy Systems: The reliance on legacy Windows servers without EDR/AV was a critical vulnerability for both the HVAC server and the data exfiltration point. Organizations must:

• Isolate and Harden: Isolate legacy systems through network segmentation.
• Compensating Controls: Implement compensating controls such as application whitelisting, strict firewall rules, and enhanced network monitoring.
• Modernization Plans: Develop and execute plans for upgrading or replacing vulnerable legacy systems that cannot be adequately secured.

1. Enhance User Awareness and Credential Security: The initial compromise stemmed from a phishing attack. Continuous and targeted phishing awareness training for all employees, especially those with access to critical systems or vendor portals, is non-negotiable. Implement Multi-Factor Authentication (MFA) for all remote access, email, and critical system logins, including those used by third-party vendors. Strong password policies and regular password rotations, particularly for service accounts and privileged users, are essential.

By proactively addressing these defensive implications, organizations can significantly reduce their attack surface, improve their detection capabilities, and enhance their resilience against sophisticated ransomware attacks targeting converged IT/OT environments.

Key Takeaways

• OT Systems are Critical Entry Points: Ransomware actors increasingly target vulnerable operational technology (OT) systems, such as HVAC servers, as initial access vectors to then pivot into broader IT networks for maximum impact.
• Complexities of IT/OT Convergence: Shared infrastructure, fragmented responsibilities (e.g., property management vs. healthcare provider), and vendor resistance to security audits create significant challenges for unified incident response and overall cybersecurity posture.
• Visibility is Paramount: Comprehensive network visibility, particularly through NetFlow logs, proved indispensable for identifying the root cause, understanding lateral movement, and baselining OT communications when host-level visibility (EDR/AV) was absent.
• Incident Response Pitfalls: Common IR mistakes, such as EDR tools deleting critical evidence (ransom notes, encrypted files) and inadequate backup strategies (on-prem, AD-tied, no offsite), can severely prolong recovery and escalate costs.
• Basic Security Hygiene Prevents Advanced Attacks: The incident began with a simple phishing attack against a vendor engineer. Robust phishing training, secure remote access controls (e.g., TeamViewer with MFA), and patching/securing legacy systems are fundamental defenses.
• Methodical Recovery is Possible: Despite significant challenges, a structured recovery approach utilizing three-stage VLANs and systematic cleaning can successfully restore operations, though it requires substantial time and resources.

About the Speaker(s)

Anthony Forde is an experienced incident response professional, specializing in complex cyber incidents that often involve critical infrastructure and operational technology (OT) environments. His expertise is evident in his detailed recounting of the ransomware attack, demonstrating a deep understanding of forensic investigation, threat actor tactics, and recovery strategies. Forde's role in overseeing the engagement and partnering with other IR teams highlights his leadership in navigating multi-party incident responses. He possesses practical experience in threat actor negotiations, data exfiltration analysis, and the implementation of robust recovery architectures like the three-stage VLAN approach. His insights are particularly valuable for organizations grappling with the unique challenges of securing converged IT/OT systems.