The Black Hat Asia Network Operations Center (NOC) Report

The Black Hat Asia Network Operations Center (NOC) Report

Speakers: Jessica (Cisco), Jonathan (Arista), Jimmy (Palo Alto Networks), Mark (Corlite)

Conference: Black Hat Asia

YouTube: https://www.youtube.com/watch?v=nCe5I38nUUk

Overview

The Black Hat Asia Network Operations Center (NOC) Report offers a unique glimpse into the intricate and often paradoxical challenge of securing one of the world's premier cybersecurity conferences. Far from a typical enterprise environment, the Black Hat network is a dynamic ecosystem where offensive security techniques, malware demonstrations, and exploit development are not just tolerated but actively encouraged as part of the learning experience. This talk, delivered by a collaborative team from Cisco, Arista, Palo Alto Networks, and Corlite, details how these industry leaders come together, setting aside corporate logos, to build, secure, and monitor an infrastructure that must simultaneously facilitate cutting-edge research and protect critical attendee data.

The presentation provides a comprehensive breakdown of the architectural components, security philosophies, and operational challenges inherent in running a network designed for "needles in needle stacks" – identifying genuinely malicious activity amidst a sea of intentional, yet benign, "bad" traffic. The speakers, each representing a crucial facet of the NOC's operation, emphasize the paramount importance of multi-vendor collaboration, continuous innovation, and adaptable security strategies. This isn't just a report; it's a testament to the real-world application of advanced security technologies and the human ingenuity required to defend a truly adversarial environment, offering invaluable lessons for security professionals across all sectors.

Background

▶ Watch: Introduction to the collaborative NOC team (0:00)

The Black Hat Network Operations Center boasts a rich history, having been a cornerstone of the conference for over two decades. Jessica, a Cisco representative and a veteran of the Black Hat NOC for ten years, highlighted this long-standing commitment. The NOC's evolution reflects the ever-changing landscape of cybersecurity, adapting from initial rudimentary setups to the sophisticated, multi-vendor architecture seen today. Early iterations involved basic malware analysis, then expanded to encompass DNS services, mobile device management (MDM) for iOS devices, and even the full network build during periods of supply chain crunch, before transitioning the core networking responsibilities to Arista. More recently, the focus has broadened to include robust identity management and physical security sensors to monitor environmental conditions within critical server rooms.

The fundamental problem the Black Hat NOC addresses is unlike that of a standard corporate network. As Jimmy from Palo Alto Networks eloquently put it, they are looking for "needles in needle stacks, not needles in haystacks." This unique threat model stems from the conference's purpose: to educate attendees on offensive security, allowing them to work with malware, exploits, and vulnerabilities in a controlled environment. Consequently, a traditional "block everything bad" firewall approach is largely impractical and would disrupt the core activities of the conference. The NOC must, therefore, allow a significant amount of potentially malicious traffic to flow, while simultaneously identifying and mitigating truly unauthorized or harmful actions. This complex balance necessitates sophisticated detection capabilities, deep visibility, and seamless collaboration between diverse security technologies and expert teams. The only exception to this permissive stance is the registration area, which handles sensitive personal information and is subjected to rigorous protection and traffic decryption.

Key Findings

▶ Watch: Arista's wired/wireless infrastructure and traffic tapping (0:50)

The Black Hat NOC's operations reveal several critical findings that underscore the complexities and unique demands of securing such a dynamic environment:

1. Unprecedented Multi-Vendor Collaboration: The NOC is a prime example of successful multi-vendor integration. Cisco, Arista, Palo Alto Networks, and Corlite, among others, contribute their specialized tools and expertise, operating as a unified team rather than competing entities. This collaborative spirit, where "logos are left at the door," is fundamental to addressing the multifaceted security challenges.
2. "Needles in Needle Stacks" Security Philosophy: The Black Hat environment is inherently adversarial. The core challenge is not merely blocking known threats but distinguishing legitimate educational activities (e.g., exploit demos, malware analysis in training) from genuinely malicious or unauthorized attacks targeting the conference infrastructure or attendees. This requires highly contextual threat hunting and advanced detection capabilities, moving beyond simple signature matching.
3. Black Hat as a Live Innovation Lab: The NOC serves as a real-world testing ground for cutting-edge security products and integrations. New features, automation workflows, and architectural designs are deployed, tested, and refined within the intense, high-stakes environment of the conference. Successful innovations often transition into production for joint customers outside of Black Hat. An example given was the testing of distributed denial of service (DDoS) solutions.
4. Prioritized Protection for Critical Assets: While most of the network operates under a high-visibility, lower-blocking paradigm, specific critical assets receive stringent protection. The registration area, containing attendees' personal information, is heavily secured with deep packet inspection and traffic decryption to prevent data leakage or exploitation.
5. Comprehensive Observability is Paramount: Given the unique threat model, pervasive visibility across the entire network is non-negotiable. This includes tapping all traffic, extensive logging, endpoint monitoring on critical servers, and advanced network performance monitoring tools like ThousandEyes to diagnose latency and connectivity issues across distributed architectures.
6. Continuous Evolution and Adaptation: The NOC's services and technologies are constantly evolving. From initial malware analysis to incorporating DNS security, mobile device management, identity services (Single Sign-On), physical sensors, and advanced XDR/SOAR platforms, the team continuously adapts to new threats and technological advancements, often driven by specific incidents or emerging needs.

Technical Deep Dive

▶ Watch: Palo Alto's network security and operations platform (2:26)

The Black Hat NOC's architecture is a sophisticated interplay of hardware, software, and services from multiple vendors, meticulously integrated to provide comprehensive network and security operations.

Arista's Core Networking Infrastructure:

Arista serves as the foundational wired and wireless infrastructure provider. For Black Hat Asia and London, Arista leverages the venue's existing infrastructure, plugging their Access Points (APs) into the venue's cabling. In this specific show, Arista deployed approximately 48 APs and 11 switches throughout the venue. A key operational aspect is zero-touch provisioning, where new devices automatically connect to CloudVision (CVQ), Arista's wireless infrastructure management platform, to pull down their configurations. Crucially, Arista also provides a network tap of all traffic, aggregating this data and forwarding it to the Palo Alto Networks and Corlite security tools for deep inspection.

Palo Alto Networks for Network Security and Operations:

Palo Alto Networks contributes two main pillars: network security via their Next-Generation Firewall (NGFW) and security operations through Cortex XSOAR (referred to as Cortex XIM Security Operations Platform in the talk, later clarified to XSOAR).

From a network security perspective, the NGFW's deployment is highly contextualized. In the registration area, where sensitive personal information and a database reside, heavy protection and inspection are applied, including SSL/TLS decryption to prevent data exfiltration or attacks on attendee data. For the rest of the conference network (training classes, arsenal, briefings), the NGFW largely allows traffic, but critically tracks it, alerts on suspicious activities, and provides deep visibility. This allows attendees to perform offensive security exercises without interruption while still enabling the NOC to monitor for genuinely malicious behavior.

Cortex XSOAR acts as the central Security Operations Platform, ingesting logs, alerts, and feeds from all partner technologies. This centralized platform facilitates investigation, detection, and response, allowing the distributed team of threat hunters from various vendors to collaborate on identifying true threats among the "needle stack" of expected adversarial traffic.

Corlite for Network Detection and Response (NDR):

Corlite functions as the open NDR provider. They deploy sensors that consume the packet stream provided by Arista's tap, capturing all traffic entering, leaving, and moving between segmented networks within Black Hat. Corlite's platform performs several critical functions:

• Packet Capture and Logging: All traffic is recorded and transformed into logs, which are then forwarded to various security platforms, including Palo Alto Networks Cortex XSOAR, Cisco XDR, and Corlite's own Investigator platform. This provides rich data for threat hunting and incident response.
• Intrusion Detection: Corlite utilizes both signature-based alerting and machine learning to detect anomalies and known threats within the network traffic.
• Threat Hunting: Corlite's own threat hunters work alongside those from other partners, analyzing logs and packet captures to identify suspicious activities and collaborate on determining their true nature (e.g., legitimate classroom activity vs. actual attack). An anecdote shared was Corlite identifying plain-text credentials from an engineer's setup, demonstrating proactive threat identification.

Cisco's Diverse Security Contributions:

Cisco's involvement has evolved significantly over the years, covering a broad spectrum of security domains:

• Identity Management: Cisco provides Single Sign-On (SSO) for securing the underlying infrastructure, ensuring controlled access to critical systems handling personal information.
• Observability and Performance Monitoring: ThousandEyes is extensively deployed across the network, including on Arista switches, iOS devices, and even Raspberry Pis with antennas, to provide end-to-end visibility into network performance and latency. This was critical in diagnosing a specific issue where a sales office experienced latency due to traffic routing from Singapore to Marseilles, France, to Paris, to Ireland, and then to AWS, resulting in over half a second of delay.
• Zero Trust and DNS Security: Cisco's Umbrella product has evolved into Secure Access, providing a Zero Trust architecture integrated with DNS security to protect users and devices regardless of location.
• Threat Intelligence and Analysis: Cisco leverages Threat Grid and Splunk Attack Analyzer for advanced malware analysis and threat intelligence.
• Extended Detection and Response (XDR) & Automation: Cisco's XDR product (integrating with Splunk for visualization) and XDR Automate (formerly SecureX Orchestrator) are used to automate integrations and streamline security operations, working in conjunction with Palo Alto's XSOAR.
• Cloud Security: Cisco also contributes to securing cloud-based components of the critical infrastructure, such as applications managing attendee badges.

Integration and Collaboration:

The synergy between these platforms is paramount. Arista's network taps feed Corlite's NDR sensors and Palo Alto's firewalls. Corlite's rich logs and alerts are then forwarded to Palo Alto's Cortex XSOAR and Cisco's XDR. Splunk is utilized as a general visualization platform, while Cortex XSOAR serves as the official Security Information and Event Management (SIEM) and Security Orchestration, Automation, and Response (SOAR) platform, handling the centralized investigation and response workflows. This multi-layered, integrated approach allows the NOC to achieve a level of visibility and control necessary for the Black Hat environment.

Demo / Proof of Concept

▶ Watch: Black Hat's unique firewall strategy and registration protection (3:30)

While the talk did not feature a live, explicit "demo" in the traditional sense, the entire Black Hat Network Operations Center functions as a continuous, live proof of concept for multi-vendor security integration and real-world threat hunting. The speakers highlighted several instances that serve as practical demonstrations of their capabilities:

One notable example involved Corlite's NDR sensors immediately identifying plain-text credentials being transmitted by an engineer during the setup phase. This incident, caught within a couple of days of deployment, showcased the proactive detection capabilities of the Corlite platform and the collaborative spirit of the NOC team, who promptly alerted the engineer to the security misconfiguration. This highlights the value of continuous monitoring and the "trust but verify" approach within the NOC.

Another powerful demonstration of the integrated system's value was the use of ThousandEyes to diagnose a critical latency issue. When the sales office struggled to load floor diagrams for the next conference, blaming the network, ThousandEyes was deployed. It quickly pinpointed the problem: network traffic was making an incredibly circuitous route from Singapore to Marseilles, France, then to Paris, Ireland, and finally to AWS, causing a significant half-second latency. This real-time diagnostic capability, deployed on Arista switches, iOS devices, and even Raspberry Pis, proved invaluable in understanding and addressing complex connectivity challenges.

Furthermore, the NOC actively experiments with new technologies. Jessica mentioned that Cisco is currently working on distributed denial of service (DDoS) solutions and demonstrating their effectiveness within the Black Hat environment, effectively using the conference as a "proof of value" for emerging security capabilities. The continuous evolution of the NOC, incorporating identity services, physical sensors for server room temperature (especially critical in cold London where servers can overheat), and advanced XDR/SOAR integrations, all represent ongoing, live proofs of concept for advanced security architectures. The upcoming briefing by Barton Grifter was also teased to include a time-lapse video demonstrating the rapid network build process, further illustrating the operational agility.

Defensive Implications

▶ Watch: Corelight's network detection and response capabilities (5:45)

The Black Hat NOC's unique operational model and technical architecture offer several crucial defensive implications for organizations of all sizes, extending beyond the conference environment itself:

1. Embrace Multi-Vendor Collaboration: No single vendor can provide a complete security solution for complex environments. The NOC's success hinges on seamlessly integrating best-of-breed products from Arista, Palo Alto Networks, Corlite, Cisco, and others. Defenders should prioritize interoperability and cultivate strong partnerships with their security vendors, viewing them as partners rather than mere suppliers.
2. Context-Aware Security Policies are Essential: A "one-size-fits-all" security posture is often ineffective. The Black Hat NOC demonstrates the necessity of tailoring security policies to the specific context and risk profile of different network segments. While the registration area demands stringent protection and decryption, other areas require a more permissive, yet highly visible, approach to facilitate intended activities. Understanding the nature of traffic and user intent is paramount.
3. Comprehensive Network Visibility is Non-Negotiable: The ability to tap all network traffic, leverage NDR solutions, and employ advanced observability tools like ThousandEyes is critical for detecting subtle anomalies and diagnosing complex issues. Defenders must invest in technologies that provide deep, pervasive visibility across their entire infrastructure, including wired, wireless, cloud, and remote endpoints.
4. Prioritize Identity and Access Management: Securing access to critical infrastructure through Single Sign-On (SSO) and robust identity management is a foundational security control. As the Black Hat NOC evolved to incorporate identity, it highlighted the importance of controlling who can access what, especially in environments handling sensitive data.
5. Automate and Orchestrate Security Operations: The integration of Cortex XSOAR and Cisco XDR Automate underscores the need for automation in security operations. SOAR platforms enable faster detection, investigation, and response by centralizing data, automating repetitive tasks, and orchestrating actions across disparate security tools. This is vital for managing the high volume of alerts and maintaining efficiency.
6. Treat Your Environment as a Continuous Innovation Lab: The Black Hat NOC constantly experiments with new technologies and integrations. Defenders should foster a culture of continuous improvement, regularly evaluating new security solutions, testing their efficacy in their own environments, and adapting their architectures to counter evolving threats. What works today may not work tomorrow.
7. Don't Forget Physical Security: While often overlooked in digital security discussions, the NOC's deployment of physical sensors (e.g., for temperature in server rooms) serves as a reminder that physical security is an integral part of an overall defense strategy. Protecting the physical infrastructure housing critical systems is as important as securing the data flowing through them.
8. Proactive Threat Hunting is Key: Relying solely on automated alerts is insufficient. The presence of dedicated threat hunters from each partner organization, collaboratively sifting through logs and packet captures, emphasizes the value of proactive, human-driven threat hunting to uncover sophisticated or novel attacks that might bypass automated defenses.

Key Takeaways

• The Black Hat NOC exemplifies unparalleled multi-vendor collaboration, with Cisco, Arista, Palo Alto Networks, and Corlite working as a unified team to secure a uniquely challenging environment.
• Security at Black Hat operates on a "needles in needle stacks" philosophy, requiring sophisticated detection and contextual threat hunting to distinguish legitimate offensive security activities from truly malicious attacks.
• The NOC functions as a live innovation and testing ground, constantly deploying and refining cutting-edge security products and integrations, such as ThousandEyes for network observability and Cortex XSOAR for centralized operations.
• Comprehensive visibility (via network taps, NDR, endpoint agents, and performance monitoring) is paramount for understanding traffic patterns and identifying anomalies in a high-volume, potentially adversarial network.
• Context-aware security policies are critical, with stringent protection (including SSL/TLS decryption) applied to sensitive areas like registration, while maintaining visibility and allowing educational activities elsewhere.
• Automation and orchestration through platforms like Cortex XSOAR and Cisco XDR Automate are essential for efficient security operations, enabling faster detection, investigation, and response in a complex, dynamic environment.

About the Speaker(s)

The Black Hat Asia NOC Report was presented by a distinguished panel of experts, each representing a crucial component of the conference's security and network infrastructure:

• Jessica (Cisco): A veteran of the Black Hat NOC, Jessica has personally been involved for ten years. She has witnessed and driven Cisco's evolving role, starting with malware analysis and expanding into DNS services, mobile device management (MDM) for iOS devices, building the core network, and establishing robust identity management solutions. Her leadership has been instrumental in the NOC's continuous innovation and growth.
• Jonathan (Arista): As Arista's representative, Jonathan is responsible for the core wired and wireless infrastructure of the Black Hat network. His team deploys and manages the APs and switches, leveraging zero-touch provisioning and ensuring that all network traffic is tapped and forwarded to the security partners for analysis.
• Jimmy (Palo Alto Networks): Affectionately known as "Happy Jimmy" in the NOC, he leads Palo Alto Networks' contribution to network security and security operations. He oversees the deployment of their Next-Generation Firewalls, particularly the heavy protection in the registration area, and champions the use of Cortex XSOAR as the central platform for investigation, detection, and response, integrating data from all partners.
• Mark (Corlite): Mark represents Corlite, the open Network Detection and Response (NDR) provider for Black Hat. His team's expertise lies in deploying NDR sensors to consume all network packets, generating logs, performing intrusion detection (both signature-based and machine learning), and conducting proactive threat hunting in collaboration with other NOC teams.