Catch-22: Uncovering Compromised Hosts using SSH Public Keys

Catch-22: Uncovering Compromised Hosts using SSH Public Keys

Speakers: Cristian Munteanu

Conference: USENIX Security

YouTube: https://www.youtube.com/watch?v=vNAth4yOdJ4

Overview

In the realm of cybersecurity, the Secure Shell (SSH) protocol stands as a cornerstone for secure remote access and administration. However, its widespread adoption across over 40 million machines globally also makes it an attractive target for malicious actors seeking persistent access and control. This talk, "Catch-22: Uncovering Compromised Hosts using SSH Public Keys," presented by Cristian Munteanu, delves into an ingenious method for identifying compromised SSH servers by leveraging a subtle, yet critical, design feature of the SSH authentication handshake.

The research, a collaborative effort with Ana Felman, Tobias Fibik from Max Planck Institute for Informatics, and George MDKis from TU Delft, introduces a novel scanning technique that exploits the server's response behavior to unknown public keys. By testing for the presence of known malicious public keys—often installed by attackers for persistence—the team has successfully identified tens of thousands of compromised systems in the wild. This work is significant as it provides a proactive mechanism for defenders to detect and address compromised infrastructure, moving beyond traditional signature-based detection to a behavioral-based approach rooted in protocol design.

The implications of this research are far-reaching. It offers system administrators a powerful new signal for identifying compromised assets within their networks and provides a blueprint for security researchers and vendors to develop enhanced threat intelligence and detection tools. The talk not only highlights a clever technical exploit of protocol behavior but also underscores the importance of ethical scanning practices and collaborative efforts with organizations like BitDefender and Shadow Server to ensure responsible disclosure and remediation of identified compromises.

Background

▶ Watch: Introduction: Uncovering compromised hosts via SSH keys (0:00)

SSH, first introduced in 1995 as a secure alternative to protocols like Telnet, quickly became the de facto standard for establishing secure connections to remote machines. Its robust encryption and authentication mechanisms ensured confidentiality and integrity for command-line access, file transfers, and tunnel creation. The protocol's popularity exploded, leading to its deployment on an estimated 40 million servers worldwide, making it an indispensable tool for system administrators, developers, and cloud infrastructure management.

However, this ubiquity also positioned SSH as a prime target for attackers. Gaining access to an SSH server grants adversaries significant control over the compromised machine, enabling data exfiltration, lateral movement, resource hijacking, and establishing long-term persistence. A common tactic observed in honeypots and incident response investigations is for attackers, once initial access is gained (often via brute-force attacks or exploiting vulnerabilities), to install their own public keys into the ~/.ssh/authorized_keys file of a target user. This allows them to log in subsequently without needing a password, maintaining covert and persistent access.

The core insight that forms the foundation of this research stems from a specific design choice within the SSH public key authentication handshake. When a client attempts to authenticate using a public key, it sends a userauth_request message containing the username and a hash of the public key it wishes to use. Crucially, the SSH server's design dictates that it will only issue a challenge for that public key if the key is actually present in the authorized_keys file for the specified user. If the key is not found, the server will respond with an error message, indicating the key's absence. This behavior is an intentional security feature, designed to prevent Denial-of-Service (DoS) attacks where a malicious client could flood a server with numerous arbitrary public keys, forcing the server to expend computational resources generating challenges for non-existent keys. This specific behavior, first publicly noted in a 2017 blog post regarding GitHub username enumeration, was scaled up by the researchers to apply to all SSH servers, creating a powerful mechanism for identifying the presence of specific public keys on remote systems without completing a full authentication.

Key Findings

▶ Watch: Scanning tool design and 'canary key' mechanism (4:10)

The research yielded several significant findings, demonstrating the viability and impact of the proposed methodology for uncovering compromised SSH hosts.

Firstly, the core finding is the successful identification of compromised hosts in the wild by leveraging the distinct behavior of SSH servers during public key authentication. By sending a userauth_request with a known malicious public key, the researchers could accurately determine if that key was present on a remote server, signaling a potential compromise.

Secondly, the extensive global scanning efforts, conducted between April and August 2024, revealed a substantial number of compromised systems. The team identified more than 21,000 unique compromised SSH hosts across the internet. These compromises were distributed globally, with higher concentrations observed in countries possessing more robust IT infrastructure, such as China, the US, Germany, Netherlands, France, Brazil, Russia, Japan, and India. This broad distribution underscores the pervasive nature of SSH compromises.

Thirdly, the research highlighted the effectiveness of proactive threat intelligence sharing and remediation efforts. The project established a crucial collaboration with the Shadow Server Foundation, a non-profit organization dedicated to combating cybercrime. All identified compromised IPs were fed into Shadow Server's systems, enabling them to generate a "special report on compromised SSH hosts." This report allows system administrators to subscribe and receive notifications if any IP within their network is identified as compromised, facilitating timely remediation.

Finally, the practical impact of these notifications was empirically demonstrated through a case study involving a European hosting provider. An initial scan identified 152 hits (compromises) across both admin and root usernames associated with multiple malicious keys within this provider's network. Following the notification sent via Shadow Server, a subsequent scan showed a dramatic reduction to just 36 hits. This significant decrease validated the efficacy of the notification system in prompting system administrators to clean up compromised systems, proving the real-world value of this detection method.

Technical Deep Dive

▶ Watch: Tool validation: Lab testing across SSH versions (5:50)

The technical ingenuity of this research lies in its meticulous understanding and exploitation of the SSH protocol's design. The foundation is built upon the specific flow of the SSH public key authentication handshake.

When an SSH client initiates public key authentication, after the initial TCP handshake and key exchange, it sends a userauth_request message. This message contains two critical pieces of information: the username the client is attempting to log in as (e.g., root, admin) and the hash of the public key it intends to use for authentication. The server's response to this userauth_request is the lynchpin of the detection mechanism. If the server finds a matching public key for the specified user in its authorized_keys file, it will respond by sending a challenge to the client. This challenge is typically a nonce that the client must sign with its corresponding private key to prove ownership. Conversely, if the server does not find a matching public key for that user, it will return an error message without issuing a challenge. This intentional design prevents DoS attacks and is the "catch-22" that the researchers leveraged.

To operationalize this, the researchers developed a custom scanning tool. They utilized Zmap, a fast network scanner, to efficiently identify open SSH ports (22 and 222) across the internet. Building upon Zmap, they patched its client component, ZRAP 2, to function as a specialized SSH client. This modified ZRAP 2 client was programmed to send the userauth_request messages containing the target username and the hash of a public key. Crucially, the client was designed to stop the connection immediately upon receiving a server challenge, thus never performing a full authentication handshake. This minimizes resource consumption and prevents actual login attempts.

A critical component of their methodology is the canary key mechanism. Before testing any known malicious keys, the scanning tool first sends a newly generated, unique canary key to the target server. The purpose of this canary key is to identify misconfigured servers or honeypots. If a server accepts this canary key (i.e., sends a challenge for it), it indicates that the server is likely misconfigured to accept any public key or is a honeypot designed to capture such attempts. In such cases, the server's response to a malicious key would be ambiguous and not indicative of a compromise. Therefore, if the canary key receives a positive hit, that host is immediately excluded from further testing with malicious keys, ensuring the integrity and accuracy of the compromise detection.

The tool's robustness was validated through extensive lab testing against various popular SSH server implementations and versions. The researchers tested OpenSSH (versions 9.4 down to 2.1), Dropbear (0.84 down to 0.23), Bitvise SSH Server (versions 9 down to 6), and WolfSSH (version 1.4). The results confirmed that their tool worked without issues on all server versions deployed after 2005. Older versions, particularly those predating 2005, did not respond correctly, primarily due to incompatibilities with older SSL libraries and ciphers no longer supported by modern scanning tools.

Ethical considerations were paramount throughout the project. The researchers meticulously adhered to the Menlo Report guidelines for ethical security research. This included implementing rate limiting to prevent overwhelming target systems, maintaining a blocklist for administrators who wished to opt-out, providing 24/7 contact availability (phone and email), setting up a dedicated project webpage detailing their methodology, and using a suggestive reverse DNS entry to identify their scans. Furthermore, they secured IRB approval from their institution, underscoring their commitment to responsible research.

For their malicious key data, the team established a collaboration with BitDefender, a leading cybersecurity company. BitDefender generously shared a dataset of public keys known to be malicious, collected from their extensive network of honeypots and incident response investigations. These keys were confirmed to be associated with various malicious activities, including malware infections and ransomware deployments. The researchers focused on the top 52 most active malicious public keys from this dataset, noting that some keys (e.g., keys 1-10) showed prolonged activity over time. The scanning parameters included targeting ports 22 and 222, testing for common usernames like root, admin, and database (also derived from BitDefender's intelligence), and scanning both IPv4 and IPv6 address spaces, utilizing public IPv6 hit lists for the latter. The scans commenced on April 4, 2024, and were ongoing at the time of the presentation.

Demo / Proof of Concept

▶ Watch: Acquiring known malicious public keys from BitDefender (7:50)

While the talk did not feature a live, interactive demonstration of the scanning tool, the effectiveness of the approach was thoroughly validated through extensive real-world scanning and the presentation of empirical results. The "proof of concept" was the successful identification of a significant number of compromised hosts and the subsequent positive impact of notification efforts.

The primary demonstration of the tool's capability was the quantitative result of over 21,000 compromised hosts identified globally between April and August 2024. A detailed visualization showed the geographical distribution of these hits, highlighting regions with robust IT infrastructure such as China, the US, and European countries like Germany, the Netherlands, and France, alongside others like Brazil, Russia, Japan, and India. This global spread illustrated the pervasive nature of the threat the tool was designed to detect.

Further substantiating the proof of concept, the researchers detailed their collaboration with the Shadow Server Foundation. This partnership enabled the flow of identified compromised IP addresses to Shadow Server, which then created a specialized report for "compromised SSH hosts." This report serves as a critical notification mechanism, allowing system administrators to subscribe and receive alerts if any IP within their network is identified as containing a known malicious SSH public key.

The most compelling real-world demonstration of the system's efficacy was the case study of a European hosting provider. During the initial scan, 152 compromised hosts were detected within this provider's network, associated with various malicious keys attempting to access admin and root accounts. Following the notification sent by Shadow Server, a subsequent scan revealed a dramatic reduction in compromised hosts to just 36. This tangible decrease served as strong evidence that the notifications were acted upon, leading to the remediation of a substantial number of compromised systems. This real-world impact underscored the practical value of the research and the detection methodology.

Defensive Implications

▶ Watch: Experiment setup and significant findings (9:00)

The findings from this research provide crucial insights for defenders, enabling them to enhance their security posture against SSH-based compromises.

1. Subscribe to Threat Intelligence Feeds: System administrators should actively subscribe to and monitor specialized reports from organizations like the Shadow Server Foundation. The "special report on compromised SSH hosts" is a direct outcome of this research and offers an invaluable, actionable intelligence feed for detecting compromised systems within their networks.

1. Regularly Audit authorized_keys Files: It is paramount for administrators to regularly review the contents of ~/.ssh/authorized_keys files for all users, especially privileged accounts like root and admin. Any unfamiliar or unauthorized public keys should be immediately investigated and removed. Automated tools for auditing SSH key usage and ensuring adherence to key management policies should be implemented.

1. Implement Strong SSH Key Management: Beyond auditing, organizations should establish robust policies for SSH key lifecycle management. This includes regular key rotation, strong passphrase protection for private keys, and restricted access to authorized_keys files. Consider using centralized SSH key management solutions for large environments.

1. Monitor for Anomalous SSH Activity: Deploy Intrusion Detection/Prevention Systems (IDPS) and Security Information and Event Management (SIEM) solutions to monitor SSH login attempts. Look for patterns such as failed login attempts, logins from unusual geographical locations, or successful logins using keys that were not explicitly provisioned. While this research identifies installed malicious keys, monitoring can catch the initial brute-force or exploitation attempts that precede key installation.

1. Understand the "Canary Key" Concept: Defenders should be aware of the implications of their SSH server configurations. If an SSH server responds positively (sends a challenge) to any arbitrary public key, it indicates a severe misconfiguration. Such servers are not only vulnerable to DoS attacks but also cannot be reliably assessed for compromise using this method. Administrators should ensure their SSH daemon (e.g., sshd_config) is correctly configured to only respond to known, legitimate keys.

1. Disallow Password Authentication: Where feasible, disable password-based SSH authentication entirely and rely solely on SSH keys. This significantly reduces the attack surface for brute-force attacks, forcing attackers to compromise private keys or exploit other vulnerabilities.

1. Keep SSH Software Updated: Ensure all SSH server implementations (OpenSSH, Dropbear, Bitvise, WolfSSH, etc.) are kept up-to-date with the latest security patches. The research noted that older versions might not respond predictably, but newer versions are critical for overall security.

1. Integrate Detection Techniques: Security vendors and researchers can integrate this technique into their own threat hunting tools and security products. By maintaining databases of known malicious SSH public keys, they can proactively scan their customer bases or the internet for compromises, offering a new layer of protection.

Key Takeaways

• The SSH protocol's design, specifically how servers handle public key authentication requests, can be cleverly leveraged to detect compromised hosts.
• SSH servers only issue a challenge for a public key if that key is actually present for the specified user, making it possible to remotely infer the presence of a key.
• A custom scanning tool, built on Zmap and ZRAP 2, can efficiently identify compromised SSH servers by sending userauth_request messages with known malicious public keys.
• The canary key mechanism is crucial for filtering out misconfigured servers and honeypots, ensuring the accuracy of compromise detection.
• Extensive global scans identified over 21,000 compromised SSH hosts between April and August 2024, demonstrating the widespread nature of this threat.
• Collaboration with threat intelligence providers (BitDefender) for malicious key data and notification services (Shadow Server Foundation) for remediation is vital for the real-world impact of such research.
• System administrators should proactively audit authorized_keys files, subscribe to relevant threat intelligence, and ensure proper SSH server configurations to defend against such compromises.

About the Speaker(s)

Cristian Munteanu is the primary speaker for this presentation, detailing the collaborative research effort. He is associated with the Max Planck Institute for Informatics, where this work was conducted. The research is a joint effort, also involving Ana Felman and Tobias Fibik from the Max Planck Institute for Informatics, and George MDKis from TU Delft. Their collective work focuses on network security and the development of novel techniques to identify and mitigate cyber threats, particularly those leveraging widely deployed protocols like SSH. Their expertise lies in understanding protocol specifics and translating them into practical security assessment tools and defensive strategies.