Regex For Hackers

Regex For Hackers

Speakers: Adam 'BuildHackSecure' Langley (Hackinghub); Ben 'nahamsec' Sadeghipour (Hackinghub)

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=mYC-rQ-HZaw

Overview

In their DEF CON talk, "Regex For Hackers," Adam Langley and Ben Sadeghipour, co-founders of Hackinghub, delivered a rapid-fire, comprehensive session on the critical role of regular expressions (regex) in cybersecurity. The presentation, packed with 198 slides, aimed to demystify regex for a hacker audience, transforming it from an intimidating concept into a powerful tool for vulnerability discovery, reconnaissance, and defensive strategies. The speakers underscored that a solid understanding of regex is not merely a niche skill but a fundamental requirement, often serving as the root cause of many client-side and server-side vulnerabilities.

The talk meticulously broke down regex fundamentals, illustrating how common misconfigurations and oversights in regex patterns lead to severe security flaws such as Post Message vulnerabilities, Open Redirects, CORS misconfigurations, and Server-Side Request Forgery (SSRF). Beyond identifying weaknesses, Langley and Sadeghipour demonstrated regex's offensive utility in bug bounty hunting, showcasing its application in advanced OSINT techniques on GitHub for subdomain discovery, API enumeration, and content extraction. They also highlighted its defensive value in code review and secret detection within data dumps.

This session is particularly relevant for security professionals, bug bounty hunters, and developers seeking to deepen their understanding of how regex operates, how it can be exploited, and how to implement it securely. By providing both theoretical foundations and practical, real-world examples, the speakers equipped attendees with the knowledge to leverage regex for more effective hacking and more robust defense, ultimately fostering a generation of more skilled and aware cybersecurity practitioners.

Background

▶ Watch: Talk introduction, speakers, and HackingHub overview. (0:00)

The necessity for regular expressions arises from the inherent limitations of basic string matching functions found in most programming languages. While methods like starts with, ends with, or contains can perform rudimentary checks, they quickly become unwieldy and insufficient when dealing with complex, variable patterns. For instance, validating a range of IP addresses, ensuring an email ends with one of several top-level domains, or detecting credit card numbers with optional hyphens or spaces, would necessitate cumbersome if or switch statements without regex. This complexity is precisely where regex shines, offering a concise and powerful language for defining intricate search patterns.

At its core, regex can be conceptualized as a state machine. Each instruction within a regex pattern represents a state, and the engine attempts to transition between these states based on character matches. If a rule matches, it moves to the next; if it fails, it backtracks, trying to find an alternative match from an earlier state. This iterative matching process allows regex to identify, extract, and validate specific parts of text from strings with remarkable precision.

The prevalence of regex-related vulnerabilities stems primarily from developers' common misunderstandings and misapplications of its syntax. Often, security-critical validation logic relies on regex patterns that are either too permissive or incorrectly constructed. Key issues frequently observed include:

• Missing Anchors: Forgetting to use ^ (start of string) and $ (end of string) can allow attackers to append or prepend malicious data that bypasses the intended validation.
• Unescaped Wildcards: The period (.) character in regex is a wild card that matches any single character (except newline). Developers frequently forget to escape it as \. when they intend to match a literal dot, leading to broad bypasses, particularly in domain validation.
• Overly Broad Quantifiers: Using quantifiers like * (zero or more) without sufficient constraints can lead to unexpected matches, allowing attackers to inject arbitrary characters.
• Lack of Specificity: Using general character sets or ranges where more restrictive patterns are needed can create loopholes.

These common pitfalls, explored in detail throughout the talk, highlight why a deep understanding of regex is crucial for both identifying and preventing a wide array of web application vulnerabilities.

Key Findings

▶ Watch: Why basic string functions fall short of needs. (3:00)

The talk "Regex For Hackers" unveiled several key findings regarding the dual nature of regular expressions as both a potent weapon for attackers and an indispensable tool for defenders. The central tenet is that regex, when misunderstood or incorrectly implemented, becomes a significant source of vulnerabilities, but when mastered, it unlocks advanced capabilities for reconnaissance, vulnerability discovery, and robust security.

1. Regex as a Root Cause of Vulnerabilities:

The speakers demonstrated conclusively that seemingly minor errors in regex patterns are often the root cause of critical web application flaws. The most frequently cited mistakes include:

• Missing Anchors (^ and $): Failing to anchor a regex to the beginning and end of a string allows for partial matches that can be easily bypassed. For instance, trusted.com without $ could match trusted.com.attacker.com.
• Unescaped Period Wildcard (.): Treating the . as a literal dot instead of escaping it as \. is a pervasive error. This wildcard matches any character, enabling attackers to substitute the intended dot with any other character (e.g., wwwxtrusted.com instead of www.trusted.com).
• Overly Permissive Quantifiers and Character Sets: Using .* (any character, zero or more times) or broad character ranges like [a-zA-Z0-9] where more specific validation is needed can create wide open doors for bypasses.

2. Practical Vulnerability Classes:

The talk showed how these regex misconfigurations manifest in real-world vulnerabilities:

• Post Message Vulnerabilities: Flawed origin validation regexes, often missing anchors or proper escaping, allow malicious subdomains to spoof trusted origins, leading to cross-site scripting (XSS).
• Open Redirects: Weak URL validation regexes enable attackers to redirect users to arbitrary domains, facilitating phishing or account takeover (as demonstrated by Ben's $14,000 bounty example).
• CORS Misconfigurations: Lax regex in Access-Control-Allow-Origin headers, typically due to unescaped periods or missing anchors, allows unauthorized domains to make cross-origin requests, leading to data exfiltration.
• Server-Side Request Forgery (SSRF): Inadequate regex for internal resource access (e.g., PDF generation services) can be bypassed to access localhost or cloud metadata endpoints.

3. Regex for Offensive Security (Bug Bounties & OSINT):

Regex is an incredibly powerful tool for offensive security operations:

• Advanced OSINT on GitHub: Ben Sadeghipour highlighted regex's capability to discover hidden assets, subdomains, and API routes on GitHub. This approach often uncovers assets missed by traditional tools, providing valuable context from leaked source code or automation scripts.
• Parsing Data Dumps: Regex is essential for sifting through large, unstructured data (like actuator heap dumps) to extract sensitive information such as passwords, API keys, JWT tokens, and AWS keys. Specific patterns can quickly isolate these secrets from a deluge of logs and memory snapshots.

4. Regex for Defensive Security (Code Review & Tooling):

On the defensive front, regex proves invaluable for:

• Code Review: Automatically identifying potential XSS sinks (e.g., echo $_GET[...] in PHP), Remote Code Execution (RCE) opportunities (e.g., system($_GET[...])), or hardcoded credentials.
• Custom Tool Development: Adam Langley demonstrated how regex underpins tools for generating targeted wordlists (e.g., extracting common parameters or routes from open-source projects) and identifying security-relevant patterns across vast codebases.

In essence, the talk established regex as a double-edged sword: a source of critical vulnerabilities when misused, and an indispensable asset for both finding and fixing those vulnerabilities when wielded with expertise.

Technical Deep Dive

▶ Watch: Defining regex: a powerful pattern search language. (5:00)

The technical core of the "Regex For Hackers" talk was a comprehensive breakdown of regular expression syntax, immediately followed by real-world security implications. Adam Langley meticulously introduced fundamental regex concepts, illustrating how each component works before demonstrating how their misapplication leads to exploitable vulnerabilities.

Regex Fundamentals

1. Literal Matches: The simplest form, matching an exact sequence of characters (e.g., hello matches "hello").
2. Word Boundaries (\b): Used to match whole words by ensuring no word characters (letters, numbers, underscore) are on either side (e.g., \bcomputer\b matches "computer" but not "supercomputer").
3. Anchors (^ and $):

• ^: Matches the beginning of the string (e.g., ^https matches strings that start with "https").
• $: Matches the end of the string (e.g., \.co$ matches strings that end with ".co").
• Crucial for full string validation; their absence is a common vulnerability.

1. The Wildcard Period (.): This is a critical point of failure. By default, . matches any single character except a newline. To match a literal period, it must be escaped as \.. Developers frequently forget this, leading to significant bypasses.
2. Character Sets ([]):

• [abc]: Matches any single character from the set (a, b, or c).
• [^abc]: Inverts the set, matching any character not in the set.
• Character Ranges: [A-Z], [a-z], [0-9] define ranges of characters. These can be combined (e.g., [A-Za-z0-9]).

1. Quantifiers: Specify how many times a preceding element (character, set, or group) must occur.

• +: One or more times (e.g., [A-Z]+ matches "ADAM").
• ?: Zero or one time (optional) (e.g., HTTPS? matches "HTTP" or "HTTPS").
• {n}: Exactly n times (e.g., \d{16} for a 16-digit number).
• {n,}: At least n times (e.g., \d{16,} for 16 or more digits).
• {n,m}: Between n and m times (e.g., \d{16,19} for 16 to 19 digits).
• : Zero or more times (e.g., \s matches zero or more spaces/tabs). This is often overused and leads to overly permissive patterns.

1. Shortcuts: Shorthand for common character sets.

• \d: Any digit (equivalent to [0-9]). \D: Any non-digit.
• \s: Any whitespace character (space, tab, newline). \S: Any non-whitespace.
• \w: Any "word" character (alphanumeric + underscore, equivalent to [A-Za-z0-9_]). \W: Any non-word character.

1. Groups (()):

• Used to apply quantifiers to multiple characters (e.g., (\d{4}\s*){3}\d{4} for credit card numbers with optional spaces).
• Crucially, they capture matched text, allowing specific parts of a string to be extracted (e.g., href="(.*)" extracts the URL).
• OR Operator (|): Within groups, | acts as an "or" condition (e.g., (space|hyphen)).

Vulnerability Deep Dive

Langley then demonstrated how these regex elements are commonly misused, leading to critical vulnerabilities:

1. Post Message Vulnerabilities:

• Vulnerable Regex: www.trusted.com (missing $ and unescaped .)
• Bypass: www.trusted.com.attacker.com (due to missing $) or wwwxtrusted.com (due to unescaped .).
• Impact: Malicious domains can spoof trusted origins, sending arbitrary data (e.g., XSS payloads) to the legitimate application, leading to cross-site scripting (XSS).
• Fix: ^https:\/\/www\.trusted\.com$ – strict anchoring and escaped periods.

1. Open Redirects:

• Vulnerable Regex: trusted.com (missing ^ and $, unescaped ., broad subdomain wildcard).
• Bypass: www.trusted.com.evil.com (missing $), wwwxtrusted.com (unescaped .), or attacker.com?url=trusted.com (missing ^). Ben Sadeghipour shared an example where a path like /application.redacted.com.evil.com led to a $14,000 account takeover by stealing an auth token.
• Impact: Phishing, credential theft, malware distribution.
• Fix: ^https?:\/\/(admin|www|download)\.trusted\.com$ – specific subdomains, anchors, escaped periods.

1. CORS Misconfigurations:

• Vulnerable Regex: https:\/\/sub\.trusted\.com (unescaped ., missing $).
• Bypass: https://subx.trusted.com (unescaped .), https://sub.trusted.com.evil.com (missing $).
• Impact: Unauthorized cross-origin requests can exfiltrate sensitive user data or session cookies (especially in Chrome, which still allows Access-Control-Allow-Credentials with third-party cookies).
• Fix: ^https:\/\/([a-z0-9_-]+\.)?trusted\.com$ – strict subdomain validation, anchors, escaped periods.

1. Server-Side Request Forgery (SSRF):

• Vulnerable Regex: https:\/\/www.linkedin.com (unescaped ., missing ^, $, no path validation).
• Bypass: https://www.linkedin.com.evil.com (missing $), https://wwwxlinkedin.com (unescaped .), attacker.com?url=linkedin.com (missing ^), or https://www.linkedin.com/redirect?to=localhost (no path validation combined with an open redirect on the target).
• Impact: Access to internal services, cloud metadata endpoints (e.g., AWS EC2 metadata), sensitive file disclosure.
• Fix: ^https:\/\/www\.linkedin\.com\/in\/[a-zA-Z0-9_-]+$ – strict domain, path, and user ID validation.

Bug Bounty & Recon Applications (Ben Sadeghipour)

Ben Sadeghipour expanded on regex's utility for offensive operations:

1. GitHub OSINT & Asset Discovery:

• Subdomain Enumeration: Using regex to find specific subdomain patterns (e.g., corp\.site\.com) on GitHub, especially useful for wildcard certificates.
• Content Discovery: Searching for API routes (e.g., /API/v1/) or application-specific paths (e.g., app-name/v1/) to uncover undocumented endpoints. Ben shared an example of finding leaked JROG/Artifactory instances with source code, leading to a $5,000-$6,000 bounty.
• Technology Stack Discovery: Identifying instances of specific technologies (e.g., GraphQL) across a target's GitHub presence.

1. Parsing Data Dumps (Heap Dumps):

• Regex is invaluable for extracting secrets from unstructured data, such as actuator heap dumps from Spring Boot applications.
• Example Patterns:
• .=.: To find potential parameters with key-value pairs.
• .*={2}: To identify Base64 encoded strings (often ending in ==).
• JWT Tokens: eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+ – a highly specific pattern for JWTs, which typically start with eyJ (Base64 for {).
• AWS Keys: Specific regex patterns can be crafted to identify AKIA... (Access Key ID) and wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY (Secret Access Key) patterns.
• Ben noted how he had missed significant data in past heap dumps by not using regex with grep, highlighting its efficiency.

Code Review & Tool Building (Adam Langley)

Finally, Adam demonstrated regex's defensive and utility applications:

1. Code Review:

• XSS Detection (PHP): echo\s*(\$_GET|\$_POST)\[['"]?([^'"]+)['"]?\] – searching for echo statements directly outputting $_GET or $_POST variables without sanitization.
• RCE Detection (PHP): system\s*(\$_GET|\$_POST)\[['"]?([^'"]+)['"]?\] – finding system calls taking user input.
• Identifying Regex in Code: (preg_match|RegExp|re.compile)\s*\(["'\/][^"'\/\n]+["'\/] – a meta-regex to find other regex patterns for manual security review.
• Hardcoded Credentials: Patterns for Basic Auth (Base64 strings) or JWT tokens.

1. Tool Building:

• creatures of habit: A tool that uses regex to parse various framework codebases (Flask, Laravel, ASP.NET, Ruby on Rails, Go) to extract API routes and build comprehensive wordlists for content discovery.
• Parameter/Header/Method Wordlists: Regex can extract common parameters, HTTP headers, and methods from vast code repositories, creating highly effective custom wordlists for reconnaissance.

This deep dive illustrates that regex is not just a theoretical concept but a practical, versatile language with direct and profound implications across the entire spectrum of cybersecurity.

Demo / Proof of Concept

▶ Watch: Literal matches and using word boundaries (\b). (6:00)

While the "Regex For Hackers" talk did not feature a live, interactive demonstration in the traditional sense, the entire presentation served as a comprehensive conceptual demo and proof of concept through its extensive use of illustrative code snippets, regex patterns, and real-world vulnerability scenarios. The speakers effectively demonstrated how regex patterns could be constructed to exploit or defend against vulnerabilities, providing concrete examples rather than abstract theories.

Adam Langley's segment meticulously walked through the construction of regex patterns, from basic literal matches to complex grouped quantifiers, immediately contextualizing each concept with a vulnerable scenario. For instance, he showed how a credit card number regex could initially be too strict, then too broad, and finally refined to accurately match various formats while avoiding false positives. This step-by-step approach acted as a guided demonstration of regex design.

Crucially, the vulnerability examples for Post Message, Open Redirects, CORS misconfigurations, and SSRF were presented with specific, vulnerable regex patterns and corresponding bypass payloads. The "fix" for each vulnerability, involving the correct application of anchors, escaped periods, and specific character sets, served as a clear demonstration of secure regex implementation. Ben Sadeghipour further solidified this with real-world proof of concept examples from his bug bounty experience. He explicitly mentioned a $14,000 bounty he received for an open redirect that chained into an account takeover by stealing an auth token, directly attributing its success to a regex bypass. He also detailed finding leaked JROG/Artifactory source code via GitHub OSINT with regex, leading to a $5,000-$6,000 bounty, and multiple $3,000 bounties from actuator heap dumps by extracting sensitive information with regex patterns.

The speakers also referenced their Hackinghub platform, implying that some of these concepts are available as interactive labs for attendees to practice. Adam Langley's mention of his tools like creatures of habit and parameter wordlist generators, which rely heavily on regex, serves as an indirect demonstration of what can be built with these skills. Therefore, while not a typical live hacking demo, the talk provided a rich collection of actionable patterns, bypasses, and success stories that functioned as compelling proof of concept for the power of regex in cybersecurity.

Defensive Implications

▶ Watch: Understanding anchors for start (^) and end ($). (6:50)

The "Regex For Hackers" talk provides a clear roadmap for defenders to fortify their applications against common regex-related vulnerabilities. The core message is that secure regex implementation requires precision, strictness, and a deep understanding of potential pitfalls.

1. Implement Strict Anchoring: Always use ^ and $ to anchor regex patterns to the beginning and end of the string, respectively, when performing full string validation. This prevents attackers from bypassing checks by prepending or appending malicious data (e.g., trusted.com.evil.com bypasses trusted.com if $ is missing).
2. Escape Special Characters Meticulously: The period (.) is the most common pitfall. When validating domains or file extensions, always escape the literal dot with a backslash (\.). Failure to do so allows x to substitute for . (e.g., wwwxtrusted.com bypassing www.trusted.com). Other special characters like *, +, ?, |, (, ), [, ], {, }, \ also need escaping if intended as literals.
3. Prioritize Specificity over Broad Wildcards: Avoid the indiscriminate use of . (any character, zero or more times) in security-sensitive contexts. Instead, define explicit character sets and quantifiers that precisely match the expected input. For instance, [a-zA-Z0-9_-]+ is far more secure than . for usernames or subdomains.
4. Validate Paths in URLs: For functionalities like open redirects or SSRF, simply validating the domain name is insufficient. Ensure that the entire URL path is validated against expected patterns, preventing attackers from chaining with other vulnerabilities like open redirects on legitimate domains (e.g., linkedin.com/redirect?to=evil.com).
5. Be Explicit with Allowed Domains/Origins: Rather than using broad regex patterns for subdomains (e.g., ([a-z0-9_-]+\.)?trusted\.com), consider explicitly listing allowed subdomains in a group (e.g., (admin|www|download)\.trusted\.com). This "allow-list" approach significantly reduces the attack surface for CORS misconfigurations and open redirects.
6. Regular Code Review for Regex Patterns: Developers and security teams should make reviewing regex patterns a standard part of their code review process, especially for authentication, authorization, input validation, and data handling functions. Use tools (or even regex itself, as demonstrated) to identify all regex instances in the codebase.
7. Implement Secure Secrets Management: The ease with which regex can identify JWT tokens, AWS keys, and other credentials in leaked code or data dumps underscores the importance of robust secrets management practices. Never hardcode sensitive information directly into source code or configuration files that might inadvertently become public.
8. Educate Developers: A fundamental understanding of regex best practices and common pitfalls should be part of developer training. Equipping developers with this knowledge is crucial for writing secure validation logic from the outset.

By adhering to these defensive implications, organizations can significantly reduce their exposure to a wide array of vulnerabilities stemming from improperly implemented regular expressions, turning a common attack vector into a robust line of defense.

Key Takeaways

• Regex is a Fundamental Security Skill: Mastering regular expressions is crucial for both offensive and defensive cybersecurity, enabling precise pattern matching for vulnerability discovery, data extraction, and robust validation.
• Common Regex Errors Lead to Critical Vulnerabilities: Many severe web application flaws, including Post Message XSS, Open Redirects, CORS misconfigurations, and SSRF, stem directly from developers' common mistakes like missing anchors (^, $), failing to escape the wildcard period (.), and using overly broad quantifiers (*).
• Regex Supercharges OSINT and Asset Discovery: Leveraging regex on platforms like GitHub allows for advanced OSINT techniques, enabling the discovery of hidden subdomains, undocumented API routes, leaked source code, and internal assets often missed by traditional reconnaissance tools.
• Efficient Secret Extraction from Data Dumps: Regex is invaluable for parsing large, unstructured data dumps (e.g., actuator heap dumps) to quickly identify and extract sensitive information such as hardcoded credentials, JWT tokens, and AWS keys, which are critical for escalating privileges.
• Essential for Automated Code Review: Security teams can use regex to automate the identification of potential vulnerabilities in codebases, such as direct output of user input (XSS sinks), dangerous system calls (RCE sinks), or the presence of hardcoded secrets, significantly streamlining security audits.
• Build Custom Security Tools and Wordlists: Regex forms the backbone of powerful custom tools for security professionals, enabling the generation of highly targeted wordlists for content discovery and brute-forcing by extracting common parameters, routes, and headers from public code repositories.

About the Speaker(s)

Ben 'nahamsec' Sadeghipour is a highly accomplished bug bounty hunter and educator. He has been actively involved in bug bounties since 2013 or 2014, accumulating over $1.8 million in bounties, with the majority earned since going full-time in 2022 or 2023. Previously, Ben served as the Head of Hacker Education at HackerOne, where he played a pivotal role in training the next generation of security researchers. He is a co-founder of Hackinghub, a company dedicated to providing workshops, courses, and talks to further hacker education. Ben is also widely recognized on social media for his bug bounty content and engages in pentesting and content creation on the side.

Adam 'BuildHackSecure' Langley is a veteran in the hacking and web development communities. He began his hacking journey in the late 1990s and boasts over 20 years of experience in web development. Adam combines these two passions by creating engaging CTFs (Capture The Flag) and learning labs for aspiring hackers. He serves as the CTO of Hackinghub, where he is responsible for building core applications, courses, labs, and CTFs. His educational content can be found on HackingHub, HackerOne, and TryHackMe, among other CTF events.