We Pwn the Night: Growing & Leading an 31337 security research team

We Pwn the Night: Growing & Leading an 31337 security research team

Speakers: Keith Hoodlett

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=VqDGkDcAbLo

Overview

In this compelling talk, "We Pwn the Night," Keith Hoodlett, former Director of IML and Application Security at Trail of Bits, shares his groundbreaking experiment in cultivating an elite security research team. Faced with surging client demand, Hoodlett set an audacious goal in 2024: to expand his team by over 25% in 2025 and ensure every new hire discovered zero-day vulnerabilities within their first 45 days on the job. This presentation meticulously unpacks the innovative hiring, onboarding, and leadership strategies that not only met but exceeded this seemingly impossible target, resulting in every new researcher identifying significant, impactful zero-days.

The talk is a masterclass for security leaders, hiring managers, and aspiring researchers alike, offering actionable insights into building high-performing teams in a challenging talent landscape. Hoodlett delves into the critical shift from traditional hiring metrics to a focus on attitude, aptitude, and engagement, alongside a transparent, people-first leadership philosophy. By detailing the successful outcomes and the underlying cultural and technical frameworks, he provides a blueprint for fostering excellence and continuous learning in the dynamic field of security research.

The significance of Hoodlett's experiment extends beyond Trail of Bits, offering a replicable model for organizations aiming to elevate their security posture through proactive research and talent development. In an era where AI-generated applications complicate hiring and the threat landscape constantly evolves, the strategies presented emphasize the enduring value of human ingenuity, critical thinking, and a passion for the craft. The talk not only celebrates individual achievements but underscores the power of a supportive, challenging, and psychologically safe team environment in driving collective success.

Background

▶ Watch: Ambitious goal: new hires find zero days in 45 days (2:16)

The genesis of this ambitious project stemmed from a significant surge in client demand at Trail of Bits in 2024, projecting substantial growth for 2025. To meet this demand, the company needed to expand its security research team by over 25%. However, traditional hiring practices for application security engineer roles proved inadequate for identifying true security research talent. Common pitfalls included an over-reliance on academic degrees or "paper certifications" like the CISSP, which often indicated theoretical knowledge rather than practical vulnerability hunting skills. Even hands-on certifications like the OCP didn't guarantee the specific research capabilities required by Trail of Bits, and years of "appsec" experience often translated to running tools rather than discovering novel vulnerabilities.

Several challenges exacerbated the hiring process. The role title itself, "Application Security Engineer," was a misnomer for the desired security researcher profile, confusing many candidates. Finding individuals with genuine security research acumen is inherently difficult due to the specialized nature of the field. Additionally, the team operated remotely across multiple time zones and countries, adding complexity to team integration and reporting structures. Financial constraints also played a role, with target salaries of $150-200k for seniors and $175-225k for principals being considered "a little bit low" for the competitive security research market. Despite these hurdles, the ultimate goal remained clear: to find researchers capable of producing impactful findings and demonstrating abilities beyond theoretical knowledge, while maintaining the company's sterling reputation and ensuring new hires were quickly billable in a consultancy model.

Key Findings

▶ Watch: Why traditional hiring fails for security research roles (4:10)

The central and most significant finding of Hoodlett’s experiment was the resounding success of his novel hiring and onboarding methodology. Out of five new security researchers hired in 2025, an astounding 100% (all five) successfully identified zero-day vulnerabilities within their first 45 days on the job. This achievement not only validated the experimental process but also led to impactful disclosures for the affected organizations. Hoodlett highlighted three specific researchers whose discoveries were publicly disclosed and could be discussed:

1. Will Vaner: A Senior IML Security Engineer who discovered two unauthenticated stack-based buffer overflows in the Nvidia Triton Inference Server.
2. Axel Mirchuk: A Senior AppSec Engineer who identified multiple remotely exploitable vulnerabilities, including unauthenticated Remote Code Execution (RCE), in the Thermo Fisher Scientific Ion Torrent OneTouch 2 lab device.
3. Darius Hool: A Senior AppSec Engineer who uncovered an integrity checking bypass in Electron, demonstrating its impact on popular Chromium-based applications like Signal, Slack, and One Password.

This unprecedented success rate demonstrated that a carefully constructed framework—prioritizing attitude, aptitude, and engagement in hiring, coupled with a structured, hands-on onboarding process and a supportive team culture—can consistently cultivate top-tier security research talent capable of immediate, high-impact contributions. The experiment proved that strategic investment in people and process can overcome traditional hiring challenges and accelerate the discovery of critical vulnerabilities.

Technical Deep Dive

▶ Watch: Introducing the new hiring approach: Attitude, Aptitude, Engagement (5:00)

The success of the "We Pwn the Night" experiment rested on a meticulously designed framework encompassing hiring, onboarding, and continuous team cultivation.

Hiring Framework: Attitude, Aptitude, and Engagement

Traditional hiring practices for security research roles were deemed ineffective, often failing to identify candidates capable of finding zero-day vulnerabilities. Instead, the team focused on three core pillars:

1. Attitude: Initial screening rounds with talent acquisition and the director focused on identifying candidates genuinely excited and opinionated about technology. Green flags included intellectual humility, a willingness to admit knowledge gaps, and an eagerness to learn from experts. Red flags were arrogance without substance and an inability to acknowledge gaps in knowledge.
2. Aptitude: This was assessed via a technical take-home assessment. Candidates received Go or C/C++ code, tasked with finding vulnerabilities and writing a professional report. A single individual, Cliff Smith, reviewed all reports for consistency. Successful candidates demonstrated thinking beyond memorization, identifying edge cases, unexpected attack surfaces, and connecting disparate security concepts. Those who provided AI-generated responses or lacked independent thought were quickly identified and deselected.
3. Engagement: A somewhat controversial but crucial aspect was looking for passion extending beyond the 9-to-5. This included participation in CTFs, bug bounties, conference talks, blog posts, open-source tool development, or active discussions on platforms like X/Mastodon. Every successful hire demonstrated evidence of this intrinsic motivation and self-directed learning, which predicted their ability to stay current and hone their craft.

The team also actively combated the influx of AI-generated applications by implementing prompt injections in job descriptions and application questions. A notable example involved asking applicants to respond with "Hilbert" if they were an AI, or "yodell go umlau" if human, effectively filtering out automated submissions.

Onboarding Structure

Once offers were made, a structured four-week onboarding process was critical to accelerating new hires' capabilities:

• Week 1: Context and Learning. New hires reviewed 5-10 recent, extensive reports (35-100 pages each) across various assessment types (code reviews, threat models, infrastructure reviews). They also studied fundamentals in ML, blockchain, and cryptography, providing context on the types of problems Trail of Bits solved and client expectations.
• Week 2: Tooling and Setup. This week focused on installing and configuring the technical stack, including analysis tools like Semgrep and CodeQL, custom analysis rules (both public on GitHub and internal), cloud code, custom configurations, and collaborative tools like Sarif Explorer and Wei Audit. This hands-on setup ensured familiarity with the tools essential for their research.
• Weeks 3 & 4: Internal Research & Development (IRAD). New hires chose a target for internal research, applying the tools and processes learned. This low-risk environment allowed them to gain hands-on familiarity, identify quality findings, and practice contributing to reports and documentation, often by shadowing audits or working as non-billable resources. This is where the "magic happened," with new hires consistently finding zero-days.

Zero-Day Discoveries in Detail

Three specific researchers exemplified the success of this process:

1. Will Vaner – Nvidia Triton Inference Server:

• Vulnerability: Two unauthenticated stack-based buffer overflows.
• Discovery: During onboarding, Will utilized Semgrep with public Trail of Bits rule sets, which flagged an "unsafe alloc" in HTTP handling code.
• Exploitation: The Nvidia Triton Inference Server derived stack allocation sizes from the number of buffer segments in incoming requests. Normally, requests have 1-2 segments. Will realized he could use chunk transfer encoding to send thousands of tiny chunks. libevent fragmented these into many segments, inflating the allocated stack size. Six-byte chunks became 16-byte stack allocations (a 2.7x amplification). A 3MB chunk request led to a stack overflow, resulting in a server crash (seg fault).
• Impact: The vulnerable pattern existed across multiple application routes (inference, model load/unload, logging). Authentication was optional and off by default, making the bug exploitable without credentials. It had existed undetected for over five years and was patched in version 25.07. Publicly listed with CVSS 9.8.
• Key Learning: Static analysis helped identify candidates, but manual reasoning about alternate input vectors (like chunk transfer encoding) was crucial for proving exploitability.

1. Axel Mirchuk – Thermo Fisher Scientific Ion Torrent OneTouch 2 Device:

• Vulnerability: Multiple remotely exploitable vulnerabilities, including unauthenticated RCE.
• Discovery: Axel, new to consulting but with prior experience in security at a financial bank in Canada and strong reverse engineering domain knowledge, combined this expertise with Trail of Bits' threat modeling methodology.
• Impact: He found four vulnerabilities with CVSS scores above 9.0. These were publicly demonstrated at the Junkyard Competition at DistrictCon.

1. Darius Hool – Electron Integrity Checking Bypass:

• Vulnerability: An integrity checking bypass in Electron, allowing local backdoor installations in signed Chromium-based applications.
• Mechanism: Electron uses "integrity fuses" to protect JavaScript contexts. However, this process overlooked V8 heap snapshots. These snapshots contained clobberable JS built-ins, leading to unsigned code execution in signed applications.
• Exploitation: Apps often install to user-writable paths, meaning privilege escalation wasn't required. Darius bypassed operating system code signing by clobbering the Array.isArray built-in, detecting the isolate type, and deploying tailored payloads against specific apps.
• Impact: Demonstrated by installing keyloggers in Signal and One Password, and achieving full Node.js access in Signal. This pattern is already exploited in the wild, notably by the Loki C2 framework from IBM X-Force. The problem extended to other Chromium derivatives like Perplexity Comet, OpenAI's Atlas, and The Browser Company's DIA browser.
• CVSS Score: 6.3 (Medium), due to requiring local user access, but the breadth of impact was massive. Darius specifically highlighted the vulnerability of many cryptocurrency wallets, which often lack integrity checks by default.
• Recommendation: All Chromium derivatives should integrity-check snapshots.

Team Culture and Leadership

Beyond hiring and onboarding, a strong team culture was the "secret sauce":

• Leadership Mantra: "People first, process second, technology third." Great people thrive with good process, but bad process will drive them away. Technology serves the researchers, with custom tools like Sarif Explorer built when open-source alternatives were subpar.
• Challenging, Engaging, Growing:
• Challenge: Provide exciting, adequately difficult work. New hires worked on real assessments immediately, supported by at least two senior engineers. Stretch goals during onboarding included building tools to aid their first 90 days.
• Engage: Prevent burnout through diverse work assignments (every 4-8 weeks) and dedicated IRAD time (2-3 weeks quarterly) for tool building, research, blog posts, and conference talks.
• Grow: Invest in continuous development through training, conferences, and clear career paths (individual contributor and technical leadership). An aspirational skills matrix helped leaders tailor work to individual growth goals.
• No Surprises Mindset: Transparency in leadership built trust and psychological safety. New hires saw drafts of work-in-progress documents and were encouraged to provide feedback, removing fear and fostering focus on work rather than politics.
• Staying Technical as a Leader: Hoodlett emphasized deliberate practice (2 hours/day, 4 days/week minimum) using a "study-do-teach" loop to avoid the "middle management trap." This allowed him to ask intelligent questions, understand task difficulty, contribute credibly, and model continuous learning for the team. Resources like Eugene Lim's "Day Zero to Zero Day" and Jason Hadex's courses were recommended for staying sharp.

Demo / Proof of Concept

▶ Watch: Screening candidates for the right attitude and mindset (5:15)

The talk effectively showcased the practical impact of the security research through the demonstrations and proof-of-concept exploits developed by the new hires.

Axel Mirchuk's findings in the Thermo Fisher Scientific Ion Torrent OneTouch 2 device, including unauthenticated RCE, were publicly presented and demonstrated at the Junkyard Competition at DistrictCon. While the exact details of the live demonstration were not fully elaborated in the talk, the mention of its public presentation at a prominent competition highlights the tangible and exploitable nature of his discoveries. These types of live demonstrations are critical for validating the severity and exploitability of vulnerabilities, often involving real-time execution of arbitrary code or unauthorized control over the target device.

Darius Hool's integrity checking bypass in Electron was demonstrated by installing keyloggers within signed applications like Signal and One Password. Furthermore, he achieved full Node.js access within Signal. These demonstrations showcased how his technique could inject and execute arbitrary code, effectively backdooring applications that were presumed secure due to their code signing and integrity checks. The ability to install a keylogger directly illustrates the profound impact on user privacy and data security, while gaining Node.js access signifies complete control over the application's runtime environment, allowing for broad malicious activities. The fact that this vulnerability could bypass operating system code signing and operate in user-writable paths made the demonstration particularly alarming, as it did not require elevated privileges to compromise widely used applications.

Defensive Implications

▶ Watch: Technical take-home assessment for aptitude (6:30)

The insights from "We Pwn the Night" offer multifaceted defensive implications, spanning organizational strategy, hiring practices, and specific technical countermeasures against the highlighted vulnerabilities.

From an organizational and leadership perspective, companies should:

• Rethink Hiring: Adopt the "attitude, aptitude, and engagement" framework for security roles. Prioritize intellectual humility, passion for technology, and demonstrable hands-on skills over paper certifications or mere years of experience. Implement safeguards like prompt injections to filter out AI-generated applications, ensuring genuine human talent.
• Structured Onboarding is Crucial: Implement a robust onboarding program that includes reviewing past reports for context, dedicated time for tool setup and mastery (e.g., Semgrep, CodeQL, Wei Audit), and crucially, Internal Research & Development (IRAD) time. This low-risk environment allows new hires to gain practical experience and discover impactful findings before client work.
• Cultivate a People-First Culture: Leaders must prioritize "people first, process second, technology third." Foster an environment of psychological safety and transparency ("no surprises") to build trust, encourage risk-taking, and prevent burnout. Invest in continuous learning, training, and clear career paths to grow and retain talent.
• Leaders Must Stay Technical: Security leaders should actively engage in deliberate practice (study, do, teach) to maintain their technical skills. This ensures credibility, allows them to understand team challenges, and models continuous learning, which is vital for high-performing research teams.

Regarding the specific technical vulnerabilities discovered:

• Nvidia Triton Inference Server: Organizations using Nvidia Triton Inference Server (or similar inference servers) should immediately update to version 25.07 or later to patch the unauthenticated stack-based buffer overflows. More broadly, developers should be vigilant about how HTTP handling code processes incoming request segments and applies dynamic memory allocation, especially when dealing with variable-length inputs and chunked transfer encoding, to prevent buffer overflows. Static analysis tools like Semgrep should be integrated into CI/CD pipelines to flag unsafe allocation patterns.
• Thermo Fisher Scientific Ion Torrent OneTouch 2 Device: Users of IoT and specialized lab equipment, particularly the Ion Torrent OneTouch 2 device, must apply all available security patches promptly. This case highlights the critical need for comprehensive security assessments, including threat modeling and reverse engineering, for embedded systems and specialized hardware, as these devices often harbor severe, unauthenticated vulnerabilities like RCE. Network segmentation and strict access controls should be applied to such devices to limit their exposure.
• Electron Integrity Checking Bypass: All Chromium-based applications, including Electron apps (Signal, Slack, One Password, etc.) and derivative browsers (Perplexity Comet, OpenAI Atlas, DIA Browser), must implement robust integrity checking for V8 heap snapshots. This is a critical gap that allows local users to bypass operating system code signing and backdoor signed applications. Developers should ensure that all executable content, including JavaScript contexts derived from snapshots, is cryptographically verified. Users should be aware that even signed applications can be compromised if installed in user-writable paths and should maintain strong endpoint security. The vulnerability also serves as a stark warning for cryptocurrency wallet developers, many of whom lack adequate integrity checks, making them susceptible to similar local backdooring techniques.

Key Takeaways

• Innovative Hiring Works: Prioritize attitude, aptitude, and engagement over traditional credentials to find true security research talent. Combat AI-generated applications with creative screening methods.
• Structured Onboarding Accelerates Impact: A dedicated, hands-on onboarding process including report review, tool mastery, and internal research time is crucial for new hires to find impactful vulnerabilities quickly.
• People-First Culture is Non-Negotiable: Foster psychological safety, transparency ("no surprises"), and invest in challenging, engaging, and growing your team. This builds trust, prevents burnout, and retains valuable talent.
• Leaders Must Stay Technical: Active technical engagement from leadership (e.g., "study-do-teach" loop) is vital for credibility, understanding team challenges, and modeling continuous learning.
• Zero-Days Are Discoverable with the Right Approach: The experiment proved that a well-designed process can lead to 100% of new hires finding impactful zero-day vulnerabilities within weeks.
• Technical Vulnerabilities Demand Attention: Specific issues like stack overflows in inference servers, RCE in lab devices, and integrity bypasses in Electron-based applications require immediate patching and broader architectural considerations for defense.

About the Speaker(s)

Keith Hoodlett is a distinguished security professional with a rich background in application security and security research. At the time of this talk, he was the former Director of IML and Application Security at Trail of Bits. His expertise is further demonstrated by his ranking as a top 300 ethical hacker on Bug Crowd, holding certifications such as OCP (Offensive Security Certified Professional) and OSWA (Offensive Security Web Assessor). Keith is also a prolific writer, contributing to a newsletter and blog posts, and has been a speaker on various conference stages and podcasts, including Risky Business, Zero Signal, Critical Thinking, and Application Security Weekly. Beyond his professional achievements, he is a dedicated cat dad, a Zen practitioner for over a decade, and a patron of the arts. Notably, since giving this talk, Keith has joined One Password as the Director of Security Research, where he is building a new greenfield team from the ground up, aiming to replicate the successful experiment discussed in his presentation.