Startup Security, 2nd Edition

Startup Security, 2nd Edition

Speakers: Evan Johnson

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=DyRxMmypt_g

Overview

Evan Johnson, co-founder and CEO of Run Reveal, delivered an insightful and practical talk titled "Startup Security, 2nd Edition" at BSidesSF 2024. This presentation served as an updated and expanded version of his highly regarded 2019 talk at AppSec Cali, offering a candid look at the unique challenges and opportunities of building a security program from the ground up at a startup. Drawing on his extensive experience as the first security engineer at industry giants like Cloudflare and Segment, Johnson provided a roadmap for security professionals navigating the hyper-growth, often chaotic environment of a nascent company.

The core of Johnson's talk revolved around actionable strategies for the first security hire, emphasizing speed, adaptability, and the critical importance of achieving tangible wins early on. He delved into the nuances of career growth within a startup, common pitfalls to avoid, and strategic decisions like when to build custom security solutions versus when to buy off-the-shelf products. The updated "2nd Edition" incorporated broader experiences in management and security leadership, offering a more holistic perspective on establishing a robust security posture in a rapidly evolving organization.

This article distills Johnson's invaluable advice, providing a detailed technical overview of his framework for startup security. It highlights the critical phases of a security professional's journey in a startup, from the initial 90 days to long-term strategic planning, and offers concrete defensive implications for organizations aiming to embed security effectively from their earliest stages. The talk is particularly relevant for security engineers, CISOs, and anyone considering or currently working in a security role at a fast-growing startup, offering a blend of technical guidance and career wisdom.

Background

▶ Watch: Speaker's background as first security engineer at Cloudflare/Segment, seeing... (00:00)

Evan Johnson's journey into startup security began with a unique vantage point: he was the first security engineer at Cloudflare and later the first security hire at Segment. His career trajectory saw him boomerang back to Cloudflare, experiencing the full spectrum of growth from a company with no dedicated security team to a post-IPO, FedRAMP-compliant organization. This firsthand exposure to the entire maturity curve of security teams, coupled with his candid admission of making "a lot of mistakes" alongside "a lot of things right," forms the bedrock of his advice. The "Startup Security" talk itself is a testament to this experience, with the 2024 "2nd Edition" building upon a successful 2019 presentation at AppSec Cali, which had garnered significant positive feedback from individuals starting security roles at startups.

Johnson began by exploring the motivations behind working at a startup. From a personal perspective, career growth stands out. He recounted starting at Cloudflare at 24, immediately being tasked with securing "4,000 lines of C code" for a release the following week. This "insane level of responsibility" is a hallmark of startups, offering unparalleled opportunities for rapid personal and professional development that might be unattainable in larger, more structured environments. Beyond growth, startups often foster enjoyment through collaboration with like-minded individuals and the potential for significant financial upside if the company achieves a successful exit, though Johnson acknowledged this is a "pretty rare" outcome.

However, Johnson also presented a realistic counterpoint, outlining reasons not to work at a startup. Compensation is typically lower initially compared to established tech giants, and this "underpaid" status can persist as the company grows. Work-life balance is often poor, characterized by chaos, long hours, and personal sacrifices, with Johnson himself recalling "losing a lot of weekends, holidays, three-day weekends at Cloudflare" due to security incidents. Finally, internal stability is inherently low in hyper-growth environments, with constant hiring, reordering, and personnel changes contributing to a chaotic atmosphere.

The central problem addressed by the talk is the daunting scenario faced by the first security hire. Johnson vividly described the "record scratch" moment: after a week of onboarding, the new hire's manager (be it CTO, CEO, or CFO) turns to them and asks, "I really trust you to build our security program... what do you think you have in store for the next six months?" This talk aims to provide a concrete answer to that question, offering a game plan to navigate this initial period and lay a strong foundation for security.

Key Findings

▶ Watch: Example of finding a critical first task: Cloudflare production database maki... (11:00)

Evan Johnson's talk distilled years of experience into several critical findings for establishing and growing a security program at a startup. These insights emphasize speed, practicality, and strategic thinking over rigid, theoretical frameworks.

A paramount finding is that the first 90 days are critical for any new security hire. Johnson referenced research on 30-60-90 day plans, outlining a clear progression:

• First 30 days: Focus on building relationships. "Meet all your stakeholders, make friends," he advised, clarifying that "you don't actually have to make friends but you do need to develop a working relationship with everybody who is going to be important," from engineers to customer success, PR, and legal teams. Understanding customer contracts is also key.
• First 60 days: Develop a clear plan for what to accomplish in the first 90 days and begin taking concrete steps towards it.
• First 90 days: It is "critical that you've accomplished something and kind of told the world about it at the company." Johnson stressed that individuals who achieve something tangible in this period are "significantly more likely to be successful over the long term." This could be as small as "flipping a bit enabling two-factor authentication," but it must be a demonstrable win.

Johnson highlighted the importance of humbling yourself, particularly for CISOs or heads of security transitioning from larger organizations. The pace and expectations at a startup are vastly different; what might take a year at a big company needs to be condensed into 90 days. Getting "hands dirty" and accomplishing something, rather than just opening job requisitions, is essential. For engineers, this adjustment is often easier, as their natural inclination to "listen and learn" helps identify immediate problems, as Johnson did at Cloudflare when a co-founder pointed out insecure curl requests from the production database.

Following initial successes, it's crucial to celebrate small wins. Startups thrive on this culture, and security teams should emulate it. Acknowledging contributions from collaborators and publicly sharing achievements fosters a positive "shipping culture" and ensures the security team is seen as a partner, not a "scary" impediment.

A significant trap to avoid is what Johnson termed "trench warfare problems"—large, slow-moving projects that yield little progress (e.g., service-to-service authentication). Instead, prioritize throughput: "anytime you could do three things that you can call done in the same time frame that you could do one thing that is like half done, you probably want to opt for the three things." This emphasizes iterative progress and tangible results.

Johnson firmly stated that security is not a spectator sport. Security teams must "roll up their sleeves and take ownership of some problems," even if they only own a part of the solution. Expecting others to fix issues without active collaboration ("throw something over the wall") is ineffective. Security must "do like half of it and help people meet you halfway."

He proposed two simple rules for any security initiative:

1. You can explain why it's important in common English, in two sentences or less, understandable by anyone in the company.
2. You can actually complete it.

This framework helps scope projects effectively and ensures their value is clear and achievable.

A holistic approach to security is vital. Johnson noted that early in his career, he focused heavily on security engineering, but later realized the equal importance of compliance (GRC), detection & response, and enterprise security. Neglecting any of these areas risks incomplete coverage.

Crucially, every company has a "one very special thing" that differentiates it. At Cloudflare, it was their "giant Edge" network; at Segment, a "giant data pipeline." Identifying and deeply understanding this unique asset is paramount for tailoring effective security controls, rather than just applying generic cloud and AppSec practices.

Finally, Johnson stressed the importance of establishing a security story early. This involves defining and communicating the unique controls built around the company's core assets, both internally and externally through blogs and conference talks. Transparency in this story is a strong indicator of a healthy company culture. He also warned against wasting time: "the longer you wait to do things the harder it becomes to do them later as the company grows." A simple "back of a napkin" calculus—if headcount doubles, will this be significantly harder?—can guide decisions on when to act. This also extends to the build vs. buy calculus, where Johnson cautioned against spending "millions in engineering resources to save tens of thousands of dollars" by building something that could be bought, especially for non-differentiated capabilities.

Technical Deep Dive

▶ Watch: Security is not a spectator sport: security teams must take ownership and bui... (13:00)

Johnson's "Big Grid" serves as the central technical framework for his talk, categorizing security initiatives into four key areas: Security Engineering, Compliance (GRC), Detection & Response, and Enterprise Security. While acknowledging some "notable absences," he presented this 4x4 grid as a comprehensive starting point for a holistic security program at a startup.

Security Engineering

This domain covers the technical aspects of securing the product and infrastructure.

• Cloud and AppSec Security: Johnson noted that these areas often look "pretty much the same every company." This includes standard practices like security reviews, implementing security controls to prevent public S3 buckets, and safeguarding against AWS key leaks. These are foundational but not necessarily differentiating.
• Unique Company Aspect: The critical insight here is to identify and secure the "one very special thing" unique to each company. Johnson provided examples:
• At Cloudflare, it was their "giant Edge" network, involving shipping servers globally, booting them up, and running their service. The unique management of this Edge presented distinct security challenges.
• At Segment, it was a "giant data pipeline" with its own set of unique problems.

Security teams must "really dig into what was unique" and tailor controls accordingly, rather than getting "too hung up" on generic AppSec.

• Engineers Should Engineer: Johnson emphasized that security is not a spectator sport. Security engineers should actively "build something," even if it's initially uncomfortable or not universally popular with other teams. This hands-on approach deepens understanding of the product and the company.
• Avoid Toil: A common pitfall is creating unnecessary operational burden. Johnson cautioned against simply "spinning up scanners," collecting results, and filing Jira tickets, expecting others to fix them. This approach generates "toil" that requires constant maintenance. Prioritizing low-toil solutions is crucial for maintaining throughput.
• Bug Bounty: While essential for providing a channel for security researchers to report vulnerabilities, a bug bounty program requires careful management. Rolling it out "too fast, too big, too quick" can "consume all of your team's resources."

Compliance (GRC)

This area focuses on governance, risk, and compliance, often driven by customer and regulatory demands.

• Security Addendums: Understanding the security promises made by the sales team in customer contracts is vital. These addendums dictate specific controls and assurances the company must provide.
• SOC 2 Type 2: Johnson highlighted the increased prevalence and importance of SOC 2 Type 2 since his 2019 talk, facilitated by platforms like Vanta, Drata, and Secureframe. He views getting SOC 2 Type 2 early (even for a year-old company) as "largely a good thing" for building a culture of security and getting compliance "out of the way." However, he also noted that it "doesn't hold a lot of water in terms of like your security practices" alone.
• Sales Enablement: Security plays a significant role in sales. Creating collateral and "greasing the wheels" ensures that security doesn't become a blocker in the sales process.

Detection & Response

This category deals with identifying and reacting to security incidents.

• Centralized Logging: The foundation of detection. Johnson advocated for "simple" and "basic logs" that are easy to manage.
• Response Playbook: More critical than just collecting logs is having a clear response playbook. This includes:
• A designated channel for reporting problems (e.g., a security@ email alias, referred to as the "hotline").
• A process for triaging incidents.
• Clear guidelines on "who you're going to involve" and assembling "The A Team" for incident response.
• Common Incidents: Johnson noted that "largely it's just like fishing emails that comes into that" for many startups.
• Endpoint Detection and Response (EDR): While a "big cost," EDR is an eventual necessity. The timing of its implementation can vary based on immediate priorities and budget.

Enterprise Security

This covers security for the internal corporate environment and employee access.

• YubiKeys (or similar FIDO2 hardware tokens): Johnson emphatically stated, "in 2024 there's no excuse not to have UB keys for and be UB key only at your company especially a startup." He stressed that the cost is minimal for a startup, and it's a worthwhile "slow" investment to prevent basic phishing attacks ("hello I'm from your IT team please go to evil.com and enter your password").
• Single Sign-On (SSO): Implementing SSO "will save you a lot of toil and make you friends with IT." It centralizes access management and improves security posture.
• Onboarding and Offboarding: Streamlining these processes securely is crucial for managing access throughout an employee's lifecycle.

Johnson concluded this section by reiterating that "there's infinite depth you can go into on of these things but just scratching the surface is usually the 80% rule tells you that you'll have a fantastic security program if you just scr do these things." He emphasized simplicity over "crazy Frameworks with 100 check boxes."

Demo / Proof of Concept

▶ Watch: Strong recommendation for UB keys and SSO for Enterprise Security, highlighti... (21:00)

The "Startup Security, 2nd Edition" talk by Evan Johnson did not feature a live demonstration or a specific proof of concept of a security tool or technique. Instead, the presentation focused on providing strategic and practical advice for building a security program within a startup environment.

Johnson utilized a hypothetical scenario involving a fictitious AI startup named "Robo brain" to illustrate the initial challenges faced by a first security hire. He also presented a "Big Grid" framework, a 4x4 matrix categorizing various security initiatives, to visually represent the holistic approach required for startup security. While these visual aids and scenarios helped convey the concepts, they were not interactive demonstrations of technical solutions. The talk's emphasis was on high-level strategy, cultural integration, and prioritization, rather than showcasing specific technical implementations.

Defensive Implications

▶ Watch: The 'Build vs. Buy' dilemma: avoiding the engineer's trap of building somethi... (25:00)

Evan Johnson's insights offer a robust framework for defenders operating in the dynamic startup environment. His advice translates directly into actionable strategies for building resilient security programs from the ground up.

1. Prioritize Foundational Controls with High Impact: Defenders should immediately focus on implementing basic, yet highly effective, security controls. Johnson's strong recommendation for YubiKeys (or similar FIDO2 hardware tokens) for all employees, coupled with Single Sign-On (SSO), is paramount. These measures directly counter prevalent and successful attack vectors like phishing and credential theft, which Johnson noted are still surprisingly effective ("it's amazing that that still works"). Investing in these "slow" but critical solutions early prevents significant future pain and cost.
2. Establish a Lean and Effective Incident Response Capability: Rather than over-engineering, defenders should prioritize a simple, centralized logging system and, more importantly, a clear response playbook. This includes a designated security@ alias for reporting, a defined triage process, and a pre-identified "A Team" of individuals to involve during an incident. This ensures that when "bad things happen," there's a clear, swift, and coordinated response, even if the primary incidents are initially "just like fishing emails."
3. Embrace Proactive Compliance and Sales Enablement: Pursuing certifications like SOC 2 Type 2 early is a strategic defensive move. While not a panacea for all security woes, it embeds a security culture, addresses customer requirements, and "greases the wheels" for sales. Defenders must understand the security addendums in sales contracts and proactively create collateral to prevent security from becoming a blocker, thereby supporting business growth securely.
4. Deeply Understand and Secure the "Special Thing": A critical defensive strategy is to identify and focus security efforts on the company's unique, differentiating asset or technology. Whether it's a global Edge network or a massive data pipeline, generic security controls are insufficient. Defenders must "dig in" to understand the specific risks and build tailored controls around this core asset, ensuring its protection is paramount.
5. Empower Security Engineers to Build (Strategically): Security engineers should be encouraged to get hands-on and build security solutions, as this fosters a deeper understanding of the product and company. However, this must be balanced with the "build vs. buy" calculus. Defenders should avoid "trench warfare problems" and building solutions that are not differentiated or could be more efficiently acquired commercially. The goal is to build what truly adds unique value and buy what provides commodity security efficiently.
6. Minimize Toil and Maximize Throughput: Defensive strategies should prioritize solutions that reduce ongoing maintenance and operational burden. Automated, low-toil security processes free up the security team to focus on higher-value, proactive tasks rather than reactive firefighting or managing complex, self-built systems that could have been bought. The emphasis on "three things quickly rather than one thing slow" applies directly to defensive project prioritization.
7. Communicate Security Value Clearly: Defenders must articulate the importance of security initiatives in plain, business-understandable language. This fosters collaboration, secures buy-in from other departments, and ensures that security is seen as an enabler rather than an impediment. This transparency is also key to establishing a strong "security story" that resonates both internally and externally.
8. Act with Urgency and Foresight: The "back of a napkin" calculus—considering how much harder a task will become as the company grows—is a vital defensive tool. Implementing critical controls and establishing foundational processes sooner rather than later prevents exponential increases in complexity and cost as headcount and infrastructure scale. This proactive approach is essential for maintaining control in a hyper-growth environment.

By integrating these defensive implications, startups can build security programs that are not only robust and effective but also agile and aligned with the company's rapid growth trajectory.

Key Takeaways

• The First 90 Days are Pivotal: As a new security hire at a startup, the initial 90 days are critical for building relationships across all departments and achieving at least one tangible, visible security accomplishment. This early success significantly impacts long-term effectiveness and credibility.
• Prioritize Throughput Over Perfection: Avoid getting bogged down in large, slow-moving "trench warfare" projects. Instead, focus on completing multiple smaller, high-impact tasks quickly. Three completed tasks are better than one half-finished large one.
• Security is an Active, Collaborative Sport: Security teams must actively engage, take ownership of problems, and work collaboratively with other departments. Expecting others to fix issues without offering hands-on support is ineffective; security must "do half of it and help people meet you halfway."
• Implement Foundational Enterprise Security Early: Essential controls like YubiKeys (or similar FIDO2 hardware tokens) for all employees and Single Sign-On (SSO) should be prioritized. These measures are cost-effective for startups and prevent common, high-impact attacks like phishing, while also streamlining IT operations.
• Identify and Secure Your Company's Unique Asset: Beyond generic cloud and AppSec, deeply understand what makes your company's technology or business model unique (e.g., Cloudflare's Edge, Segment's data pipeline). Tailor specific security controls to protect this core asset, as it represents the most critical risk.
• Strategic Build vs. Buy Decisions: Carefully evaluate whether to build custom security solutions or purchase commercial products. While engineers may be tempted to build, buying can often save significant time and resources, especially for non-differentiated capabilities, allowing the team to focus on truly unique and impactful security engineering.
• Establish a Transparent Security Story: Define and openly communicate your company's security posture, especially regarding its unique assets and controls, both internally and externally through blogs and conference talks. Transparency fosters trust and demonstrates a strong security culture.

About the Speaker(s)

Evan Johnson is the co-founder and CEO of Run Reveal. He possesses extensive experience in building and scaling security programs within high-growth startup environments. Johnson was notably the first security engineer at Cloudflare and subsequently the first security hire at Segment. His career trajectory includes boomeranging back to Cloudflare, providing him with a comprehensive understanding of the entire maturity curve of security teams, from inception to post-IPO and FedRAMP compliance. This talk, "Startup Security, 2nd Edition," marks his second time speaking at BSidesSF and his first time sponsoring the conference, building upon the success of his initial "version 1.0" talk given at AppSec Cali in 2019.