What a False Alarm Taught Us About Security as a 2-Person Startup

What a False Alarm Taught Us About Security as a 2-Person Startup

Speakers: Alex Chantavy, Co-founder, Subimage; Kunaal Sikka, Co-founder, Subimage

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=jflIdG5Q4gU

Overview

In a candid and surprisingly vulnerable talk at BSides SF, Alex Chantavy and Kunaal Sikka, co-founders of the security startup Subimage, shared a deeply personal and professionally embarrassing incident: a security false alarm that nearly derailed their nascent company. This talk delves into the unique challenges faced by security startups, the psychological toll of a perceived breach, and the invaluable lessons learned when operating under immense pressure with limited resources. Far from being a mere recounting of a mistake, Chantavy and Sikka transform their experience into a powerful narrative about customer empathy, product development, and the enduring importance of fundamental security practices, even for teams that consider themselves experts.

The presentation highlights the irony of two seasoned security professionals—one from the NSA and Microsoft's Azure Red Team, the other with extensive experience at Lyft and Anthropic—falling prey to a basic misconfiguration. Their journey from panic and self-doubt to profound insights offers a rare glimpse into the human side of incident response within a high-stakes startup environment. By openly discussing their "holy moly moment," the speakers aim to provide a cautionary tale and a blueprint for other founders and security practitioners, emphasizing that even with a robust security stack, vigilance, process, and a clear head are paramount.

The core of their story revolves around a database authentication failure that escalated into a full-blown incident response scenario, complete with wiped laptops, deleted databases, and the looming threat of a Data Protection Agreement (DPA) breach notification within a strict 72-hour window. This perceived crisis, ultimately triggered by a single, case-sensitive character error in a username, became a crucible that forged their understanding of what truly matters in building a security product and serving customers effectively.

Background

▶ Watch: Co-founders introduce Subage, talk's embarrassing purpose (0:25)

Alex Chantavy, with 15 years in security spanning the NSA, Microsoft's Azure Red Team, and Lyft, co-founded Subimage with Kunaal Sikka, who brought experience from Lyft (where he used Chantavy's open-source Cartography tool) and Anthropic. Their company, Subimage, offers a hosted version of Cartography, a tool designed to ingest diverse data sources into a knowledge graph for advanced security use cases like identifying attack paths and vulnerability management.

Having successfully secured a $4.2 million seed round and landed their first enterprise customers, Subimage was at a pivotal growth stage. However, as a security startup, they faced unique challenges: a highly skeptical audience of security professionals, stringent vendor intake processes, and the need to demonstrate a maturity level far beyond their two-person team size. To address this, they prided themselves on establishing a robust security posture from day zero. Their initial security stack included Okta for Identity Provider (IDP), Rippling MDM for Mobile Device Management, an Endpoint Detection and Response (EDR) solution (later criticized for its usability), diligent use of password managers, widespread adoption of OAuth, and comprehensive AWS security features like GuardDuty, along with extensive metric collection. They humorously referred to this as being on the "midwit" part of the bell curve, perhaps overthinking their initial security tooling for a two-person operation, yet believing it was essential for credibility and SOC 2 compliance.

The incident was sparked by an infrastructure migration. Subimage was transitioning its backend from a scrappy EC2 instance to AWS ECS Fargate for containerized deployments, aiming for greater reproducibility and faster development cycles. The heart of their system, the Neo4j graph database, was being migrated to Neo4j Aura, a hosted cloud version. As part of this process, a subset of customer data was moved to the new database. The migration seemed straightforward, involving minimal code changes and primarily infrastructure configuration. Initial tests were successful, but two days before the "incident," they noticed an inability to authenticate to the new database from their laptops, despite using their trusted password manager. Dismissing it as a potential typo amidst other startup priorities, they unknowingly set the stage for the dramatic events that followed.

Key Findings

▶ Watch: High bar and skepticism for security startups (3:10)

The perceived crisis began the following day when, after a morning of continued local authentication failures, Chantavy and Sikka observed a "successful log on" in their Neo4j Aura logs that they did not recognize. This immediately triggered a "holy moly moment." With their laptop credentials still failing, the successful login from an unknown source painted a picture of a potential breach. The stakes were incredibly high: as a young company with new enterprise contracts, their reputation and very existence hinged on maintaining customer trust. Adding to the pressure was their Data Protection Agreement (DPA), which mandated notifying customers of a material data breach, as defined by GDPR, within 72 hours. The clock was ticking, and panic began to set in.

Their initial assessment found no evidence of brute-forcing or suspicious IPs beyond the single successful login. They questioned whether their laptops or phones had been compromised but found no evidence from their EDR solution, which they noted was difficult to use and invoked little trust. Furthermore, their internet-exposed assets, including the new Neo4j database, were carefully configured. While the database itself was temporarily open to the internet during migration, access was behind 2FA, and their own product confirmed no other unintended exposures.

Under the immense pressure of the DPA deadline and the fear of complete company failure, they took drastic containment measures. In a decision they later regretted, they wiped both their laptops, losing potential forensic evidence and setting back their operational capabilities. Simultaneously, in a desperate attempt to contain any perceived attacker, Alex deleted the entire Neo4j database, inadvertently losing crucial security logs from the past that were tied to the resource itself (a design flaw they later reported to Neo4j, as major cloud providers typically retain logs).

The turning point came on day two, when, exhausted and out of ideas, they contacted Neo4j support for urgent assistance, specifically requesting restoration of the deleted logs. Neo4j's prompt response led to a call the next morning, where their support team shared the restored authentication activity logs. The revelation was both simple and deeply embarrassing: the successful logins were legitimate, originating from their own ECS Fargate production environment. The perceived "unrecognized" login was their own application. The local authentication failures, which had triggered the entire panic, were due to a subtle, case-sensitive error in the username: "Neo4j" had been incorrectly saved as "Neo4J" in their password manager for local development. There was no breach, no unauthorized authentication, and nothing to report to customers. The entire incident was a false alarm born from a combination of a minor typo, a lack of holistic log visibility, and the blinding effects of panic.

Technical Deep Dive

▶ Watch: Our "robust" security stack as a two-person startup (4:05)

Subimage's product, Cartography, is a powerful knowledge graph for security. It operates by ingesting data from various sources (e.g., AWS, Okta) into a centralized graph database. This allows security teams to identify attack paths, manage vulnerabilities, and gain comprehensive visibility into their infrastructure and assets. The core of their system relies on Neo4j, a leading graph database, which for this incident, was being migrated to its hosted cloud offering, Neo4j Aura.

The infrastructure migration was from AWS EC2 instances to AWS ECS Fargate. This transition aimed to leverage the benefits of containerization, such as improved reproducibility and faster deployments. As part of this, a new Neo4j Aura database instance was provisioned, and a subset of customer data was migrated to it. The application logic, now running on Fargate, was configured to connect to this new database.

The technical misstep that initiated the incident was a subtle case-sensitivity error in the username used for local development. The default privileged user for Neo4j is "neo4j". However, in their password manager, they had inadvertently saved it as "Neo4J" for their local development environment connecting to the new database. This meant that while their production environment (ECS Fargate) was successfully authenticating with the correct, case-sensitive "neo4j" username, their local development attempts consistently failed.

The critical missing piece in their initial investigation was the inability to easily correlate security logs (authentication attempts) with query logs (actual data operations) within the Neo4j Aura console at the time. While they saw failed logins from their laptops and a successful login, they did not immediately realize the successful login was from their own ECS Fargate production application performing legitimate operations. The Neo4j Aura log viewer, as described by the speakers, required requesting logs in 30-minute increments, making a comprehensive, real-time correlation extremely cumbersome during an active incident. Had these logs been easily viewable and correlatable, they would have seen that the query logs associated with the "successful login" were entirely normal and consistent with their application's expected behavior, immediately clarifying the situation.

Their pre-existing security stack, while comprehensive for a two-person startup, also presented challenges. They used Okta for IDP and Rippling MDM for device management, which were robust. However, their chosen EDR solution proved difficult to navigate, with a "UI from like '98," leading to a lack of trust in its ability to provide actionable insights. This meant they couldn't confidently rule out endpoint compromise, contributing to the decision to wipe their devices. The incident also exposed a gap in their log management strategy: while they had AWS GuardDuty and collected metrics, they hadn't set up centralized log forwarding for third-party applications like Neo4j Aura. This meant that when Alex deleted the database, all associated security logs were lost, further hindering their ability to perform forensics and validate their assumptions, a stark contrast to how major cloud providers typically retain CloudTrail logs even after resource deletion.

In summary, the technical core of the false alarm was a fundamental authentication detail (case sensitivity) exacerbated by a fragmented and difficult-to-use logging interface, which, under the psychological duress of a looming DPA deadline, prevented two experienced security professionals from quickly identifying the root cause.

Demo / Proof of Concept

▶ Watch: What Subage does: hosted Cartography knowledge graph (6:05)

This talk was a retrospective account of a security incident, not a demonstration of a tool or a proof of concept exploit. The speakers focused on narrating their experience, the lessons learned, and the subsequent improvements to their own product, rather than showcasing technical demonstrations.

Defensive Implications

▶ Watch: The incident begins with an infra migration (7:00)

The "false alarm" incident, despite its embarrassing root cause, provided Subimage with invaluable, albeit painful, lessons that offer critical defensive implications for any organization, particularly startups and security companies.

1. Develop and Adhere to Incident Response Playbooks: Panic is a powerful inhibitor of rational thought. Even experienced security professionals can be blinded by fear and pressure, leading to impulsive, regrettable actions (like wiping devices or deleting databases without preserving logs). Having a clear, well-documented incident response playbook is crucial. This playbook should outline the steps for initial assessment, containment, eradication, recovery, and post-incident analysis. It provides a structured approach, helping teams remain disciplined and methodical, even when "shooting from the hip" feels intuitive.

1. Implement Centralized Log Management: Relying solely on a third-party vendor's log viewer, especially one with a poor user interface or cumbersome retrieval process, is a critical vulnerability. Organizations should establish centralized log forwarding for all sensitive resources, including third-party applications and cloud services, into a dedicated Security Information and Event Management (SIEM) or log aggregation platform. This ensures log persistence, even if the original resource is deleted, and facilitates efficient correlation of events across different systems. The speakers highlighted that if their Neo4j logs had been forwarded, they would have retained them even after the database deletion, potentially accelerating their investigation.

1. Prioritize Log Correlation and Context: The incident underscored the importance of correlating different types of logs. Seeing authentication failures in security logs alongside legitimate application activity in query logs would have immediately provided the necessary context to de-escalate the situation. Security tools and platforms should strive to present a holistic view of events, enabling users to easily connect disparate data points and understand the full narrative of an incident.

1. Practice Careful Endpoint Device Handling: In the heat of an incident, the instinct to immediately wipe a potentially compromised device is strong. However, this destroys valuable forensic evidence. The correct procedure is to quarantine the device, isolating it from the network while preserving its state for thorough forensic analysis. Only after comprehensive investigation and data extraction should a device be wiped or reimaged.

1. Proactively Test EDR Solutions: An EDR solution is only as good as its tested functionality and usability. The speakers' experience with a "horrendous" EDR UI highlights that deploying a tool is not enough; it must be tested proactively to ensure it functions as expected and that analysts can effectively use it during an incident. Regular drills, like attempting to detonate an EICAR file on an endpoint, can validate that alerts are routed correctly and that the security team is proficient with the platform.

1. Maintain Security Posture from Day Zero: While many startups are advised to prioritize Product-Market Fit (PMF) over security, the Subimage team argued for the opposite, especially for security companies. Establishing a robust security stack early on, even if it feels like "overkill," provides invaluable customer empathy. Experiencing an incident (even a false one) with their own tools allowed them to understand the pain points of their customers, driving product improvements focused on speed, usability, and comprehensive integrations.

1. Focus on Usability and Speed in Security Products: The incident profoundly influenced Subimage's product development. They realized that in an incident, customers need answers fast. This led them to make their product 10 times faster, ensuring queries load in milliseconds, and to hire a dedicated designer to enhance usability. Security tools, regardless of their underlying power, must be intuitive and quick to navigate, especially for users under pressure.

1. Ensure Comprehensive Integrations: A security tool that doesn't integrate with a critical part of a customer's infrastructure creates blind spots. Subimage committed to supporting "every integration under the sun," recognizing that security teams cannot afford to have gaps in visibility due to vendor limitations or corporate development relationships.

By internalizing these lessons, organizations can build more resilient security programs, foster a culture of preparedness, and develop security products that genuinely address the real-world challenges faced by defenders.

Key Takeaways

• Panic Blinds; Playbooks Guide: Even seasoned security professionals can make critical errors under pressure. Adhering to well-defined incident response playbooks is crucial to maintain discipline and avoid rash decisions that destroy evidence or prolong an incident.
• Centralize and Correlate Logs: Relying on disparate vendor log viewers is insufficient. Implement centralized log forwarding for all sensitive assets and ensure systems can easily correlate security events with operational logs for complete context. Logs must persist independently of the resource.
• Quarantine, Don't Wipe: During an incident, preserve potential forensic evidence by quarantining suspicious devices rather than immediately wiping them. Wiping destroys crucial data needed for investigation.
• Proactively Test Your Tools: Don't assume your security tools work as advertised. Regularly test EDR solutions, log forwarding, and alerting mechanisms to ensure they are functional and your team is proficient in using them before a real incident occurs.
• Customer Empathy Drives Product Excellence: Experiencing a security incident firsthand, even a false alarm, provides invaluable empathy for customers. This direct understanding of their pain points (e.g., needing fast answers, intuitive UIs, comprehensive integrations) should directly inform product development and prioritize user experience.
• Even Experts Make Basic Mistakes: The incident, triggered by a simple case-sensitivity error, highlights that fundamental misconfigurations can lead to significant escalations, even for highly skilled teams. Humility, double-checking, and a structured approach are vital.

About the Speaker(s)

Alex Chantavy is a co-founder of Subimage. With approximately 15 years of experience in the security field, his career began at the NSA. He later transitioned to the private sector, working at Microsoft on the Azure Red Team. Prior to co-founding Subimage, Alex was at Lyft, where he developed the open-source tool Cartography, which later became the foundation of Subimage's product.

Kunaal Sikka is also a co-founder of Subimage. He started his career at Lyft, where he encountered Alex Chantavy and became an early user of the open-source Cartography tool, recognizing its power. During his four years at Lyft, Kunaal gained experience in various areas, including van management, data security, and basic incident response. He then moved to Anthropic, where he was involved in many early-stage operational aspects of the company. Together, Alex and Kunaal leverage their extensive security and startup experience to lead Subimage.