Titans of Scale: Strategies to Scale Security in Expanding Organizations

Titans of Scale: Strategies to Scale Security in Expanding Organizations

Speakers: Unknown

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=IRgpg5bEx4g

Overview

This panel discussion, "Titans of Scale: Strategies to Scale Security in Expanding Organizations," brought together security leaders from prominent technology companies—Netflix, Chime, Rippling, and Snowflake—to share their real-world experiences, solutions, and hard-learned lessons in adapting security practices to rapidly growing enterprises. Moderated by Ariel Shin from Twilio, the session delved into the multifaceted challenges of scaling security as companies accumulate technical debt, expand product features, and face increased exposure to threats. The core premise was that an effective security team must evolve its people, processes, and tools to meet these escalating demands.

The panelists, representing organizations of varying sizes and growth trajectories, offered diverse perspectives on how to strategically approach security at scale. The discussion explored critical areas such as team structure, the development and deployment of security tooling, and the implementation of security programs that can keep pace with engineering velocity. A central theme was the non-linear nature of scaling, where the complexity of security problems increases disproportionately as an organization grows from hundreds to thousands of developers, introducing unforeseen challenges and higher costs for unaddressed issues.

The conversation highlighted the shift from traditional security paradigms to more integrated, product-centric approaches. It emphasized the importance of fostering a collaborative culture with engineering teams, treating security initiatives as internal products, and empowering developers to take ownership of security. The insights provided are invaluable for security professionals grappling with the complexities of maintaining robust security postures within dynamic, expanding technical environments.

Background

▶ Watch: Scaling challenges: complexity increases disproportionately with developer co... (02:50)

The fundamental challenge addressed by the panel is the inherent difficulty of scaling security operations in lockstep with the rapid growth of modern technology companies. As organizations expand, they inevitably accrue technical debt, introduce a proliferation of product features, and consequently increase their exposure to threats. An effective security team cannot remain static; it must adapt and scale its people, processes, and tools to meet these evolving demands.

Jacob Cassie of Snowflake articulated a key insight: the difference in scaling challenges is not linear. While the jump from supporting 10 to 100 developers, or 100 to 1,000, is intuitively understood as significant, the leap from 1,000 to 2,000, or 2,000 to 3,000, introduces a far greater degree of complexity. Problems that were once manageable become exponentially more difficult, and the cost of failing to anticipate these issues rises dramatically with scale. Furthermore, the fate of a security program at scale is often "coupled with the maturity of the overall engineering organization," including their developer practices and the practices of the infrastructure teams. This means security cannot operate in a vacuum but must deeply integrate with and influence the broader engineering culture and infrastructure choices, especially when aiming to reduce choice or enforce opinionated security standards.

Jim Singh of Rippling provided a concrete example of this scaling challenge in communication. Rolling out a new vulnerability management program to a few hundred people at Twilio was straightforward, involving Slack messages and emails. However, scaling the same program to several thousand people proved problematic, as "no one's going to read their emails, no one reads their slacks." This necessitated more intensive strategies like "roadshows" to every business unit, highlighting the breakdown of traditional communication channels at enterprise scale. Mukun of Chime underscored the importance of a collaborative culture with the engineering function, stating that its absence is the "biggest challenge." Tools, while helpful, are secondary to maintaining open lines of communication and fostering a collaborative nature with engineering teams across all levels.

Julia Connect of Netflix summarized these points by noting that "people are still people at the end of the day, that's very hard to scale." Communications and relationships are equally challenging to scale. She also highlighted that scaling often means "trading a set of problems for a different set of problems," such as transitioning from traditional security operations to "engineering ops," which requires a different skill set and team composition. This background sets the stage for understanding why traditional security approaches often fail at scale and why innovative strategies are essential.

Key Findings

▶ Watch: Failed program: secure defaults and static analysis rules at scale due to spr... (11:00)

The panel discussion yielded several critical findings regarding the strategies required to scale security effectively in expanding organizations:

• Non-Linear Scaling Complexity: The complexity of security challenges increases disproportionately with organizational growth, particularly beyond 1,000 developers. Unanticipated problems at this scale incur significantly higher costs.
• Interdependence with Engineering Maturity: The success of security scaling is intrinsically linked to the maturity of the broader engineering organization, including its developer practices and infrastructure teams. Reducing choice and enforcing security opinions requires deep integration.
• Communication Breakdown at Scale: Traditional communication methods (email, Slack) are ineffective for rolling out security programs to thousands of engineers. Direct engagement, such as roadshows, becomes essential.
• Collaborative Culture is Paramount: A strong, collaborative relationship with engineering teams is more critical for scaling security than any specific tool or process. Without it, even the best technical solutions will struggle to gain adoption.
• Treat Security Initiatives as Products: Security teams must adopt a product mindset, treating their programs and tools as internal products. This involves understanding user needs (developers), effective communication, and robust rollout strategies, akin to production services.
• Accountability Resides with Risk Owners: Effective scaling requires shifting security accountability to the engineering teams (risk owners). The security team's role evolves to providing support, frameworks, and oversight, rather than being solely responsible for fixing every issue.
• Hybrid Security Models are Optimal: A blend of centralized security teams (for building platform-level solutions and ensuring consistency) and embedded security engineers (for deep contextual understanding and integration within business units) offers the most balanced and effective approach.
• Hiring Software Engineers for Security Roles: There's a growing trend towards hiring security engineers with strong software engineering backgrounds. This enables the security team to build scalable tooling and "paved paths," recognizing that "software will eat security."
• Strategic Management of Interpersonal Dynamics: Addressing "toxic" team members or leaders involves assuming good intent, understanding their viewpoints, aligning goals, and, when necessary, strategic escalation. Protecting the security team from burnout caused by unproductive conflict is a key leadership responsibility.
• Security Teams as Potential Bottlenecks: Security teams themselves can become bottlenecks, particularly if they adopt a "no" culture or lack empathy for developer workflows. A shift towards providing solutions and enabling engineering is crucial.
• Developer-Centric Tooling: Successful security tools are those that integrate seamlessly into developer workflows (e.g., GitHub) and are perceived as "developer tools" rather than separate "security tools." They should aggregate information and provide clear, actionable steps.

Technical Deep Dive

▶ Watch: Hiring: more software engineers. Security Architects as 'risk product owners'... (20:00)

The panel offered several detailed insights into technical strategies and tooling that have proven effective, or conversely, challenging, when scaling security.

Democratized Vulnerability Management (Jim Singh at Twilio/Rippling):

Jim Singh highlighted a democratized vulnerability management program as his most impactful initiative. The core principle is to shift ownership of vulnerabilities to the risk owners, which are the engineering teams themselves. The security team's role transitions from chasing individual engineers to providing oversight and aggregate reporting.

Key technical and process elements include:

• Automated SLA Extensions: For low-priority vulnerabilities (P4/P5), SLA extensions are automatically approved upon request, acknowledging that "no one really cares about low information bugs."
• Tiered Approval for Higher Priorities: As vulnerability severity increases, the approval for SLA extensions escalates:
• P3: Approved by Directors.
• P2: Approved by VPs.
• P1: Approved by C-suite.
• Impact on Accountability: This tiered approval process significantly increases visibility of security technical debt at higher organizational levels. VPs, for instance, become directly aware of P2 vulnerabilities under their purview. Jim recounted a scenario where a VP, seeing a P2 extension request for the second time, demanded a concrete plan and timeline from the team. This direct executive involvement led to the vulnerability being fixed in 1.5 weeks, significantly faster than the initially estimated 6 weeks, demonstrating how executive accountability can "move the needle."

Authorization Solutions and the Challenge of Generalization (Mukun at Chime):

Mukun shared an experience with building a platform-level authorization solution at Chime. While highly successful for service-to-service authorization, the attempt to expand this same solution to member authorization and employee authorization failed. The technical challenge lay in the fundamental differences between these authorization domains. Mukun noted, "member authorization is completely different from how services-to-service authorization works versus employee authorization." The mistake was trying to force a single solution across disparate use cases without adequately understanding the specific problems and requirements of each. This highlights the need for pragmatism and deep listening to developer needs, even when a platform approach seems initially appealing.

Struggles with Secure Defaults and Sprawling Choice (Jacob Cassie at Snowflake):

Jacob Cassie admitted that Snowflake initially "never got hyped on secure defaults" and struggled to implement them at scale. They faced difficulties writing static analysis rules that were meaningful. Two primary reasons were cited:

1. Domain Specificity: "There's no database OAS top 10," indicating a lack of standardized security guidance for their specific domain (data).
2. Sprawling Choice: In its early days, Snowflake allowed "a lot of choice" in engineering practices. Jacob explained that "in the face of sprawling choice, it is very difficult to express invariants or opinions of any kind."

This illustrates how a lack of standardization and excessive flexibility in engineering can directly impede the implementation of foundational security controls. However, Jacob noted that as the organization matured and recognized the "business imperative" to get its arms around this sprawl, they began to ship successful secure defaults, including a "high-performance secrets engine" (another Vault implementation).

Internal Application Context System (Julia Connect at Netflix):

Julia Connect described an internal system at Netflix that "understands a whole lot about all of our applications." This system aggregates facts and context about their thousands of microservices. Its primary purpose is to enable the security team to make highly informed decisions about when to "interrupt" developers.

• Risk-Based Interruption: The system helps determine with "reasonably good idea that what we were telling them to do represents risk to Netflix and means that they should take action."
• Developer Productivity: Netflix highly values developer productivity, so interruptions are reserved for situations where there is high confidence in the necessity of a behavioral change or action.
• Contextual Trade-offs: With thousands of microservices, this tool allows the security team to make informed trade-offs and avoid overwhelming developers with an unmanageable volume of tasks (e.g., "cutting 14,000 Jira tickets"). This system is crucial for targeted, high-impact security interventions.

The "Software Eats Security" Paradigm and Hiring (Jacob Cassie, Jim Singh, Julia Connect, Mukun):

A significant technical and organizational shift discussed was the increasing need for security professionals with strong software engineering skills. Jacob Cassie posited, "will software eat security and the answer is yes." He argued that security teams need to "respect software engineering for what it is and not ask security engineers to do that and put them in a tough spot where you're expecting them to know software engineering and they may not."

He proposed a model of Security Architects (who are "risk product owners" focused on "risk decomposition" and defining "control architecture") paired with Software Engineers (who build the solutions). Jim Singh, Julia Connect, and Mukun echoed this, emphasizing hiring security engineers with software backgrounds to build "tooling," "scaling our frameworks," and creating a "paved path" to keep up with engineering velocity. Mukun specifically mentioned building "developer tools" rather than "security tools," as developers "want to find those issues where they are," such as on GitHub. He cited "Monocle," an internal tool that aggregates information and presents service owners with "three things you need to do" that "will take you 5 minutes to fix it."

Distributed Security without Embedded Teams (Jacob Cassie at Snowflake):

Jacob Cassie described a unique approach at Snowflake where they "hired zero people to embed on any teams and we have zero volunteers." Instead, they focused on defining "what every developer should do on security," making it "a little bit less than you might think." The model relies on:

• Developer Ownership: Every developer owns their own security.
• Peer Review: Senior developers peer-review security aspects of code.
• Lightweight Process: The process is designed to be very lightweight for partners, preventing it from becoming a "cognitive offloading activity."
• "Call Us" Model: The security team optimizes for situations where developers "don't know the answer, call us," making it very obvious how to engage.

These technical deep dives illustrate a clear trend towards automation, platform-level solutions, developer empowerment, and a strategic integration of security into the core engineering workflow, often requiring a re-evaluation of traditional security team structures and skill sets.

Demo / Proof of Concept

▶ Watch: Managing toxic ICs: protect the team, shut it down. Escalate to leaders to ho... (29:00)

As a panel discussion, this session focused on sharing experiences, strategies, and lessons learned rather than demonstrating specific tools or proof-of-concept implementations. The panelists described various programs and internal systems, such as Rippling's democratized vulnerability management, Chime's authorization platform, Netflix's application context system, and Snowflake's approach to distributed security, but no live demonstrations or technical walkthroughs were part of the presentation.

Defensive Implications

▶ Watch: Focus on platform-level solutions, developer tools (not 'security tools'). De... (41:00)

The insights from the "Titans of Scale" panel offer several crucial defensive implications for organizations aiming to strengthen their security posture as they grow:

• Adopt a Hybrid Security Model: Defenders should move towards a hybrid security model that combines a centralized security team responsible for building scalable platforms, defining consistent policies, and providing expertise, with embedded security engineers who deeply understand specific business units and product contexts. This balances consistency with contextual relevance, as advocated by Julia Connect and Jacob Cassie.
• Empower Engineering for Security Ownership: Implement democratized vulnerability management programs where engineering teams are the primary risk owners and are accountable for fixing their own security issues. The security team's role shifts to providing the frameworks, tooling, and oversight necessary to enable this ownership, as exemplified by Jim Singh's program at Rippling. This includes tiered SLA approvals by senior leadership to ensure high-priority issues receive executive attention.
• Invest in Developer-Centric Security Tooling: Prioritize building or integrating security tools that seamlessly fit into existing developer workflows (e.g., GitHub, IDEs) rather than requiring developers to learn new, separate "security tools." These tools should aggregate information, provide clear context, and offer actionable, low-effort steps for remediation, as described by Mukun and Julia Connect. The goal is to create a "paved path" for secure development.
• Foster a Collaborative Security Culture: Actively cultivate strong relationships and open communication channels with engineering teams. Security initiatives should be presented as collaborative efforts to achieve shared business goals, rather than as mandates. This involves understanding engineering's challenges and treating security programs as internal products with user empathy, as highlighted by Mukun and Ariel Shin.
• Strategic Hiring for Software Engineering Skills: Recruit security professionals with strong software engineering backgrounds who can build scalable, automated security solutions. Complement these roles with security architects who specialize in risk decomposition and control architecture, effectively pairing builders with strategic thinkers, as suggested by Jacob Cassie.
• Proactive Anticipation of Scaling Challenges: Recognize that scaling introduces new, often non-linear, security problems. Security leaders must proactively anticipate these challenges and adapt team structures, processes, and tools accordingly, rather than reacting to crises. This includes preparing for the shift from traditional security operations to "engineering ops."
• Reduce Choice to Enforce Secure Defaults: Work with engineering leadership to strategically reduce sprawling choices in development frameworks and infrastructure. This enables the implementation of effective secure defaults and invariants, making it easier to enforce security policies across a large codebase, as Jacob Cassie learned at Snowflake.
• Manage Interpersonal Dynamics and "Toxicity": Develop strategies for addressing resistance or "toxic" behavior from team members or leaders. This involves assuming good intent, seeking to understand viewpoints, aligning goals, and, when necessary, escalating issues through appropriate channels. Crucially, security leadership must protect their teams from burnout caused by unproductive conflict.
• Avoid Becoming a Bottleneck: Security teams must guard against becoming a bottleneck by adopting a "no" culture. Instead, focus on enabling engineering, providing solutions, and empowering developers to make secure decisions within defined guardrails. This requires a shift from gatekeeping to guidance and support.

By implementing these defensive strategies, organizations can build resilient security programs that not only keep pace with rapid growth but also integrate security as a fundamental, enabling aspect of their engineering culture.

Key Takeaways

• Scaling Security is Non-Linear: The complexity and cost of security challenges increase disproportionately as organizations grow, especially beyond 1,000 developers, requiring continuous adaptation and anticipation of new problems.
• Culture and Communication are Paramount: A collaborative culture with engineering and effective communication strategies (beyond traditional emails/Slack) are more critical for scaling security than any specific tool or process.
• Treat Security as a Product: Security programs and tools must be developed and rolled out with a product mindset, focusing on user empathy (developers), clear value propositions, and robust communication to ensure adoption and effectiveness.
• Hybrid Models and Distributed Ownership: The most effective security models combine centralized platform teams for consistency and scale with embedded expertise for contextual understanding, while empowering engineering teams to own their security risks and accountability.
• Software Engineering Skills are Essential: Security teams need to increasingly hire professionals with strong software engineering backgrounds to build automated, scalable solutions and "paved paths," complementing them with security architects focused on risk decomposition.
• Executive Visibility Drives Accountability: Mechanisms that bring security technical debt and risks to the attention of senior leadership (e.g., tiered SLA approvals) can significantly accelerate remediation and foster greater accountability across the organization.

About the Speaker(s)

The panel featured a diverse group of security leaders from prominent technology companies:

• Ariel Shin: The moderator of the panel, Ariel is a Product Security Engineering Manager at Twilio. She set the stage for the discussion and guided the conversation among the panelists.
• Julia Connect: Julia leads the Security Platforms Engineering team at Netflix. Her insights focused on scaling security platforms and managing the human element in security rollouts within a large, dynamic organization.
• Mukun: Mukun leads the Product Security team at Chime. He shared experiences related to building authorization solutions and the importance of a collaborative culture with engineering.
• Jim Singh: A longtime developer and security engineer, Jim runs the product security team at Rippling. He highlighted the success of democratized vulnerability management programs and the challenges of communication at scale.
• Jacob Cassie: Jacob has been leading the Snowflake security program for the last five years and has a long history in technology, including being on IRC since 1992. He provided perspectives on secure defaults, managing sprawling engineering choices, and the strategic hiring of software engineers for security.