The road to developers' hearts

The road to developers' hearts

Speakers: Sing Ambikapathi

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=nk58pb-UdSM

Overview

This talk, "The road to developers' hearts," delivered by Sing Ambikapathi at BSidesSF 2024, addresses the critical challenge of fostering productive and peaceful collaboration between security practitioners and software developers. Ambikapathi, drawing from his extensive experience as both a software developer and a security professional, highlights the inherent differences in perspective between these two crucial teams and how these differences, compounded by project constraints like finite time and budget, often lead to friction and escalated issues. The core premise of his presentation is that for long-term success in safeguarding businesses and customers, security and software teams must learn to work together as a unified entity.

Ambikapathi's presentation is not merely a call for better communication; it's a practical guide offering actionable strategies to reduce friction and build a collaborative security culture. He delves into various facets, from the foundational importance of building trust and ensuring easy access to security expertise, to leveraging technology through automation, providing realistic recommendations, and implementing consistent risk assessment frameworks. The talk underscores that security cannot be a solitary endeavor; it requires allies across the organization, particularly within development teams, to effectively protect data and customers in the long run.

The significance of this topic cannot be overstated in today's rapidly evolving threat landscape. As software development cycles accelerate and applications become increasingly complex, embedding security effectively from inception is paramount. Ambikapathi's insights are particularly valuable because they come from someone who has walked in both shoes, understanding the pressures and priorities of both development and security. His strategies aim to transform security from a perceived bottleneck or an "afterthought" into an integrated, value-adding component of the software development lifecycle, ultimately leading to more secure products and a more resilient organization.

Background

▶ Watch: Speaker's background: 14+ years as software developer, then security. (0:40)

The genesis of the challenges discussed in "The road to developers' hearts" lies in the fundamental divergence of perspectives between security practitioners and software developers. Sing Ambikapathi, with over 14 years as a software developer and several years in security-focused roles, has observed firsthand how these differing viewpoints manifest in the approach to problem-solving and even in the identification of what constitutes a "problem" in the first place. Developers are often driven by the need for rapid iteration, feature delivery, and meeting product roadmaps, while security teams are inherently focused on risk mitigation, compliance, and preventing vulnerabilities.

This inherent difference is exacerbated by common organizational constraints: finite time and budget. When security requirements are introduced late in the development cycle, or when security teams operate in isolation, these constraints transform differing perspectives into significant friction. Ambikapathi notes that this friction can escalate, leading to unproductive conflicts rather than collaborative solutions. Historically, security has often been treated as an "afterthought," a gate that projects must pass through at the very end, rather than an integral part of the design and development process. This reactive approach inevitably leads to costly rework, delays, and a strained relationship between teams.

The problem is further compounded by the nature of engagement. Security teams often interact with development teams on short-term, project-specific engagements, making it difficult to establish the deep, consistent trust necessary for effective collaboration. Without this trust, developers may be hesitant to proactively share potential issues or seek early consultation, fearing judgment or delays. Even the industry-wide push to "shift left"—integrating security earlier in the development lifecycle—can inadvertently create new forms of friction if not implemented thoughtfully. If "shifting left" means imposing rigid, bureaucratic processes on developers who need to move fast and iterate quickly in early stages, it can hinder velocity and lead to resistance, effectively undermining the very goal of early security integration. Ambikapathi's talk aims to provide a roadmap for overcoming these historical and systemic challenges, transforming the relationship from adversarial to allied.

Key Findings

▶ Watch: Challenge: building trust in short-term security engagements. (2:00)

Sing Ambikapathi's talk distills several critical findings and strategies for fostering effective collaboration between security and development teams, moving beyond mere compliance to genuine partnership.

First and foremost, building trust is paramount. Ambikapathi emphasizes that while seemingly obvious, consistent trust-building is challenging, especially given the often short-term nature of security engagements with development teams. He illustrates its importance with a personal anecdote: early in his career, he was hesitant to share potential issues with security professionals, but once trust was established, he began proactively seeking their advocacy for security features in his product roadmap. This trust transforms security from a gatekeeper into an ally, enabling developers to bring "interesting stuff that you didn't even know existed" to the security team, creating valuable reconnaissance opportunities.

Secondly, easy access to security teams significantly reduces development velocity bottlenecks. The speaker recalls a time when security consultations took months. Modern approaches, such as periodic office hours, review forums, or asynchronous communication mechanisms, are crucial. These avenues ensure that developers can get timely answers and guidance, allowing them to build faster without compromising security, and making security teams feel like an integrated part of the development process.

Thirdly, engaging early, or "shifting left," is beneficial but requires customization. While early detection of issues reduces cost, rigidly applying security processes too early can disrupt rapid iteration in product development. The key is to work with software teams to customize processes, ensuring they can move fast and scrappy while remaining secure. Ambikapathi provides a stark example of a team forced to fix a "bunch of issues" in December, just before vacation, due to a delayed annual audit—a situation easily avoidable with advanced notice and early engagement.

Fourth, automation is critical for efficiency and consistency. Repetitive tasks like automated code scanning or checking for software misconfigurations are chores that nobody likes. Automating these processes frees up human resources and ensures consistent application of security checks. A notable example involved a database programming language for which no widely available static code analysis tool existed. After procuring a tool, it was "borderline unusable" due to false positives, requiring significant investment in fine-tuning custom rules with language-specialized developers. This highlights that automation is not a set-and-forget solution; it requires ongoing collaboration and refinement to be effective.

Fifth, security recommendations must be realistic and specific. Ambikapathi recounts an instance where an infrastructure security person provided "deep-down networking solutions" that were not prototyped or integrated into the existing ecosystem, leaving the application team with "more questions than answers." Vague "it depends" answers are also unhelpful. Security professionals don't need to have all the solutions but should work with developers to understand options and find the right fit for their context, potentially through lightweight tabletop exercises before project initiation.

Finally, consistent risk assessment and ruthless prioritization are essential. In large organizations with multiple security teams, development teams can be overwhelmed by a deluge of issues with uncoordinated priorities. Ambikapathi stresses the importance of avoiding unnecessary escalation, having a consistent risk assessment framework, and defining clear mitigation or success criteria for escalated issues. Prioritizing issues based on impact, both within security teams and across the organization, prevents development teams from being resource-constrained and forced to make their own, potentially misaligned, prioritization decisions.

Technical Deep Dive

▶ Watch: Automation: leveraging tools like static code analysis for efficiency. (8:00)

While Sing Ambikapathi's talk primarily focuses on the human and process aspects of security collaboration, it touches upon several technical areas where these principles are applied, particularly concerning automation, tool implementation, and the nature of security recommendations.

A significant technical finding revolves around automation. Ambikapathi emphasizes automating repetitive security tasks, which can include "running automated score scanning tools" and "checking for software misconfigurations." These refer to common security tools and practices:

• Static Application Security Testing (SAST): Tools that analyze source code, bytecode, or binary code to identify security vulnerabilities without executing the program.
• Dynamic Application Security Testing (DAST): Tools that test a running application to find vulnerabilities.
• Software Composition Analysis (SCA): Tools that identify open-source components in an application and check for known vulnerabilities (e.g., CVEs).
• Infrastructure as Code (IaC) Security Scanners: Tools that analyze configuration files (e.g., Terraform, CloudFormation) for misconfigurations before deployment.

Ambikapathi provides a specific, detailed example of the challenges and benefits of automation: his team was heavily using a database programming language for which "no widely available static code tool" existed. This necessitated "a lot of manual reviews," which were "error prone." The team eventually "procured the tool" but found it "borderline unusable because of the false positives." This anecdote highlights a critical technical hurdle in security automation: the accuracy and usability of security tools. False positives not only waste developer time but also erode trust in the security team and the tools themselves. The solution involved "a lot of time working with developers specialized in the language to fine tune the custom rules." This process of custom rule development and fine-tuning is a highly technical endeavor, requiring deep understanding of both the programming language's nuances and the security tool's configuration capabilities. It often involves:

• Writing custom regex patterns or Abstract Syntax Tree (AST) queries: To identify specific insecure coding patterns unique to the language or application context.
• Developing suppression rules: To ignore known false positives or expected behaviors.
• Integrating with existing CI/CD pipelines: To ensure automated scans run consistently and provide feedback directly to developers. Ambikapathi mentions "integrating with existing CI/CD integration" for "mandatory security test cases," which could include unit tests, integration tests, or specific security tests (e.g., for authentication, authorization).

Another technical aspect discussed is the nature of security recommendations. Ambikapathi cautions against providing "deep down networking solutions" that are "not prototyped" or don't fit into the "existing ecosystem." This refers to complex infrastructure security measures that might involve:

• Advanced network segmentation: Dividing a network into smaller, isolated segments to limit lateral movement in case of a breach.
• Microsegmentation: Applying granular security policies to individual workloads.
• Complex firewall rules and access control lists (ACLs): Detailed configurations to control traffic flow.
• Intrusion Detection/Prevention Systems (IDS/IPS) deployment: Implementing and configuring systems to monitor and block malicious network activity.

Such solutions, while potentially ideal in the long term, can be technically challenging and time-consuming for application developers to implement, especially if they require significant changes to existing infrastructure or architecture. The speaker advocates for "realistic recommendations" that developers can integrate without undertaking a separate, large-scale security initiative.

Finally, the concept of lightweight tabletop exercises before project initiation, where teams "talk through all the foreseeable scenarios and how to address them," has technical implications. These exercises can involve:

• Threat modeling: Identifying potential threats and vulnerabilities in a system design.
• Attack surface analysis: Mapping out all possible entry points for an attacker.
• Reviewing architectural diagrams: Identifying security weaknesses in the proposed system architecture.

By doing this in advance, security requirements can be "embedded in the process" from the start, influencing technical design decisions and ensuring security is considered proactively rather than reactively. This proactive approach helps establish a "Baseline for software teams to apply some of the security things when they start building the product."

In essence, while the talk's focus is on collaboration, the underlying technical message is clear: security tools and practices must be carefully selected, customized, and integrated into the development workflow in a way that is both effective and developer-friendly. This requires technical expertise from both security and development teams to bridge the gap between theoretical security ideals and practical implementation realities.

Demo / Proof of Concept

▶ Watch: Customizing tools: fine-tuning security tools to reduce false positives and m... (9:00)

During the presentation, Sing Ambikapathi mentioned that he had prepared several GIFs for his final presentation that were not displaying, which he noted was "on me though." However, beyond these missing visual aids, the talk did not include a live demonstration or a specific proof of concept of any security tool, technique, or collaborative platform. The speaker's focus was on sharing practical strategies, anecdotes, and lessons learned from his extensive experience in both software development and security roles, rather than showcasing a particular technical implementation. The content was delivered through confident, analytical prose, supported by real-world examples of successful and unsuccessful collaborative efforts.

Defensive Implications

▶ Watch: Prioritization: consistent risk assessment and avoiding multiple security tea... (12:00)

The insights shared by Sing Ambikapathi offer a robust framework for strengthening an organization's defensive posture by fostering a more integrated and proactive security culture. The defensive implications span both strategic and tactical levels, primarily targeting how security teams can empower developers to become frontline defenders.

For Security Teams:

1. Prioritize Trust and Accessibility: The most significant defensive implication is the need for security teams to actively cultivate trust with development teams. This means being approachable, responsive, and seen as enablers rather than blockers. Establishing "periodic office hours," "review forums," or "asynchronous communication mechanisms" ensures developers have easy access to security expertise. This proactive engagement encourages developers to seek security consultation early, preventing vulnerabilities from being baked into the code and reducing the cost of remediation. When developers trust security, they become an extension of the security team, acting as early warning systems and advocates.

1. Provide Actionable and Context-Aware Guidance: Security recommendations must be "realistic" and "specific." Generic advice or "deep-down networking solutions" that don't fit the existing ecosystem are counterproductive. Security professionals must understand the development team's context, including their frameworks, existing architecture, and constraints. This means working with developers to find solutions, rather than dictating them. Conducting "lightweight tabletop exercises" before project initiation can help embed security considerations into the design phase, leading to more secure architectures from the outset.

1. Automate and Refine Security Tools: Automating repetitive tasks like "automated score scanning tools" and "checking for software misconfigurations" is crucial. However, the defensive value of these tools is directly tied to their accuracy. Security teams must invest the effort to "fine tune the custom rules" of tools, especially for specialized languages, to reduce "false positives." A tool that constantly flags non-issues will be ignored, negating its defensive value. Integrating "mandatory security test cases" into CI/CD pipelines ensures consistent security checks are performed automatically, catching issues early and consistently.

1. Implement Consistent Risk Assessment and Prioritization: In large organizations, development teams can be overwhelmed by uncoordinated security findings from multiple teams. A "consistent risk assessment framework" and "ruthless prioritize issues" are essential. This prevents "unnecessary escalation" and ensures that the most impactful vulnerabilities are addressed first, based on a clear understanding of business risk. Defining "mitigation or success criteria" for escalated issues provides developers with a clear path to resolution, avoiding ambiguity and wasted effort.

1. Cultivate Security Champions and Mentors: Security teams "can't possibly be in all the rooms" where decisions are made. Therefore, "force multiply through people you mentor and coach." By identifying "allies within the team" and mentoring developers to become "security advocates," security teams can embed security consciousness directly into development workflows. These champions can ask critical security questions during planning meetings and reinforce security culture, ensuring constant vigilance.

For Development Teams:

1. Embrace Proactive Engagement: Developers should view security teams as valuable resources and engage them early and often. This means not waiting for an audit or a security gate, but actively seeking input during design and development phases. Sharing potential issues or architectural decisions with security early can prevent costly rework later.

1. Provide Feedback on Security Tools and Processes: Developers are the primary users of security tools integrated into their workflows. Their feedback on tool usability, false positive rates, and integration challenges is invaluable. Collaborating with security teams to "fine tune the custom rules" of static analysis tools, for example, directly improves the effectiveness of these defensive measures.

1. Integrate Security into Daily Workflows: By participating in "lightweight tabletop exercise[s]" and embedding security considerations into their "process," developers can make security a natural part of their daily work, rather than an external imposition. This includes writing security-focused unit tests and ensuring security requirements are part of their definition of "done."

1. Become Internal Security Advocates: Developers who understand security principles can become powerful advocates within their own teams. By asking questions like, "have you considered time for doing security things in this?" or "this doesn't sound right, can you go talk to a security person?", they reinforce a security-first mindset and help to "change their security culture over the period of 12 to 18 months."

In summary, Ambikapathi's talk champions a shift from a reactive, adversarial security model to a proactive, collaborative one. By empowering developers, providing them with the right tools and guidance, and building a foundation of trust, organizations can significantly enhance their defensive capabilities, making security an intrinsic part of product development rather than an external burden.

Key Takeaways

• Trust is the Foundation of Collaboration: Building consistent trust between security and development teams is paramount. It encourages developers to proactively seek security input, share potential issues, and view security professionals as allies rather than adversaries, ultimately leading to better security outcomes.
• Accessibility and Early Engagement are Crucial: Security teams must be easily accessible through various channels (office hours, asynchronous communication) to provide timely guidance. Engaging early in the development lifecycle ("shifting left") is vital for cost reduction and issue prevention, but processes must be customized to avoid disrupting developer velocity and iteration.
• Automate and Refine Security Tools Thoughtfully: Leverage automation for repetitive security tasks like code scanning and misconfiguration checks. However, simply deploying tools is insufficient; significant investment is required to fine-tune custom rules and reduce false positives, ensuring tools are usable and trusted by developers.
• Provide Realistic, Specific, and Actionable Guidance: Security recommendations must be practical, context-aware, and integrate seamlessly with existing development ecosystems. Vague advice or overly complex solutions that require separate initiatives are counterproductive. Collaborating with developers to understand their context and options is key.
• Prioritize Ruthlessly and Consistently: In organizations with multiple security teams, a consistent risk assessment framework and ruthless prioritization of issues based on impact are essential. This prevents overwhelming development teams with uncoordinated demands and ensures that critical vulnerabilities are addressed efficiently with clear mitigation criteria.
• Foster Cultural Change Through Advocacy and Mentorship: Transforming an organization's security culture takes time and constant reinforcement. Security teams should identify and mentor "allies" within development teams to become security advocates, asking pertinent questions and reinforcing security principles, thereby force-multiplying security efforts across the organization.

About the Speaker(s)

Sing Ambikapathi is a seasoned professional with a unique background that bridges the often-disparate worlds of software development and security. He has accumulated over 14 years of experience as a software developer, providing him with a deep understanding of the intricacies of product creation, development velocity, and the challenges faced by engineering teams. Complementing this, Ambikapathi has also dedicated several years to security-focused roles, specializing in security and compliance. This dual perspective is central to his insights, allowing him to articulate the differing viewpoints of both security practitioners and software developers. His personal journey, including being mentored into the security field and becoming a security advocate, underscores his commitment to fostering collaboration and building a strong security culture.