CISO Series Live Podcast Recording

CISO Series Live Podcast Recording

Speakers: David Spark, Producer, CISO Series; Mike Johnson, CISO, Rivian; Sarah Madden, CISO, Convera

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=GzY0hsDy4y8

Overview

This article delves into a live podcast recording of the CISO Series, captured at BSides San Francisco 2026. Hosted by David Spark, the session featured veteran CISOs Mike Johnson of Rivian and Sarah Madden of Convera, engaging in a candid discussion about the multifaceted challenges and evolving landscape of cybersecurity leadership. The talk served as a practical guide for security professionals, covering critical areas such as strategic vendor management, the nuanced integration of artificial intelligence into security operations, the efficacy of incident response planning, and the broader professional development within the cybersecurity community.

The discussion moved beyond theoretical concepts, offering actionable insights derived from the speakers' extensive real-world experience. It highlighted the importance of learning from mistakes, the necessity of critical thinking when adopting new technologies like AI, and the continuous need to adapt security strategies to dynamic business environments. This session is particularly relevant for current and aspiring CISOs, security engineers, and anyone involved in strategic cybersecurity decision-making, providing a grounded perspective on navigating the complexities of modern digital defense.

Background

▶ Watch: Introduction and live podcast setup (0:00)

The contemporary cybersecurity landscape is characterized by relentless technological innovation, an ever-expanding threat surface, and intense pressure on security leaders to balance business enablement with robust protection. This talk addresses several persistent and emerging problems that CISOs confront daily. Historically, vendor selection has been a minefield, often leading to costly, ineffective solutions due. The discussion acknowledges the common trap of signing multi-year contracts based on impressive demos, only to face product obsolescence or integration issues later. This problem is exacerbated by the rapid evolution of technology and the competitive vendor market, where marketing often outpaces actual product utility.

The advent of Generative AI (Gen AI) introduces a new layer of complexity. While promising significant productivity gains, AI also presents unique security and operational challenges. The speakers highlight the "confidence trap," where AI's authoritative output can mislead users into accepting flawed information without critical validation. This issue is particularly acute in security, where false positives are already a known challenge. Furthermore, the proliferation of Shadow AI – AI tools adopted outside of official IT channels – creates new visibility and governance gaps, mirroring past struggles with Shadow IT. The talk also touches on the critical human element, emphasizing that even the most advanced tools are ineffective without skilled personnel and resilient processes, particularly in high-stress scenarios like incident response. The discussion implicitly draws on prior work in risk management, vendor governance, and human factors in security, applying these established principles to the rapidly changing technological context.

Key Findings

▶ Watch: CISO Series podcast officially begins (2:30)

The CISO Series live recording yielded several critical findings and practical insights for cybersecurity professionals:

1. Strategic Vendor Management is Paramount: The speakers strongly advocated against signing initial multi-year contracts with vendors, especially for new solutions. Mike Johnson emphasized starting with one-year deals to evaluate performance and fit, a lesson learned from past mistakes where vendors were acquired, and product development ceased. Sarah Madden reinforced this, detailing how she leverages renewal periods to negotiate one or two-year licenses and applies pressure on vendors to maintain software stability and quality releases, pushing back against a perceived "new normal" of low-quality software. The importance of peer networks and insights from entry-level engineers was also highlighted as a superior method for building a vendor shortlist compared to relying solely on market research firms like Gartner.

1. AI Requires Critical Trust and Human Oversight: While acknowledging the significant AI productivity gains (74% of organizations according to a Gartner stat shared by Howard Holton of Gigome), the CISOs stressed that only 11% see clear ROI. The core finding here is that AI output, despite its polished formatting and authoritative tone, must not be blindly trusted. Sarah Madden articulated her team's philosophy: "We don't trust it and we use AI every day." This involves rigorous questioning of output, identifying false positives, and maintaining human-in-the-loop controls until AI accuracy improves. The discussion also revealed a rapid shift in the market, with CISOs already phasing out traditional vendors where generic LLMs can provide undifferentiated functionality, emphasizing the need for vendors to highlight unique value propositions beyond basic AI capabilities.

1. Agentic AI is Evolving Rapidly, Demanding New Governance: The conversation noted the significant evolution of Agentic AI from standalone agents to complex systems where "agents are controlling other agents," often with a master control agent overseeing smaller, specialized ones. This shift necessitates advanced AI governance strategies, particularly concerning Shadow AI across the tech stack, including chatbots, agents, MCP server connections, and AI embedded in supply chains of other SaaS tools. Tools like Nudge Security and Quiller AI were mentioned as essential for discovering and managing these risks, moving beyond mere monitoring to real-time decision enforcement.

1. Realistic Tabletop Exercises are Essential for Revealing "Fault Lines": The speakers agreed with Joshua Copelan's (Crescendo) assertion that traditional tabletop exercises often fail to simulate real-world incident response because they lack "stakes." The key finding is that a breach often exposes failures in authority and escalation politics rather than just technical controls. Sarah Madden's quarterly tabletops, where she intentionally holds back her input to let her team "fail forward," demonstrated the value of allowing mistakes in a controlled environment. Mike Johnson further suggested bringing in outside firms to introduce unfamiliarity and inject a greater sense of urgency, thereby exposing genuine weaknesses in decision-making and cross-functional collaboration, especially between security engineers, identity management, and audit/compliance teams.

1. Human Capital and Empowerment are Critical Vulnerabilities and Assets: The "What's Worse" game highlighted that a security program "held together by a few very exhausted security heroes" is worse than a poorly managed environment with unmanaged service accounts and API tokens. This finding underscores the profound vulnerability of over-reliance on individual "heroes" and the potential for collapse if key personnel leave. Conversely, the discussion on AI demonstrated its power to empower non-technical roles, with compliance teams using AI bots to automate audit responses and displace vendor solutions. This signifies that investing in empowering all team members with AI, even those who don't traditionally write code, can turn human capital into a significant asset.

Technical Deep Dive

▶ Watch: Guest CISO Sarah Madden introduced (3:45)

The CISO Series podcast, while strategic in nature, touched upon several critical technical domains and their implications for security leadership. A significant portion of the discussion revolved around the technical challenges of vendor selection and management. Mike Johnson and Sarah Madden underscored the practical difficulties of integrating and maintaining SaaS products. The rapid release cycles of these products, often leading to instability, were a key concern. Sarah Madden specifically challenged vendors on "low-quality releases," noting a shift where the traditional N-1 release strategy (avoiding the latest version for stability) is no longer sufficient, with some vendors now recommending waiting "months if not quarters" for a stable "preferred release." This highlights a fundamental breakdown in software development lifecycle (SDLC) quality assurance that directly impacts operational security.

The most technically rich discussion centered on Artificial Intelligence (AI). The speakers addressed the inherent complexities of Generative AI (Gen AI) and Large Language Models (LLMs). The "confidence trap" of AI output, where polished formatting and authoritative tone can mask inaccuracies, is a pervasive technical challenge. This directly relates to the concept of false positives in security tools, a long-standing issue that AI exacerbates due to its persuasive presentation. The solution advocated for is human-in-the-loop control, a technical and procedural mechanism where human experts validate AI-generated insights or actions before implementation. This is particularly crucial in SecOps workflows, where AI-powered IDE tools are used for code analysis, providing contextualized results that still require human review to filter out dramatic or inaccurate findings.

The concept of Shadow AI was introduced as a significant technical risk. Nudge Security, a sponsor, was highlighted for its ability to discover shadow AI across an organization's tech stack, including chatbots, agents, MCP server connections, and AI embedded within the supply chain of other SaaS tools. This reflects a technical challenge similar to Shadow IT, where unsanctioned tool adoption creates visibility gaps, unmanaged data flows, and potential compliance issues. The discussion on Agentic AI further elaborated on its technical evolution, moving from standalone agents to more complex architectures where "agents controlling other agents" and master control agents are emerging. This necessitates advanced AI governance and real-time security controls. Quiller AI, another sponsor, was presented as a solution for this, providing a "decision engine" that operates within every interaction (browser, endpoint, SAS, LLM, agent workflows) to evaluate "content, context, and intent" before an action completes, rather than merely generating alerts after the fact. This represents a shift towards proactive, inline security enforcement for AI-driven actions.

Finally, the "What's Worse" game and the tabletop exercise discussion brought to light technical vulnerabilities related to identity and access management (IAM) and incident response (IR). The scenario of "unmanaged service accounts, API tokens, and non-human identities that no one fully owns" describes a critical technical debt. These unmanaged identities represent significant attack vectors, often exploited for lateral movement and privilege escalation. The discussion on IR tabletops, particularly the failures related to firewalls, detection processes (e.g., "let me validate one more thing"), and containment strategies, underscores the technical and procedural gaps that emerge under pressure. The emphasis on cross-functional participation in tabletops, including identity management teams and audit/compliance teams, highlights the interconnectedness of various technical domains in effective incident handling.

Demo / Proof of Concept

▶ Watch: Discussing vendor selection process and avoiding traps (5:50)

As a live podcast recording, this session did not feature a traditional technical demonstration or proof of concept. The format was entirely discussion-based, allowing the speakers to share their extensive professional experiences and insights through candid conversation. The focus was on strategic decision-making, operational challenges, and the philosophical underpinnings of cybersecurity leadership rather than showcasing specific tools in action or demonstrating exploit techniques.

Defensive Implications

▶ Watch: Mike Johnson's key vendor contract advice (6:20)

The insights shared by Mike Johnson and Sarah Madden offer a robust framework for enhancing an organization's defensive posture, particularly in the face of evolving technological landscapes and human-centric challenges.

1. Re-evaluate Vendor Strategy and Demand Quality:

Defenders must adopt a more discerning and agile approach to vendor selection. The recommendation to avoid initial multi-year contracts is crucial; instead, implement one-year pilot programs or short-term agreements to thoroughly evaluate a vendor's product stability, support, and actual utility within your environment. CISOs should actively push back against the trend of "low-quality releases" from SaaS providers, demanding robust N-1 release strategies or clear commitments to stable product versions. Leverage peer networks for candid reviews and consider insights from entry-level engineers who may have experience with innovative, less-hyped solutions. This proactive vendor management minimizes technical debt and reduces exposure to unstable or unsupported software.

2. Implement Robust AI Governance and Human-in-the-Loop Controls:

The pervasive nature of AI necessitates a comprehensive AI governance framework. Defenders must assume that AI outputs, especially from Generative AI (Gen AI) and Large Language Models (LLMs), are prone to false positives and the "confidence trap." Therefore, human-in-the-loop controls should be mandatory for all critical AI-driven processes, particularly in security operations, code analysis (using IDE tools), and automated response systems. Invest in training security teams to critically evaluate AI-generated content and to understand the limitations and biases of various AI models. Tools like Nudge Security are essential for discovering and managing Shadow AI – including chatbots, agents, MCP server connections, and AI embedded in third-party SaaS tools – providing visibility and control over unsanctioned AI adoption. For Agentic AI, solutions like Quiller AI offer real-time decision enforcement by evaluating intent and context before actions are completed, moving beyond reactive alerting to proactive risk mitigation.

3. Strengthen Incident Response Through Realistic Tabletop Exercises:

Organizations must move beyond perfunctory tabletop exercises. To truly reveal "fault lines" and prepare for real breaches, exercises need to inject stakes and simulate the political and human elements of an incident. This includes allowing teams to "fail forward" without immediate correction from leadership, fostering a learning environment. Consider engaging external firms to facilitate tabletops, as their lack of familiarity with internal assumptions can expose deeper procedural and authority-related weaknesses. Ensure cross-functional participation, including teams beyond traditional SecOps, such as identity management, legal, communications, and business units, to identify inter-departmental dependencies and communication breakdowns. Regular, quarterly exercises, as practiced by Sarah Madden, help build muscle memory and continuous improvement.

4. Address Human Capital Vulnerabilities and Empower Teams:

The reliance on "security heroes" creates a significant single point of failure. Defenders must prioritize cross-training, documentation, and knowledge transfer to distribute critical expertise across the team. Invest in tools and training that empower all team members, not just traditional engineers, to leverage AI for productivity gains. As demonstrated by compliance teams using AI bots, democratizing access to AI tools (e.g., providing "tokens" or licenses) can automate mundane tasks, free up security heroes for more strategic work, and foster innovation across the organization. Simultaneously, ensure that the use of these tools is guided by the aforementioned AI governance policies.

5. Prioritize Foundational Security: Identity and Access Management:

The "What's Worse" scenario underscored the critical risk posed by unmanaged service accounts, API tokens, and non-human identities. Defenders must make the discovery, inventory, and secure management of these identities a top priority. Implement robust Identity and Access Management (IAM) practices, including regular audits, least privilege enforcement, and automated lifecycle management for all non-human identities, as these represent prime targets for adversaries.

By integrating these defensive implications, organizations can build more resilient, adaptable, and human-empowered security programs capable of navigating the complexities of the modern threat landscape.

Key Takeaways

• Strategic Vendor Management is Crucial: Avoid multi-year contracts for new solutions, leverage peer networks for informed vendor selection, and relentlessly pressure vendors for stable, high-quality software releases to prevent technical debt and operational instability.
• AI Demands Critical Trust and Human Oversight: Treat AI output with skepticism, enforce human-in-the-loop controls for validation, and actively manage false positives. AI is a powerful augmentation tool, not a replacement for human judgment.
• Proactive AI Governance is Essential: Develop robust frameworks to discover and control Shadow AI across the organization, including Agentic AI systems. Implement real-time decision engines to secure AI-driven actions rather than just monitoring for post-factum alerts.
• Realistic Tabletop Exercises Expose True Weaknesses: Design incident response tabletops that inject genuine "stakes" and allow teams to make mistakes and learn. Involve cross-functional teams and consider external facilitators to reveal deeper issues related to authority and inter-departmental coordination.
• Empower Your Team, Don't Over-rely on Heroes: Distribute knowledge and skills to avoid single points of failure. Actively encourage and enable all team members, including non-engineers, to use AI tools responsibly to automate tasks and increase overall team effectiveness.

About the Speaker(s)

• David Spark: The producer of the CISO Series podcast, David Spark serves as the host and moderator for these engaging discussions. He is known for guiding conversations that explore the practical challenges and strategic insights relevant to cybersecurity leaders.
• Mike Johnson: As the CISO for Rivian, Mike Johnson brings extensive experience in securing innovative technology environments. He is also a long-standing co-host of the CISO Series podcast, offering seasoned perspectives on vendor management, AI adoption, and team leadership.
• Sarah Madden: The CISO of Convera, Sarah Madden is a returning guest to the CISO Series, providing valuable insights from her role in building greenfield security programs and managing complex security landscapes. Her expertise includes navigating multi-year vendor contracts, demanding software quality, and strategically integrating AI into security operations.