PRSA: Prompt Stealing Attacks against Real-World Prompt Services

A Mixed-Methods Study of Open-Source Software Maintainers On Vulnerability Management and Platform Security Features

Speakers: Jessy Ayala, Researcher, University of California (Irvine); Yu-Jye Tung, Researcher, University of California (Irvine); Joshua Garcia, Professor, University of California (Irvine)

Conference: USENIX Security

YouTube: https://www.usenix.org/conference/usenixsecurity25/presentation/ayala

Overview

In the rapidly evolving landscape of modern software development, open-source software (OSS) forms the backbone of countless applications and services. However, this pervasive reliance on OSS has brought to the forefront a critical challenge: the escalating number of software vulnerabilities. While much research has focused on the perspectives of vulnerability reporters and the security practices of OSS contributors, the crucial viewpoint of OSS maintainers—especially those managing projects with a history of known vulnerabilities—has remained significantly understudied. This paper, "A Mixed-Methods Study of Open-Source Software Maintainers On Vulnerability Management and Platform Security Features," by Jessy Ayala, Yu-Jye Tung, and Joshua Garcia from the University of California, Irvine, addresses this critical gap.

The study investigates how OSS maintainers, particularly those whose projects are listed in the GitHub Advisory Database, approach vulnerability management and interact with platform security features (PSFs). Through a comprehensive mixed-methods approach, combining a listing survey with 80 participants and semi-structured interviews with 22 maintainers, the researchers uncover the multifaceted challenges, barriers, and desires of these frontline defenders. The findings reveal systemic issues ranging from profound supply chain mistrust and a desperate need for automation to a striking lack of awareness and usability concerns surrounding existing PSFs.

This research is paramount because OSS maintainers, often working without formal security training or adequate resources, are the primary custodians of software integrity in a deeply integrated ecosystem. Their experiences and insights are vital for developing more effective tools, policies, and support mechanisms that can bolster the security posture of the entire software supply chain. By focusing on maintainers of previously vulnerable projects, the study offers a unique and pragmatic perspective, highlighting areas where OSS platforms and the research community can make the most impactful interventions to secure the future of open-source.

Background

The pervasive nature of open-source software in the modern digital infrastructure cannot be overstated. A recent report by Synopsys [94] revealed that an astounding 96% of 1,067 scanned codebases across 17 industries contained OSS components. This widespread adoption means that vulnerabilities deep within the software supply chain can have "crippling effect on society" [33, 37, 42]. At the vanguard of defending against these threats are OSS maintainers, many of whom "may not have experience handling a security incident" [101]. Despite the availability of various platform security features (PSFs) on platforms like GitHub—such as dependency management tools [52]—these features are often underutilized [11, 12, 14, 15].

Prior research has extensively explored the landscape of vulnerability management from the perspectives of vulnerability reporters [4, 6, 39, 60, 71, 75, 108] and the general security practices of OSS contributors [57, 101]. However, a significant knowledge gap persists regarding the specific challenges and needs of OSS maintainers, particularly those who have already experienced and patched vulnerabilities in their projects. This gap is critical because these maintainers are precisely the individuals with firsthand experience in triaging and remediating security flaws, offering invaluable insights into the practicalities and pain points of the vulnerability management lifecycle.

The challenges faced by maintainers extend beyond technical issues. Many experience burnout [67] or abandon their projects entirely [63] due to the demanding responsibilities. The recent xz-utils incident (CVE-2024-3094) [80], where a malicious actor infiltrated a widely used utility, starkly underscored the profound challenges of vulnerability management and the pervasive mistrust in the software supply chain. This incident highlighted the fragility of relying on a small number of maintainers, often with limited resources, to secure critical components that underpin vast swathes of the internet.

Existing literature on vulnerability management in OSS has identified several shortcomings. Studies have shown that current OSS vulnerability management practices are often insufficient, with a lack of security policies in many GitHub repositories [11]. Furthermore, initial security fixes are frequently inadequate, necessitating multiple patches [17, 69], and a significant portion of security issues can persist in repositories for up to three years before remediation [69]. While some studies interviewed security practitioners [9] or general OSS maintainers [101], they often found that security plays a minor role in maintainers' overall duties, with few having dedicated security roles or extensive experience with incidents. This study differentiates itself by specifically targeting maintainers of previously vulnerable projects, offering a unique lens into the practical realities of managing security post-incident and engaging with platform-provided security tools.

Key Findings

The mixed-methods study, comprising a listing survey with 80 participants and semi-structured interviews with 22 maintainers, yielded 37 distinct factors influencing vulnerability management, categorized into current practices, general challenges, PSF challenges, PSF barriers, and maintainer wants. These findings illuminate critical areas for intervention and support within the OSS ecosystem.

Regarding current vulnerability management practices, the study found that most maintainers prioritize private communication channels for vulnerability reporting. Email or mailing lists (F1) were the most listed method (L=50, I=11), followed by an established proactive disclosure process (F2) (L=46, I=7) after patching. While many have security policies (F3) (L=37, I=10), their primary use is often just providing a contact point. Interestingly, some maintainers (L=9, I=6) still use public GitHub Issues (F6) for reporting, and two survey participants even admitted to ignoring vulnerabilities (F8) altogether, citing lack of motivation or perceived unnecessity. GitHub's private vulnerability reporting (F4) feature and automation tooling (F5) like Dependabot and Renovate Bot were also noted as current practices, but often with caveats.

The most pressing general challenges included supply chain mistrust (F9) (L=36, I=6), driven by concerns about unmaintained dependencies, delays in fixes, and the risk of malicious actors, exemplified by the xz-utils incident. A pervasive lack of understanding (F10) (L=35, I=12) regarding vulnerability complexity, patch development, and testing was also prominent. Maintainers consistently cited a lack of time (F11) (L=25, I=9) and resources (F12) (L=23, I=11) as significant hurdles, often leading to burnout. Ethical dilemmas and frustration surrounding CVE relationships (F13) (L=8, I=9), including slow processes and pressure to downgrade severity, were also reported.

When it came to platform security features (PSFs), the primary challenges were a lack of automation (F17) (L=39, I=11) and too much noise (F18) (L=24, I=10) from false positives in dependency management and static analysis tools. This noise often led to notification fatigue and negatively impacted maintainer motivation. Difficulties with vulnerability scoring (F19) (L=21, I=9), particularly calculating CVSS scores, were widespread. A critical technical finding was the absence of CI processes in private forks (F20) (L=8, I=7), hindering proper testing of fixes and leading to broken tests and builds (F21) (L=6, I=7) upon merging.

Barriers to PSF adoption were equally significant. The most frequently cited barrier was a stark lack of awareness (F24) (L=33, I=12) about available features like private vulnerability reporting and security advisories; 26.7% of survey participants had none of the three mentioned PSFs enabled. PSFs were also perceived as complex to set up or use (F25) (L=26, I=7) and often deemed unnecessary (F26) (L=23, I=6) for projects perceived as less important. Concerns about project reputation (F27) (L=11, I=4) and general maintainer burnout (F29) (L=5, I=4) further deterred adoption.

Finally, maintainers expressed clear wants and opportunities for improvement. The top requests included assisted vulnerability analysis and triaging (F31) (L=39, I=9) to reduce noise and help with patch development, and assisted PSF setup (F32) (L=37, I=10) with interactive guidance and recommendations. There was strong interest in security-specific funding (F33) (L=32, I=7) and cyber defense gamification (F35) (L=13, I=6), such as a "green shield" for projects with good security posture. User-friendly resources (F34), checklists (F36), and nudges (F37) were also highly desired.

Technical Deep Dive

The study delved into the practicalities of vulnerability management by examining maintainers' current practices, the technical and operational challenges they face, and the specific barriers to adopting platform security features (PSFs). The insights gleaned from 80 survey respondents and 22 interviewees provide a granular view of the OSS security landscape.

Current Vulnerability Management Practices

Maintainers employ a diverse set of strategies, both on and off GitHub, for handling vulnerabilities. The most prevalent method for initial vulnerability reporting is using email or a private mailing list (F1) (L=50, I=11). This preference stems from a desire to keep reports private, preventing public disclosure until a fix is ready. As one participant (P4) noted, email facilitates a "conversation" rather than a mere report dismissal. Some larger projects also leverage external tooling (F7) like Coverity [93] for static analysis or receive bundled reports from security companies (P17).

After a vulnerability is patched, a proactive disclosure process (F2) (L=46, I=7) is common, often involving mailing list announcements, GitHub security advisories, or quickly requesting a CVE (L=5). This demonstrates a commitment to informing dependent projects, though some (P6) felt disclosure was only necessary for particularly important vulnerabilities due to "a lot of bandwidth on security disclosures."

Security policies (F3) (L=37, I=10) are frequently in place, primarily to guide reporters to private channels. GitHub's built-in private vulnerability reporting (F4) feature, which allows contributors to privately report issues and maintainers to develop fixes in private forks, was used by a notable subset (L=20, I=9). Maintainers appreciated its ease of setup and centralized nature (P9).

Automation tooling (F5) plays a role, with Dependabot (L=12, I=6) and Renovate Bot (L=4, I=3) being popular for dependency updates. Renovate Bot, in particular, was lauded for its auto-merge capabilities for minor updates, reducing manual overhead (P1). Despite these efforts, some maintainers (L=9, I=6) still use public GitHub Issues (F6) for reporting, sometimes for minor issues or to solicit public help (P19). Disturbingly, two survey participants admitted to ignoring vulnerabilities (F8) due to lack of motivation or a belief that security features were unnecessary.

General Vulnerability Management Challenges

A significant concern is supply chain trustworthiness (F9) (L=36, I=6). Maintainers struggle to keep up with dependency updates (L=20) and are frustrated by unmaintained upstream projects or delays in vulnerability fixes (L=12). The specter of malicious actors intentionally introducing vulnerabilities, as seen in the xz-utils incident (CVE-2024-3094) [80], looms large (P6).

A pervasive lack of understanding (F10) (L=35, I=12) of complex vulnerabilities, patch development, and testing processes leads to delays. Many feel their "brain is far too small to understand [vulnerabilities]" (P9). This is compounded by a chronic lack of time (F11) (L=25, I=9) and resources (F12) (L=23, I=11), especially for lone maintainers (I=11), who must balance security with ongoing development (P17).

Negative relationships with CVEs (F13) (L=8, I=9) are common. Maintainers described the CVE program as slow, managed by a single entity (USA, not 24/7), and prone to poor quality content due to analyst backlogs. Some felt pressured to "lie to make the CVE [severity] lower because it makes my project look bad" (P15), highlighting an ethical dilemma. The lack of procedures (F14) (L=7, I=5) and challenges with coordinated disclosure (F15) (L=7, I=4) also contribute to "imposter syndrome" (P13). Finally, negative attitudes (F16) (L=6, I=9) from argumentative reporters or those solely focused on financial gain add emotional strain (P2).

Platform Security Feature (PSF) Challenges

The most prominent PSF challenge is the lack of sufficient automation (F17) (L=39, I=11). While tools like Renovate Bot offer auto-merging for dependency updates, maintainers desire more comprehensive automation, especially for private vulnerability reports if builds don't break. However, current limitations often force some to merge automated pull requests "regardless of testing" (P5), underscoring a tension between convenience and rigor.

Too much noise (F18) (L=24, I=10) is a major detractor, primarily from false positives generated by dependency management tools (L=8, I=6) and static analysis (L=4, I=2). This noise leads to notification fatigue and, as one maintainer (P8) lamented, can "negatively impacts their motivation to continue maintaining OSS projects," causing them to abandon projects.

Challenges with vulnerability scoring (F19) (L=21, I=9) are widespread, with maintainers struggling to calculate or adjust CVSS scores, finding existing documentation geared towards security researchers rather than everyday developers (P3).

A critical technical hurdle is the missing CI processes in private forks (F20) (L=8, I=7) associated with GitHub's private vulnerability reporting. This means maintainers cannot run automated tests on fixes in the same environment as their main branch, leading to workarounds like hosting parallel private repositories or testing locally (P14). This often results in broken tests and builds (F21) (L=6, I=7) after patches are merged publicly, highlighting a significant workflow impedance.

Usability issues include cluttered notifications (F22) (L=4, I=4), especially for dependency updates, and too much manual setup (F23) (L=3, I=3) required to enable PSFs individually across projects or add collaborators to private forks.

Platform Security Feature (PSF) Barriers

A pervasive lack of awareness (F24) (L=33, I=12) is the leading barrier to PSF adoption. Many maintainers were unaware of features like security policies, private vulnerability reporting, or public security advisories; 26.7% (21/80) of survey participants had none of these three PSFs enabled. This indicates a significant gap in discoverability and promotion. Relatedly, bad UI presentation (F28) (L=8, I=6) makes PSFs feel "hidden" or like "second-class features" (P10, P11).

PSFs are often perceived as complex to set up or use (F25) (L=26, I=7), with maintainers expressing "general ignorance as to why they are not aware of what particular PSFs do" (L=3, I=7). This complexity, combined with a lack of clear understanding of their benefits (F30), deters adoption (P12).

The perception that PSFs are unnecessary (F26) (L=23, I=6) is another significant barrier, especially for projects perceived as less important (L=16, I=6) or where security is not a "main goal" (P8). Concerns about project reputation (F27) (L=11, I=4)—the fear that past vulnerabilities reflect negatively—can also discourage transparent security practices, though some (P14) viewed patched vulnerabilities as a sign of a "healthy project." Finally, lack of motivation (F29) (L=5, I=4) due to maintainer burnout further hinders engagement with security tasks.

Demo / Proof of Concept

This paper describes a comprehensive mixed-methods study involving surveys and semi-structured interviews with open-source software maintainers. The research methodology focuses on gathering qualitative and quantitative data about maintainers' experiences, challenges, and perceptions regarding vulnerability management and platform security features. As such, the study does not involve the demonstration or proof of concept of any technical vulnerability, tool, or system. The findings are based on the collective insights and self-reported data from the participant maintainers.

Defensive Implications

The findings from this study offer critical implications for various stakeholders involved in the OSS vulnerability management lifecycle, including OSS platforms, maintainers, and the research community. Addressing the identified challenges and barriers requires a concerted, multi-pronged approach.

For OSS Platforms (e.g., GitHub)

OSS platforms are uniquely positioned to alleviate many of the challenges identified:

• Enable CI in Private Forks (F20): A top request, platforms should investigate and implement secure mechanisms to allow CI/CD processes in private forks used for vulnerability remediation. This would enable maintainers to thoroughly test fixes before public merging, reducing the risk of broken tests and builds (F21) and the need for cumbersome workarounds. Research challenges include sandboxing and access controls for secure CI in private environments.
• Improve PSF Usability and Awareness (F23, F24, F25, F28, F30):
• "Enable Best Practices" Button (F32, F36): Platforms should offer a "one-click" option to enable a recommended set of easily configurable PSFs, coupled with clear, user-friendly documentation (F34) tailored for maintainers without a security background. This could significantly overcome lack of awareness (F24) and complexity (F25) barriers.
• Context-Aware Nudges and Checklists (F36, F37): Implement gentle nudges or checklists, especially when security-related keywords are detected in public issues, to guide maintainers towards private vulnerability reporting (F4) or other relevant PSFs.
• Enhanced UI/UX (F28): Make PSFs more discoverable and integrate them seamlessly into the development workflow, moving them from "second-class features" (P10) to prominent, intuitive components.
• Automated Vulnerability Management Tooling (F17, F31):
• Assisted Triaging and Impact Analysis (F31): Develop tools that automatically analyze the context of vulnerabilities, reduce false positives (F18), and assist with patch development and test generation. This could leverage Large Language Models (LLMs) to interpret reports and suggest fixes, while carefully addressing their potential for misinterpretation or inaccurate patches.
• Automated Vulnerability Scoring (F19): Research and implement context-aware tooling that helps maintainers accurately calculate CVSS scores, taking into account deployment environment, project purpose, and component interactions. This is a particularly understudied area.
• Incentivize Security Practices (F33, F35):
• Security-Specific Funding (F33): Facilitate mechanisms for funding OSS security efforts, such as dedicated bounty pools or direct support for maintainers and reporters.
• Cyber Defense Gamification (F35): Introduce features like a "green shield" icon for projects that adopt recommended PSFs and maintain a strong security posture. This could foster positive project reputation (F27) and motivate engagement, potentially through leaderboards for maintainers, reporters, and projects.
• Address Public Reporting (F6): While discouraging public vulnerability reporting, platforms should research how to automatically identify likely vulnerabilities in public issues and, with maintainer consent, convert them into private reports, thereby promoting the use of private vulnerability reporting (F4) and preventing premature disclosure.

For OSS Maintainers

Maintainers, as the ultimate guardians of their projects' security, can take proactive steps:

• Actively Explore PSFs: Despite initial complexities, maintainers should dedicate time to explore and enable available PSFs, particularly those that are easily togglable like private vulnerability reporting. Many participants expressed interest after learning about them.
• Prioritize Security Education: Seek out user-friendly resources (F34) and training materials to bridge knowledge gaps (F10) and overcome "imposter syndrome" (P13) when dealing with vulnerabilities.
• Establish Clear Policies: Implement and clearly document security policies (F3) and disclosure processes (F2) to guide reporters and ensure timely, coordinated responses.
• Advocate for Support: Communicate needs and challenges to OSS platforms and the wider community, especially regarding the need for automation, funding, and improved tooling.

For the Research Community

The research community has a vital role in developing the next generation of security solutions:

• Secure CI for Vulnerability Management: Research into designing and implementing secure CI features for private forks is crucial, potentially involving novel sandboxing or access control mechanisms.
• Automated Vulnerability Triaging with LLMs: Investigate the effective and safe application of LLMs for interpreting vulnerability reports, generating patches, and developing regression tests, while meticulously addressing issues of accuracy, interpretability, and potential for misinterpretation.
• Context-Aware Vulnerability Scoring: Advance research into automated systems that can accurately score vulnerabilities by considering the specific project context, deployment environment, and interaction with other components.
• Usability-Focused PSF Design: Conduct user studies to inform the design of PSFs that are intuitive, minimize noise (F18), and provide clear guidance for non-security experts.
• Effective Gamification Strategies: Research how to best leverage gamification (F35) to encourage consistent adoption of security best practices and foster a positive security culture.
• Supply Chain Trust Mechanisms: Explore and improve attestation practices, such as Software Bill of Materials (SBOMs) [83, 84, 104], to foster upstream trust and provide greater transparency in the software supply chain.

By addressing these implications, the OSS ecosystem can collectively move towards a more secure, resilient, and maintainer-friendly future.

Key Takeaways

• Pervasive Challenges for Maintainers: OSS maintainers, especially those managing previously vulnerable projects, face significant hurdles in vulnerability management, primarily due to supply chain mistrust, lack of understanding of complex vulnerabilities, and acute time and resource constraints.
• Underutilization of Platform Security Features (PSFs): GitHub's PSFs are widely underutilized, not because they are inherently bad, but due to a critical lack of awareness, perceived complexity in setup and use, and a feeling that they are unnecessary or generate too much noise (false positives).
• Operational Gaps in PSF Functionality: A key technical impedance is the absence of CI processes in private forks, forcing maintainers into cumbersome workarounds and increasing the risk of broken builds when deploying fixes. This highlights a need for better integration of security workflows into development.
• Desire for Automation and Support: Maintainers overwhelmingly desire more automation for vulnerability analysis and triaging, assisted PSF setup, and user-friendly resources tailored for those without a security background, indicating a strong demand for simplified security.
• Incentives and Reputation Matter: Security-specific funding and gamification (e.g., "green shield" for good security posture) are seen as powerful motivators to encourage better security practices and positively influence project reputation, which currently can be a barrier to transparent disclosure.
• Call for Collaborative Action: Securing the OSS ecosystem requires concerted effort from OSS platforms (improving tools, UI/UX, awareness), maintainers (proactive engagement), and researchers (developing advanced, usable security tooling, especially with LLMs and secure CI).

About the Speaker(s)

The research paper "A Mixed-Methods Study of Open-Source Software Maintainers On Vulnerability Management and Platform Security Features" was authored by Jessy Ayala, Yu-Jye Tung, and Joshua Garcia, all affiliated with the University of California, Irvine. Their collaborative work focuses on understanding and improving the security posture of the open-source software ecosystem. Specifically, their research delves into the human-centered aspects of software security, investigating the perspectives of OSS maintainers regarding vulnerability management and the adoption of platform security features. Their contributions aim to inform better tooling, policies, and support mechanisms for the vital community of OSS maintainers.