Sharpness-Aware Initialization: Improving Differentially Private Machine Learning from First Principles

Confusing Value with Enumeration: Studying the Use of CVEs in Academia

Speakers: Moritz Schloegel (CISPA Helmholtz Center for Information Security); Daniel Klischies (Ruhr University Bochum); Simon Koch (TU Braunschweig); David Klein (TU Braunschweig); Lukas Gerlach (CISPA Helmholtz Center for Information Security); Malte Wessels (TU Braunschweig); Leon Trampert (CISPA Helmholtz Center for Information Security); Martin Johns (TU Braunschweig); Mathy Vanhoef (KU Leuven); DistriNet (KU Leuven); Michael Schwarz (CISPA Helmholtz Center for Information Security); Thorsten Holz (CISPA Helmholtz Center for Information Security); Jo Van Bulck (KU Leuven); DistriNet (KU Leuven)

Conference: USENIX Security

YouTube: No video available as this is a peer-reviewed conference paper.

Overview

This distinguished paper from USENIX Security 2025, titled "Confusing Value with Enumeration: Studying the Use of CVEs in Academia," presents a critical and systematic examination of how Common Vulnerabilities and Exposures (CVE) identifiers are used within academic security research. Authored by a collaborative team from leading European institutions including CISPA Helmholtz Center for Information Security, Ruhr University Bochum, TU Braunschweig, and KU Leuven, the research delves into the prevalent misconception that assigning a CVE inherently signifies a serious security issue or real-world impact. The paper argues that academics frequently claim CVEs to demonstrate the practical relevance of their findings, often overlooking the original purpose of CVEs as mere unique identifiers.

The study employs a multi-faceted approach, combining quantitative analysis of A* security conference papers, a qualitative review of claimed CVEs, and a comprehensive survey of academic reviewers and authors. Its findings reveal a concerning trend: the increasing frequency of CVE mentions in academic publications, coupled with a significant portion of these claimed CVEs lacking public confirmation or facing disputes from affected software maintainers. This research is crucial for the security community, as it exposes widespread misunderstandings about the CVE system and provides actionable recommendations to foster a more accurate and responsible use of CVEs in academic discourse and practice.

The core message of the paper resonates with growing concerns about the "metrification" of academic success, where quantitative metrics like CVE counts can be "gamed" (Goodhart's law), potentially leading to a focus on obtaining identifiers rather than genuinely enhancing software security. By meticulously dissecting the lifecycle and perception of academic CVEs, the authors aim to recalibrate the understanding of these identifiers, urging both researchers and practitioners to exercise caution when interpreting CVEs as definitive proxies for real-world impact.

Background

The Common Vulnerabilities and Exposures (CVE) program, managed by MITRE, was established with the explicit goal of providing unique identifiers for security vulnerabilities, thereby standardizing communication and tracking across the industry. Launched in September 1999, the system has grown exponentially, now cataloging over 250,000 CVE records with an average of 164 new entries daily in 2024 (Section 1). To facilitate this, organizations can register as CVE Numbering Authorities (CNAs) for their products, while CNA Numbering Authorities of Last Resort (CNA-LRs) like MITRE and CISA handle assignments for projects without dedicated CNAs.

Despite its widespread adoption, the CVE system has faced increasing criticism for several inherent shortcomings. Firstly, a single CVE does not always correspond to a single vulnerability, as exemplified by microarchitectural CPU vulnerabilities like Spectre, which may necessitate numerous patches across the software stack but receive only one CVE ID. Secondly, the assignment process lacks unified procedures, allowing CNAs discretion that can lead to inconsistencies; some companies may avoid assigning CVEs to maintain a false sense of security, while CNA-LRs might have lower assignment standards due to a lack of expert knowledge. Thirdly, invalidating disputed CVEs is notoriously difficult, prompting major open-source projects like Linux and Python to become their own CNAs to regain control over the process.

Crucially, the paper highlights that CVEs have arguably "outlived their original role as vulnerability identifiers and have become de-facto proxies for real-world impact and prestigious indicators of competence in the field" (Section 1). This phenomenon is particularly pronounced in academia, where security papers, especially those presenting novel vulnerability identification techniques, often report many CVEs as a metric for newly discovered bugs. The authors cite observations from prominent developers who criticize CVEs for being "abused by security developers looking to pad their resumes" and "used in ways that elevate their importance well beyond the level that makes sense" (Section 1). This academic embrace of CVEs has even led to explicit instructions in conference calls for papers, such as IEEE S&P 2025, to blind CVE identifiers to preserve anonymity, underscoring their significance beyond mere identification.

Prior academic work has explored CVEs as datasets for tool testing and vulnerability patching, and various individuals have criticized the assignment process or overall effectiveness. However, this paper distinguishes itself as the first to systematically review CVE usage in academia through quantitative, qualitative, and survey methods, aiming to capture the community's perception and the actual correlation of academic CVEs with real-world implications.

Key Findings

The paper presents several critical findings that collectively challenge the current academic practice of using CVEs as a proxy for impact:

• Increasing Prevalence of CVEs in Academia: The quantitative analysis reveals a clear and steady increase in the relative proportion of papers mentioning CVEs at top-tier security conferences (IEEE S&P, USENIX Security, ACM CCS, NDSS) over the last 25 years. This trend is particularly pronounced in certain research areas, such as fuzzing, microarchitectural security, and browser security, which show significantly higher CVE mentions per paper compared to fields like adversarial machine learning or cryptography (Section 3, Takeaway #1).
• Questionable Quality of Claimed CVEs: A deep qualitative review of 1,803 CVEs claimed in papers published between 2020 and 2024 found that only 66% (63% fixed, 3% confirmed) were agreed upon by both the reporter and project maintainers. A significant 34% of these claimed CVEs had not been publicly confirmed or were disputed by maintainers, with 13% being explicitly problematic (No-info, Ignored, or Opposed) and 21% being Reserved or Unsupported, leaving their quality uncertain (Section 4, Takeaway #2).
• CNA-LRs as Sources of Disagreement: The analysis of CVEs by their assigning CNA highlighted that CNA-LRs (such as MITRE) are almost exclusively the source of CVEs that project maintainers explicitly oppose or ignore. In contrast, CVEs assigned by vendor-specific CNAs rarely faced such disagreement, suggesting a lack of rigorous verification in CNA-LR processes (Section 4.1.2, Takeaway #3).
• Bug-Finding Research Areas Exhibit Higher Disagreement: Papers focused on discovering bugs, particularly in areas like fuzzing and web security, were found to claim the most CVEs, but also exhibited a higher likelihood of these CVEs facing opposition from maintainers. This suggests that researchers in these fields might be more inclined to push for CVE assignments, even against maintainer disagreement, potentially to demonstrate tool capabilities (Section 4.1.3, Takeaway #4).
• Widespread Misconceptions Among Academics: A survey of 103 academic reviewers and authors revealed significant misunderstandings about the CVE system. While 95% correctly identified CVEs' primary purpose as unique identification, a striking 54% incorrectly believed that vulnerability verification is a mandatory step in the CVE assignment process. Many also viewed CVEs as indicators of validity and impact, creating unwarranted confidence (Section 5, Takeaway #6).
• Perceived Desirability and Unclear Influence on Acceptance: Most participants (71%) expressed a desire to obtain CVEs for discovered vulnerabilities, believing they demonstrate practical impact (76%). While 48% of authors felt CVEs positively impacted paper acceptance, the survey initially found no implicit bias in abstract evaluation. However, 68% of participants stated that their impression of a paper is positively affected by the presence of CVEs, and with four reviewers per paper, there's a 99% chance at least one reviewer's impression will improve, and an 85% chance at least one will actively look for CVEs (Section 5, Takeaway #7 & #8).
• Discrepancy in Desired vs. Actual Influence: Despite the clear positive effect on reviewer perception, 67% of participants opposed the idea that CVEs should influence paper acceptance decisions. This highlights a significant disconnect between the community's stated ethical stance and the observed reality of how CVEs impact evaluation (Section 5, Takeaway #9). Academics also preferred alternative impact metrics, such as evaluation on popular targets and demonstrated exploitability, over mere CVE counts (Section 5, Table 6).

Technical Deep Dive

The study's methodology comprised three main components: a large-scale quantitative analysis, an in-depth qualitative review, and a community-wide survey, each designed to shed light on different facets of CVE usage in academia.

Quantitative Analysis of CVE Usage (Section 3):

The researchers constructed a dataset of 7,785 papers published between 1999 and 2024 from the top four A* security conferences: IEEE S&P, USENIX Security, ACM CCS, and ISOC NDSS. Paper PDFs were processed using pdftotext and Nougat (for OCR fallback) to extract full text. CVE IDs were identified using a robust regular expression ([cC][vV][eE]\s*[--]?\s*\d{4}\s*[--]?\s*\d{4,7}) to account for variations in formatting. The dataset ultimately contained 1,167 papers (15.0%) mentioning a total of 6,506 unique CVE identifiers.

Key quantitative observations include:

• Prevalence Trends: The percentage of papers mentioning CVEs has risen sharply, from 10.7% between 2010-2014 to 18.7% between 2020-2024 (Figure 1). USENIX Security consistently showed the highest prevalence (20.3% over the entire period).
• CVE Status: A comparison of the 6,506 academic CVEs with all 276,123 published CVEs revealed that 94.4% of academic CVEs were 'Published', 4.7% 'Reserved', and less than 1% 'Rejected' or 'Disputed/Unsupported' (Table 1). Academic CVEs were less frequently rejected than CVEs overall.
• Research Area Correlation: Using KeyBERT for unsupervised topic extraction from paper abstracts, the study found a strong correlation between CVE usage and specific research areas. Papers classified under fuzzing (104 papers) mentioned an average of 7.6 CVEs per paper, followed by microarchitectural security (59 papers) with 1.5 CVEs, and browser security (100 papers) with 0.86 CVEs. In contrast, areas like adversarial machine learning, backdoors, differential privacy, and various cryptography subfields had significantly lower or no CVE mentions, highlighting the differing importance of CVEs across disciplines.

Qualitative Analysis of CVE Reception (Section 4):

To assess the actual outcome and maintainer agreement, the researchers focused on 304 papers published at the A* venues between 2020 and 2024, manually extracting and analyzing 1,803 claimed CVEs (i.e., those resulting from the paper's contributions, not merely cited). Four independent expert researchers individually assessed each CVE, assigning one of seven labels:

• Fixed (F): Issue rectified.
• Confirmed (C): Issue confirmed, but not yet fixed.
• Reserved (R): CVE record withheld, no information.
• Unsupported (U): Software unsupported, no maintainer reaction.
• No-info (N): Insufficient public information to assess outcome.
• Ignored (I): Bug report ignored despite active software development.
• Opposed (O): Maintainers explicitly disputed the issue as a bug or security impact.

The labeling process involved leveraging official CVE records, linked resources (repositories, bug trackers), and Google searches. Conflicts were resolved through expert discussion.

• Overall Outcomes: 63% (1,135) of CVEs were Fixed, and 3% (54) were Confirmed, indicating agreement. However, 2.9% (52) were Opposed, 1.4% (26) Ignored, and 8.9% (161) had No-info. Another 15% (274) were Reserved and 5.6% (101) were for Unsupported software (Table 2). This means 34% of claimed CVEs lacked clear agreement or sufficient information.
• Unspecified and Reserved CVEs: 57 papers claimed 981 CVEs without specifying IDs, making verification impossible. 274 CVEs remained in the reserved state, with 99 of these being for older bugs (before 2024), raising questions about delayed disclosure.
• CNA Roles: 66 different CNAs were observed. MITRE, as a CNA-LR, assigned 41% (626) of the analyzed CVEs and was the primary source of Opposed (36) and Ignored (21) CVEs. Almost all disputed CVEs originated from CNA-LRs, underscoring their lower verification standards compared to vendor-specific CNAs (Table 3).
• Paper Topics vs. Outcomes: Fuzzing papers claimed the most CVEs (38%, 694 total) and also accounted for the highest number of Opposed CVEs (31). Other bug-finding topics like web security and general bug finding also showed significant opposition, suggesting a potential incentive for researchers to claim CVEs even when maintainers disagree (Table 4).
• Outlier Analysis (Section 4.1.4): The study identified four "Opposed" outliers and one "Uncertain" outlier (Opposed-3 was in both categories). These papers exhibited patterns such as claiming CVEs for issues already disputed or rejected by developers, issues requiring administrative credentials (not considered vulnerabilities), or vulnerabilities in end-of-life products. Examples included lack of threat analysis (e.g., reporting wrapper functions as command injection), lack of communication with maintainers, and reporting CVEs despite explicit vendor statements against assignment.

Survey of Academics (Section 5):

A survey was conducted with 103 individuals (professors, PhD students, postdocs, industry researchers) from the PC committees of the top 4 security conferences in 2023. The survey assessed implicit bias, knowledge of CVEs, author/reviewer behavior, and desired CVE usage.

• Misconceptions: 54% of participants incorrectly believed that vulnerability verification is a mandatory step in CVE assignment. 37% wrongly thought third-party CNAs could assign CVEs even if a product's vendor was a CNA (violating CNA rules).
• Impact on Acceptance: 71% of authors try to obtain CVEs, and 76% believe CVEs demonstrate practical impact. 48% of authors believed CVEs influenced paper acceptance. While an initial implicit bias test yielded no statistical significance, 68% of participants admitted that their impression of a paper is positively affected by CVEs. This leads to a 99% chance that at least one reviewer's impression will be positively affected and an 85% chance that at least one reviewer will actively check for CVEs.
• Desired Use: 67% of participants opposed CVEs influencing paper acceptance, and preferred alternative metrics like evaluation on popular targets (Table 6).

Demo / Proof of Concept

This paper is a meta-science study focused on analyzing academic practices and perceptions rather than presenting a novel security vulnerability, tool, or attack. Therefore, it does not include a traditional "Demo" or "Proof of Concept" in the sense of demonstrating an exploit or a new security mechanism. Its strength lies in its comprehensive empirical data collection, systematic analysis of existing CVEs, and the insights derived from surveying the academic community. The "proof" is in the data and the identified trends, rather than a live demonstration.

Defensive Implications

The findings of this paper carry significant defensive implications for various stakeholders within the security ecosystem, from academic researchers and conference organizers to software maintainers and security practitioners.

For Academic Researchers and Authors:

• Re-evaluate CVEs as Impact Metrics: Academics must move away from using CVE counts as the primary or sole metric for demonstrating real-world impact. The study conclusively shows that a significant portion of claimed CVEs are questionable or disputed, undermining their value as an impact proxy (Takeaway #2).
• Focus on Nuanced Impact Arguments: Instead of merely listing CVEs, authors should provide detailed arguments for the practical relevance of their work. This includes elaborating on vendor responses, proposed mitigations, exploitability analysis, and evaluation on popular, real-world targets (Table 6).
• Adhere to Responsible Disclosure and Communication: The analysis of outlier papers highlighted issues like lack of threat analysis, poor communication with maintainers, and even reporting CVEs for issues explicitly deemed non-security relevant or already rejected. Researchers must prioritize clear, responsible disclosure, engage in dialogue with maintainers, and conduct thorough threat modeling before seeking CVE assignments. The example of CVE-2021-34141, where NumPy maintainers labeled a reporter as a "known bad actor" due to bogus CVEs and lack of response, serves as a stark warning (Section 4.1.4).
• Understand CVE Assignment Processes: The survey revealed widespread misconceptions about CVE verification and CNA rules (Takeaway #6). Authors should educate themselves on the distinction between CNAs and CNA-LRs and understand that CVEs assigned by CNA-LRs may lack vendor verification, making them more susceptible to dispute.
• Consider Blinding CVEs in Submissions: To address potential deanonymization and reduce the incentive for "CVE farming" driven by peer review, authors should consider blinding CVE identifiers in initial submissions, similar to the mandate in IEEE S&P 2025 (Recommendation 1).

For Academic Reviewers and Program Committees:

• Challenge the CVE-as-Impact Fallacy: Reviewers and PC members should actively resist the temptation to view CVEs as an automatic indicator of a paper's real-world impact or quality. Conference calls for papers and PC guidelines should explicitly clarify expectations, discouraging or even prohibiting the use of CVEs as a primary metric (Recommendation 3).
• Prioritize Rigorous Verification: Instead of passively accepting CVE claims, reviewers should scrutinize the details surrounding claimed CVEs, including vendor responses, fix status, and the assigning CNA. Artifact evaluation processes could be extended to include verification of claimed CVEs, requiring authors to provide valid explanations for reserved or unconfirmed CVEs (Recommendation 4).
• Focus on Alternative Impact Metrics: Reviewers should prioritize alternative, more robust metrics for real-world impact, such as thorough evaluations on popular software, clear arguments for exploitability, and successful reproduction of known vulnerabilities, as preferred by the surveyed academics (Table 6).

For Software Maintainers and Practitioners:

• Be Aware of Questionable Academic CVEs: Maintainers should be aware that CVEs originating from academic papers, especially those assigned by CNA-LRs, may not always represent verified or impactful security vulnerabilities. The study shows that 34% of academic CVEs are unconfirmed or disputed (Takeaway #2).
• Engage in the Dispute Process: While arduous, maintainers should utilize the formal mechanisms for disputing or rejecting CVEs if they believe an assignment is invalid. A simplified dispute process would further empower maintainers (Recommendation 6).
• Improve Internal CNA Processes: For organizations acting as their own CNAs, maintaining clear and consistent procedures for vulnerability verification and CVE assignment is crucial to avoid low-quality or disputed CVEs.

For the CVE Ecosystem (MITRE, CNAs, CNA-LRs):

• Enhance Verification by CNA-LRs: CNA-LRs should explore ways to implement additional verification steps for bugs before assigning CVEs, even acknowledging the inherent challenges of not owning the code (Recommendation 5).
• Simplify and Publicize the Dispute Process: The current multi-stage dispute process is ineffective and often unknown to maintainers. A more transparent, efficient, and well-documented appeal process is needed to allow maintainers to effectively challenge questionable CVE assignments, thereby reducing the prevalence of bogus CVEs (Recommendation 6).

In essence, the defensive implications call for a collective shift towards greater scrutiny, clearer communication, and a more accurate understanding of CVEs across the entire security community, moving beyond their superficial appeal as badges of honor to their true purpose as mere identifiers.

Key Takeaways

• The use of CVEs in academic security papers has significantly increased over the last 25 years, particularly in bug-finding research areas like fuzzing and web security.
• A substantial portion (34%) of CVEs claimed in academic papers published between 2020 and 2024 were found to be unconfirmed by maintainers or explicitly disputed, challenging their perceived real-world impact.
• CNA-LRs (e.g., MITRE) are almost exclusively the source of CVEs that project maintainers disagree with, highlighting issues with their verification processes compared to vendor-specific CNAs.
• Widespread misconceptions exist within academia regarding CVEs; 54% of surveyed participants incorrectly believe vulnerability verification is a mandatory step in the CVE assignment process.
• Despite academic reviewers' stated preference against CVEs influencing acceptance, the presence of CVEs significantly and positively affects reviewer impressions, making paper acceptance more likely.
• The academic community should adopt more rigorous impact metrics beyond mere CVE counts, focusing instead on detailed vendor responses, demonstrated exploitability, and evaluation on popular, real-world targets.

About the Speaker(s)

This detailed technical article is based on a distinguished paper co-authored by a large team of researchers from prominent European academic institutions.

• Moritz Schloegel, Lukas Gerlach, Leon Trampert, Michael Schwarz, and Thorsten Holz are affiliated with the CISPA Helmholtz Center for Information Security. CISPA is a national research institution for cybersecurity and privacy, known for its cutting-edge work in various areas of IT security.
• Daniel Klischies is associated with Ruhr University Bochum, a leading German university with a strong focus on IT security research.
• Simon Koch, David Klein, and Malte Wessels, and Martin Johns are from TU Braunschweig (Technische Universität Braunschweig), another prominent German technical university engaged in significant security research.
• Mathy Vanhoef and Jo Van Bulck are affiliated with KU Leuven / DistriNet, a highly respected Belgian university and research group known for its contributions to computer security, particularly in network and system security.

The collaborative nature of this work, bringing together experts from multiple institutions, underscores the interdisciplinary and comprehensive approach taken to study the complex issue of CVE usage in academia.