History and Significance of the TTBR and PEASWG

History and Significance of the TTBR and PEASWG

Speakers: Debra Bowen

Conference: Voting Village

YouTube: https://www.youtube.com/watch?v=F_Xz9rMgWzE

Overview

This talk, delivered by the Honorable Debra Bowen, former Secretary of State of California, delves into the foundational work that reshaped election security in the United States: the Top-to-Bottom Review (TTBR) and the Post-Election Audits Standards Working Group (PEASWG). Bowen, who served as California's Secretary of State from 2007 to 2015, recounts the political courage and technical foresight required to challenge the status quo of voting systems. Her initiatives were pivotal in exposing vulnerabilities in election technology and establishing robust, evidence-based methodologies for ensuring election integrity.

The talk highlights how these efforts, undertaken nearly two decades ago, laid the groundwork for modern election security practices, including the widespread adoption of Risk Limiting Audits (RLAs) and a shift in thinking from expecting perfect software to designing software-independent election processes. Bowen's work not only exposed critical flaws in then-prevalent Direct Recording Electronic (DRE) voting machines but also galvanized a community of experts and officials to demand greater transparency and verifiability in democratic processes. This historical perspective is crucial for understanding the evolution of election cybersecurity and the ongoing challenges in safeguarding democratic institutions.

Background

▶ Watch: Introduction to Secretary Debra Bowen and her legacy (0:00)

The genesis of the TTBR and PEASWG can be traced to a period of profound disillusionment and concern surrounding election integrity in the early 2000s. The contentious 2000 presidential election in Florida, marred by ballot design issues and recounts, and the equally problematic 2004 Ohio election, which featured the controversial Diebold voting machines and accusations of partisan influence, severely eroded public confidence. In response to these events, the federal government passed the Help America Vote Act (HAVA), which injected significant funding into states to upgrade voting systems. This influx of cash, however, often led states to purchase sophisticated, paperless DRE machines that were aggressively marketed as modern solutions but frequently lacked robust security and auditability. Bowen describes these DREs as "vaporware," noting that the technology often lagged behind marketing claims.

At the time, election officials, including Secretaries of State, generally lacked the technical expertise to critically evaluate these systems, often relying on vendor assurances. This created a "black box" problem, where the inner workings of voting machines were opaque to the public and even to many election administrators. A turning point for Bowen personally came in 2006 when she encountered the work of Blackbox Voting, a nonprofit led by Bev Harris. Harris had famously acquired and publicized the source code for Diebold voting machines, exposing significant vulnerabilities after it was mistakenly posted on a public company website. Reading this material convinced Bowen of the urgent need for comprehensive, independent review, leading her to run for Secretary of State on a platform centered on conducting a "top-to-bottom review" of California's voting systems.

Bowen's personal background, though not initially in election administration, provided a unique foundation for this challenge. With a law degree from the University of Virginia and a history as an early technology adopter (including introducing legislation to put the California legislature online in 1992 and making digital signatures acceptable), she possessed both the legal acumen and an understanding of complex IT projects and the importance of standards. Her college programming classes in FORTRAN and COBOL, while outdated, instilled in her an understanding of computer logic and logic trees, which proved invaluable for identifying potential failure points in systems. Coupled with a strong sense of civic duty and a history of standing up to authority, Bowen was uniquely positioned to tackle what many considered an intractable problem.

Key Findings

▶ Watch: TTBR's impact: exposing weaknesses, eliminating DREs (2:00)

The Top-to-Bottom Review (TTBR) yielded several critical findings that fundamentally altered the perception and practice of election security. The most significant discovery was the documentation of profound weaknesses in virtually every aspect of the voting systems then in use across California. These weaknesses spanned from software vulnerabilities to glaring physical security flaws. The TTBR, which involved independent experts, gained unprecedented access to source code, a rarity in the election cybersecurity community at the time. This access revealed that the DRE voting systems were highly susceptible to both hacking and simple errors.

A crucial insight from the TTBR was the realization that relying solely on software to tally votes was inherently risky. As Bowen noted, "if you ask a computer to give you the total of something that it's already tallied, it's going to just spit out the same result." This underscored the urgent need for an independent mechanism for auditing election results, beyond what the voting machines themselves reported. This finding directly paved the way for the establishment of the Post-Election Audits Standards Working Group (PEASWG), which Bowen convened. The PEASWG's primary contribution was to formalize and place the concept of post-election audits on a firm scientific foundation.

The PEASWG, under the guidance of experts like Professor Philip Stark, eventually led to the creation of Risk Limiting Audits (RLAs). RLAs represent a paradigm shift, moving away from the unattainable goal of "perfectly secure" software towards a more robust concept of software independence and evidence-based elections. RLAs provide a statistically sound method to verify election outcomes by manually examining a sufficient number of paper ballots to gain high confidence that the reported results are accurate, even if the underlying software contains bugs or malicious logic. This work established the "gold standard" for election audits and fundamentally changed how the election community approaches security, focusing on verifiability rather than just system certification.

Technical Deep Dive

▶ Watch: TTBR's legacy: software independence, evidence-based elections (3:00)

The Top-to-Bottom Review (TTBR) employed a multi-faceted approach to assess the security of California's voting systems, combining rigorous technical analysis with practical penetration testing. The methodologies included:

1. Document Review: A comprehensive analysis of system specifications, user manuals, and existing security documentation.
2. Source Code Review: This was a groundbreaking aspect, as vendors were "strong-armed" into providing their proprietary source code to independent experts. This allowed for an unprecedented examination of the underlying logic and potential vulnerabilities within the voting machine software.
3. Red Team Penetration Testing: Independent security researchers were given an "open-ended invitation to hack into a system." This involved active attempts to exploit vulnerabilities, simulate attacks, and demonstrate how an adversary could compromise the integrity of the voting process.
4. Accessibility Review: Beyond security, the TTBR also assessed how voting systems served voters with disabilities, particularly those relying on touchscreen machines, which were often the only accessible option in some precincts.

To facilitate this review, Bowen's office created a highly controlled testing environment, akin to a "clean room" in a manufacturing facility. Strict physical security measures were enforced, including sign-in/out procedures for all personnel, and even checking for vulnerabilities like removable ceiling panels. For experts working remotely, Secure Enforcement Facilities (SNIFF or SPEC) were established at university offices to ensure the integrity of the source code review process, mirroring practices from highly sensitive environments.

The technical findings from the source code and red team reviews were alarming. One major concern was the inherent vulnerability of Direct Recording Electronic (DRE) machines, which lacked a voter-verifiable paper trail. These systems, designed to be paperless, offered no independent means of auditing the electronic tallies. The red team testing exposed critical flaws, some surprisingly simple. For example, one vendor's voting equipment was housed in a clamshell case with a front lock, but the case was also secured by Phillips head screws. An attacker with a common Phillips head screwdriver could remove the screws, use the lock as a hinge, and gain full access to the machine's internal components, bypassing the intended security.

Another significant vulnerability highlighted by Bowen was the ease with which programming cards for Diebold machines could be acquired. She personally purchased three genuine Diebold programming cards on eBay for $29.95, directly contradicting vendor claims of strict control over access to such critical components. This demonstrated a pathway for insiders or external actors to potentially upload malicious software or a virus onto a voting machine. Such a virus, once uploaded, could then spread to other machines when results were aggregated at the county level or when machines were prepared for subsequent elections.

The PEASWG, building on the TTBR's findings, then focused on developing a robust, statistically sound method for post-election audits. This led to the development of Risk Limiting Audits (RLAs), primarily championed by Philip Stark. RLAs are designed to provide a high statistical confidence that the reported election outcome is correct by comparing a sample of paper ballots to the machine tallies. If the discrepancy exceeds a predefined risk limit, more ballots are reviewed, systematically reducing the risk of an incorrect outcome until the desired confidence level (e.0.00009) is achieved. This shift from simply counting all ballots or conducting fixed-percentage audits to a risk-based, adaptive approach was a monumental technical and methodological advancement in election security.

Demo / Proof of Concept

▶ Watch: Bowen's essential qualities: strong spine and caring (7:00)

While Debra Bowen's talk did not feature a live "demo" in the traditional sense, she effectively illustrated key vulnerabilities and the impact of the TTBR through compelling anecdotes and real-world examples that served as proofs of concept.

One powerful demonstration of a critical flaw was Bowen's personal anecdote about purchasing Diebold programming cards on eBay. For a mere $29.95, she acquired three genuine cards, directly undermining the vendor's assurances of stringent security and controlled access. This practical experiment highlighted how easily an unauthorized individual could obtain the tools necessary to potentially tamper with voting machine programming, demonstrating a clear lack of physical and procedural security.

Another significant "proof of concept" came from the red team penetration testing conducted during the TTBR. Bowen recounted a particularly simple yet impactful vulnerability: a voting system vendor used a clamshell case secured by a front lock, but the case itself was attached by standard Phillips head screws. This meant that anyone with a common Phillips head screwdriver could easily remove the screws, effectively bypassing the lock by using it as a hinge, and gaining unauthorized access to the voting machine's internal components. This revealed a fundamental flaw in physical security design that could allow for the direct manipulation of the machine's hardware or software.

Beyond the California review, Bowen also referenced a notable proof of concept from outside her immediate work: an internet voting test briefly attempted by Washington D.C. Students from the University of Michigan, who had previously worked on the TTBR, demonstrated its insecurity by programming the system to play the Michigan fight song instead of displaying an "I Voted" message. This creative hack served as a clear, undeniable demonstration that the internet voting system was not secure, effectively shutting down the initiative. These examples, though recounted rather than performed live, powerfully conveyed the practical exploits and security weaknesses that the TTBR and related efforts sought to address.

Defensive Implications

▶ Watch: Bowen's early tech advocacy: putting legislature online (8:00)

The findings and recommendations of the TTBR and the PEASWG led to significant and lasting defensive implications for election security, fundamentally reshaping how elections are administered in California and influencing national standards.

The most immediate and impactful action taken by Secretary Bowen was the decertification of most touchscreen voting systems (DREs). Recognizing their inherent lack of a voter-verifiable paper trail and susceptibility to undetectable manipulation, DREs were largely phased out. A limited exception was made for one DRE per polling place to ensure accessibility for disabled voters, but even these were subject to new, stringent security protocols. This move underscored the principle that paper ballots are the bedrock of auditable elections, providing an independent record that can be manually verified.

Beyond decertification, Bowen imposed a comprehensive series of new security requirements. These included:

• Enhanced Physical Security: Equipment, especially when stored between elections, had to be secured in warehouses with physical security measures. This included using banker tags to lock equipment cases, ensuring that any tampering would be evident.
• Equipment Tracking: Detailed logs were mandated for each machine, including tracking its unique number, who opened it, and who closed it. This "paperwork," as Bowen called it, was crucial for maintaining a chain of custody and preventing machines from going missing or being tampered with undetected. She cited an instance where a vendor's machine went missing in a county, highlighting the real-world risk.
• Procedural Safeguards: Drawing lessons from the banking industry, requirements for double signatures on critical actions were introduced, adding a layer of accountability and making it more difficult for a single insider to compromise a system.
• Improved Training for Poll Workers: Recognizing that human elements are often the weakest link, efforts were made to improve the training for election officials and poll workers, who often lack technical backgrounds and may not undergo rigorous background checks.

The adoption of Risk Limiting Audits (RLAs), born from the PEASWG, is perhaps the most significant long-term defensive measure. RLAs provide a statistically robust method to ensure that election results are accurate, even if the underlying voting machines have software bugs or malicious logic. This shifts the focus from trying to perfect inherently fallible software to establishing a verifiable, evidence-based process that can detect and correct errors. By systematically examining paper ballots until a high level of confidence is achieved, RLAs offer a powerful safeguard against both accidental errors and deliberate attacks on electronic vote tabulation.

Bowen emphasized the need to treat election systems as the "keys to the kingdom"—the fundamental infrastructure of democracy. This perspective calls for applying the highest standards of security, comparable to those in the banking and finance industries, including robust physical security, strict chain of custody, and multi-person controls. The ultimate defensive implication is a move towards software independence, where the accuracy of election results does not depend solely on the perfect functioning of software, but can be verified by independent, auditable means, primarily through voter-marked paper ballots and RLAs.

Key Takeaways

• Necessity of Independent Review: The Top-to-Bottom Review (TTBR) demonstrated the critical importance of independent experts, free from government or vendor influence, to thoroughly examine voting systems, including access to source code.
• Vulnerability of DREs and the Power of Paper: The TTBR exposed profound weaknesses in Direct Recording Electronic (DRE) voting machines, leading to their widespread decertification and reinforcing the necessity of voter-verifiable paper ballots as the foundational element for auditable elections.
• Shift to Software Independence and Risk Limiting Audits: The work of the Post-Election Audits Standards Working Group (PEASWG), particularly the development of Risk Limiting Audits (RLAs), transformed election security by providing a statistically rigorous, evidence-based method to verify election outcomes, acknowledging that software will always have bugs.
• Physical and Procedural Security are Paramount: Simple vulnerabilities, like easily accessible programming cards or voting machine cases secured by common screws, highlighted that physical security, chain of custody, and robust procedural safeguards (e.g., double signatures, detailed tracking) are as crucial as software security.
• Courageous Leadership and Public Engagement: Challenging established norms and powerful vendors required immense political courage from Secretary Bowen, who faced lawsuits and criticism. Her experience underscores the need for elected officials with a strong spine and a commitment to transparency, and the importance of citizens engaging with election officials to voice concerns.
• Elections are Unique and Non-Fixable with Money: Unlike banking, where errors can be rectified financially, election errors are "not fixable" with money and can undermine the legitimacy of democracy, requiring a "get it right on the first try" approach through rigorous security and audits.

About the Speaker(s)

Debra Bowen served as the elected Secretary of State of California from 2007 to 2015, holding the position for two terms. Arguably one of the most powerful election officials in the United States due to California's size, she made indelible contributions to election integrity and cybersecurity. Prior to her role as Secretary of State, Bowen had a distinguished career in the California legislature, serving three two-year terms in the State Assembly and two four-year terms in the State Senate, starting in 1992. During her legislative tenure, she was an early adopter and champion of technology, introducing bills to put the California legislature online and to make digital signatures acceptable. She also worked on public interest cases and had a background in computer logic from college programming classes.

Bowen's decision to run for Secretary of State was spurred by her deep concerns about voting machine vulnerabilities, particularly after encountering the work of Blackbox Voting. As Secretary of State, she fearlessly convened the Top-to-Bottom Review (TTBR), a panel of independent experts, and the Post-Election Audits Standards Working Group (PEASWG), initiatives that faced significant political and legal challenges. Her courage and commitment to transparent, secure elections earned her the Profile in Courage Award from the John F. Kennedy Presidential Library and Museum. Now retired from public office, Bowen continues to advocate for election integrity as a "citizen," emphasizing the ongoing need for rigorous review and robust safeguards in democratic processes.