Was This You? Investigating the Design Considerations for Suspicious Login Notifications

Was This You? Investigating the Design Considerations for Suspicious Login Notifications

Speakers: Sena Sahin

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=180W2E1lGTo

Overview

In an increasingly interconnected digital landscape, account security remains a paramount concern for users and service providers alike. A critical first line of defense against unauthorized access is the suspicious login notification (SLN), an automated alert triggered when a user's account is accessed from an unusual location, device, or time. Despite their ubiquity, these notifications vary significantly in their design, tone, and the level of detail they provide, often leading to user confusion and potentially ineffective responses. This talk by Sena Sahin, presented at the NDSS Symposium, delves into the intricate world of SLNs, aiming to understand how users interact with them, identify their legitimacy, and decide on appropriate actions.

Sahin's research tackles a fundamental problem: the lack of standardization in SLN design, which hinders user comprehension and response efficacy. By examining real-world notifications and conducting in-depth user interviews, the study uncovers crucial insights into user preferences for the content, format, and actionable advice within these alerts. The findings offer a compelling call for a more thoughtful, user-centric design approach, highlighting the importance of holistic information, clear language, and trustworthy action prompts to empower users in safeguarding their digital identities. This work is essential for anyone involved in designing or implementing account security measures, from platform developers to security awareness trainers.

Background

▶ Watch: Introduction and research questions (0:00)

The proliferation of online services has made robust account security indispensable. Suspicious login notifications serve as a vital early warning system, alerting users to potential compromises. However, the effectiveness of these notifications hinges on their ability to convey critical information clearly and unambiguously, guiding users to take the correct protective actions. The existing landscape of SLNs is characterized by a significant lack of uniformity. As Sahin's initial analysis revealed, notifications from various websites exhibit considerable variations in their components, the granularity of information provided, and their overall design decisions. This inconsistency can be a major source of user bewilderment, making it difficult for individuals to discern whether an alert is legitimate, whether the reported login is genuinely suspicious, and what steps they should take.

Prior to this research, there was a gap in understanding the specific design elements that capture user attention, help them verify the legitimacy of the notification itself, and enable them to make informed decisions about the reported login activity. Many organizations likely design their SLNs based on internal data or assumptions, but a comprehensive, user-centric investigation into these design considerations was overdue. This study sought to bridge that gap by systematically investigating user preferences and behaviors. The research employed a two-phase approach. First, the team collected data by creating test accounts on the top 100 websites identified by Tranco and intentionally triggering SLNs. This yielded 21 distinct email notifications, which served as a foundation for understanding current industry practices and their inherent variations. The second phase involved recruiting US-based participants, aged 18 and above, via the Prolific platform for semi-structured interviews. These interviews explored participants' prior experiences with SLNs, their typical workflows upon receiving such notifications, and critically, allowed them to design their own ideal suspicious login notifications based on design options derived from the real-world data. The subsequent discussion and critique of these user-designed notifications provided rich qualitative data, directly informing the study's key findings.

Key Findings

▶ Watch: Methodology: User interviews and design insights (4:00)

The research primarily focused on addressing two core questions: how users identify if a login is legitimate or malicious (Research Question 3), and how they decide on and execute subsequent actions (Research Question 4). The findings reveal a clear demand for comprehensive and contextually rich information, alongside a strong aversion to certain design patterns.

Regarding identifying legitimate or malicious logins, participants consistently expressed a need for specific data points:

• Account Name: A majority of participants (over 50%) desired the inclusion of their account name, with a strong preference for their username, to immediately link the notification to a specific online identity.
• Browser Information: Most participants wanted to see the browser vendor name if the login occurred via a web browser, providing crucial context about the access method.
• IP Address: While several participants requested the IP address, they specifically preferred the IPv4 format. The speaker noted that this preference highlighted a "limited understanding of the end users towards the IP addresses," suggesting that users perceive IPv4 as more readily interpretable or familiar, even if they don't fully grasp its technical implications. They saw it as an "additional layer of verification."
• Device Information: All participants emphasized the importance of device brand and model to determine login legitimacy, as this is a highly personal and recognizable identifier.
• Date: All participants wanted the date of the login attempt. For US-based participants, the month-day format was preferred, though the study acknowledges that users in other regions might favor a day-month format.
• Operating System (OS): Tech-savvy participants specifically requested the operating system name for additional verification.
• Time, Time Zone, and Location: All participants deemed time, time zone, and location information essential. They preferred the 12-hour format for time due to familiarity in the US. For location, a textual format was universally preferred, with some expressing interest in additional map visualizations (pinpoint or circular). However, participants also critically highlighted that IP-based location data is not always accurate, underscoring the need for other holistic information.

When it came to deciding on and taking actions based on the notification, user preferences diverged significantly depending on whether the login was legitimate or malicious:

• Legitimate Login: For logins users identified as legitimate (e.g., they themselves logged in from a new device), the majority desired an explanation stating that no action was needed, explicitly to "avoid unnecessary password changes."
• Malicious Login: For confirmed malicious logins, all participants wanted a clear password change suggestion. Crucially, they also appreciated additional information stating that a password change would result in "logging out all active sessions," providing transparency about the immediate security impact.
• Future Account Security: Several participants expressed interest in options for enhancing future account security, such as enabling two-factor authentication (2FA) or instructions to review account activities and general security pages.
• Action Prompts (Links/Buttons): A critical finding was the overwhelming suspicion participants held towards clicking any links or buttons within the notification email. While a majority preferred links over buttons if a redirect was necessary, they insisted that the "instructions should not require them to click any links or buttons." This highlights a significant user security concern, suggesting that direct action within the email should be minimized or avoided, and alternative methods for users to take action (e.g., logging in directly to the service) should be emphasized.

Technical Deep Dive

▶ Watch: Key findings: User preferences for login legitimacy details (4:30)

The technical depth of this research primarily resides in its meticulous analysis of existing SLN designs and the subsequent derivation of user-centric design principles. The initial phase involved a comprehensive review of 21 distinct email notifications collected from the top 100 websites (by Tranco ranking). This analysis revealed a profound lack of standardization, with significant variations across several key components:

• Sender Details: Observed variations included sender names featuring a username, security-related terms (e.g., "Security Alert"), or utilizing a registered subdomain name to convey legitimacy.
• Subject Line: Subject lines differed in their tone (e.g., urgent vs. informative), personalization (e.g., including the user's name), and the level of detailed information provided upfront.
• Login Details: The granularity and presentation of login information varied widely. Some notifications offered minimal detail, while others attempted to provide more context.
• Action Prompts: These ranged from suggesting immediate actions related to current account security to outlining future preventative steps, or providing general instructions.
• Legitimacy Signals: Notifications incorporated various signals such as logos, personalized greetings, explicit email legitimacy warnings, and legal disclaimers to build trust and help users distinguish genuine alerts from phishing attempts.

From the semi-structured interviews, the study constructed a detailed understanding of preferred holistic components for a truly effective SLN. Users advocated for a design that integrates multiple data points to build a robust picture of the login event, acknowledging that no single piece of information is infallible.

For instance, while an IP address was desired for verification, participants recognized its potential inaccuracy for location. Therefore, pairing it with other details like device brand/model, operating system, and browser vendor name becomes crucial. The preference for IPv4 format for IPs, despite potential technical limitations in availability or accuracy, underscores the importance of presenting information in a format that users perceive as understandable and actionable. Similarly, the demand for textual location data alongside optional map visualizations demonstrates a need for both unambiguous information and supplementary contextual aids. The explicit mention of month-day vs. day-month format for dates highlights a critical, often overlooked, cultural and regional design consideration.

A significant "technical" aspect of the findings lies in the psychological impact of language tone and framing. The study found that ambiguous phrases, such as "you signed in," could lead users to automatically assume the login was legitimate, even if they couldn't recall performing the action. This suggests that the precise wording and framing of notifications are not merely stylistic choices but fundamental design considerations that directly impact user interpretation and behavior. The suggested design emphasizes suspicion of activity without forcing immediate, urgent action or coercing clicks, instead prioritizing detailed login information and an explicit "legitimacy warning" within the email itself. This approach shifts the burden of verification away from risky in-email interactions and towards informed user judgment based on comprehensive data.

Furthermore, the study implicitly highlights the technical challenge of accurately gathering and presenting all these desired data points (e.g., precise device models, accurate IP-based location, OS versions) across diverse login scenarios and user environments. While not explicitly detailing code or protocols, the findings provide a blueprint for the data points that security systems should strive to collect and present in their notifications, thereby influencing the technical specifications of logging and alerting mechanisms.

Demo / Proof of Concept

▶ Watch: Suggested design principles for effective notifications (8:50)

While the talk did not feature a live, interactive demonstration of a new security tool or a working exploit, Sena Sahin presented a conceptual "suggested design" for suspicious login notifications. This proposed design effectively serves as a proof of concept for the research's findings, visually encapsulating the user preferences and best practices identified during the study.

The suggested design emphasizes several key principles:

1. Highlighting Suspicion, Not Urgency: The notification should clearly indicate that an activity is suspicious without creating a sense of panic or forcing immediate, unthinking action.
2. No Forced Immediate Action: Users should not feel compelled to click links or buttons within the email itself. The design prioritizes providing information for the user to make an informed decision rather than directing them to an external site via a potentially risky click.
3. Detailed Login Information: The design incorporates all the critical components identified by participants: account name (username), device brand and model, browser information, date, time, time zone, and location (textual with optional map visualization).
4. Email Legitimacy Warning: A prominent section within the email itself provides guidance on how users can verify the legitimacy of the notification, helping them distinguish genuine alerts from phishing attempts. This might include instructions to never click links directly but instead navigate to the service's website directly.

This conceptual design visually demonstrates how a thoughtful, holistic approach can address user needs, enhance clarity, and build trust, thereby improving the overall effectiveness of suspicious login notifications.

Defensive Implications

▶ Watch: Main takeaways: User familiarity drives preferences (9:40)

The findings from this research offer crucial actionable insights for security professionals, product designers, and platform developers responsible for user account security. Implementing these recommendations can significantly enhance the effectiveness of suspicious login notifications as a critical defensive mechanism:

1. Standardize Notification Designs: The most overarching implication is the urgent need for standardization in SLN design. Security teams should advocate for industry-wide best practices or, at minimum, ensure consistency across their own multiple services and properties. A predictable format reduces user cognitive load and helps them quickly identify legitimate alerts.
2. Provide Holistic and Detailed Login Context: Defenders must strive to include comprehensive and accurate login details. This includes account name (username), device brand and model, browser vendor, operating system (for tech-savvy users), date, time, time zone, and textual location data. Relying on a single, potentially inaccurate data point like IP-based location is insufficient.
3. Prioritize Clarity and Unambiguous Language: The language used in SLNs must be crystal clear and avoid any ambiguity. Phrases like "You signed in" should be replaced with more neutral or interrogative language (e.g., "Is this you? An unusual login was detected..."). The tone should be informative and helpful, not alarming or coercive.
4. Minimize Clickable Elements in Email: Given the high user suspicion of links and buttons within SLNs, defenders should redesign notifications to minimize or eliminate the need for in-email clicks. If links are absolutely necessary, they should clearly display the destination URL and users should be explicitly advised to navigate directly to the service's website to take action, rather than clicking a link in the email.
5. Educate on Email Legitimacy Verification: SLNs should proactively include instructions on how users can verify the email's authenticity. This could be a dedicated section advising users to check the sender's email address, look for specific security warnings, or, crucially, to always go directly to the service's official website to review account activity.
6. Tailor Actions to Incident Type: Notifications should clearly differentiate between actions needed for legitimate vs. malicious logins. For legitimate but unusual logins, explicitly state "no action needed" to prevent unnecessary password changes. For malicious activity, provide clear, concise instructions for a password reset and inform users that this action will log out all active sessions.
7. Consider Cultural and Regional Preferences: Designs should account for diverse user groups, including preferences for date and time formats (e.g., month-day vs. day-month, 12-hour vs. 24-hour clock) and cultural nuances in language.
8. Account for Shared Accounts: While not deeply explored, the mention of shared accounts suggests that notifications could potentially incorporate information that helps users distinguish activity by a legitimate co-user from a malicious actor.
9. Empower Users with Future Security Options: Offer clear pathways within the service (not necessarily the email) for users to enable 2FA or review their account's security settings and activity logs.

By adopting these defensive strategies, organizations can transform suspicious login notifications from a source of confusion into a powerful and trusted tool for user empowerment and enhanced account security.

Key Takeaways

• Lack of Standardization: The current landscape of suspicious login notifications suffers from a significant lack of standardization across different websites and services, leading to user confusion and diminished effectiveness.
• Holistic Information is Key: Users demand comprehensive and diverse data points (device, browser, OS, accurate location, time, account name) presented holistically to confidently verify login legitimacy, as no single piece of information is always accurate or sufficient.
• Language Matters Critically: Ambiguous language, such as "you signed in," can mislead users into assuming a login is legitimate even when it's not, underscoring the need for clear, unambiguous, and carefully framed messaging.
• User Distrust of In-Email Actions: Users are highly suspicious of clicking links or buttons within suspicious login notifications, highlighting a critical need for alternative, secure methods for users to take action or verify information directly on the service's website.
• Thoughtful Design for Trust: Effective SLNs require a thoughtful design that emphasizes suspicion without creating panic, provides detailed context, includes explicit legitimacy warnings, and accounts for user preferences regarding data formats and presentation.
• Defensive Imperative: Security professionals must prioritize standardizing SLN designs, providing rich and accurate login details, utilizing clear language, and guiding users to take actions securely outside of the email to build trust and fortify account security.

About the Speaker(s)

Sena Sahin is a researcher who presented her work on "Was This You? Investigating the Design Considerations for Suspicious Login Notifications" at the NDSS Symposium. Based on the transcript, her research focuses on user interaction with security notifications and improving their design for better effectiveness in account security. The provided metadata and transcript do not specify her title or affiliation.