Temporary Access to the Cloud: A Case Study

Temporary Access to the Cloud: A Case Study

Speakers: Tomas Rabczak

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=o4eoE7cF56M

Overview

This talk, presented by Tomas Rabczak, a Staff Software Engineer at Chime, delves into the critical security challenge of managing employee access to cloud resources and the development of an internal solution called "Access Service." The presentation outlines a comprehensive case study on how Chime transitioned from a culture of permanent access to a model of temporary, just-in-time access, significantly reducing its attack surface. Rabczak highlights the pervasive issue of the "human element" in data breaches, citing statistics and real-world incidents to underscore the necessity of such a system.

The core of the talk revolves around the design, development, and rollout of Access Service, an application built on top of Octa, Chime's single sign-on provider. This service aims to empower engineers with self-service access while enforcing security best practices, such as streamlined approval flows, emergency access mechanisms, and activity-based access refresh. The initiative was inspired by a similar service developed by Segment, emphasizing the value of cross-company collaboration in tackling common security problems.

The importance of this work lies in its direct impact on an organization's security posture. By minimizing the duration and scope of access, companies can drastically reduce the potential blast radius of compromised employee accounts. Rabczak's detailed account of Chime's journey, including the technical architecture, cultural challenges, and the crucial role of product metrics, provides a valuable blueprint for other organizations grappling with similar access management complexities in the cloud.

Background

▶ Watch: Problem Statement: 68% of breaches involve human element, leading to major in... (1:00)

The impetus for Chime's Access Service stemmed from a stark reality: employees frequently represent a significant attack vector in data breaches. Tomas Rabczak cited a 2024 data breach report indicating that 68% of breaches involved a human element, specifically referring to instances where individuals are exploited for their access rather than acting as malicious insiders. This alarming statistic is consistently backed by headlines detailing high-profile incidents, including hacks at Uber, Microsoft (where an engineer's account was compromised), Zendesk (falling for a phishing attack), Twilio, and MGM Resorts (reportedly shut down by a 10-minute phone call). A particularly concerning example highlighted was a hacker using a deepfake of an employee's voice to breach an IT company, as reported by Retool, showcasing the evolving sophistication of these attacks.

Recognizing this pervasive problem, Chime sought to understand its own exposure. The inspiration for building an internal solution came from a blog post by Segment titled "Access Service: Temporary Access to the Cloud," which detailed a similar service. Chime's team, led by a colleague named Mun, decided to embark on building their own version. A notable aspect of their initial phase was proactive cross-company collaboration: they engaged in a call with Segment to gather insights, recommendations, and discuss the build-versus-buy dilemma.

Data gathering at Chime revealed a significant "gray area" of unused access. An analysis of access across top Octa groups for applications like AWS, Snowflake, and StrongDM showed a staggering percentage of users who had access but had not logged in or used that access within the last 30 days. This "gray part" represented the organization's attack surface, and the primary goal of Access Service was to minimize this area as much as possible, shifting from a default of permanent access to a model where access is granted only when needed and for a limited duration.

Key Findings

▶ Watch: Technical Foundation: Leveraging SCIM with OCTA for streamlined, custom-code-... (6:00)

The development and rollout of Chime's Access Service yielded several critical findings and contributions:

• Significant Reduction in Attack Surface: The most impactful finding was the demonstrable reduction in the "gray area" of unused access. For one application, after transitioning from permanent to temporary access, approximately 50% of access dropped off within 30 days. This 50% reduction proved consistent across various migrations, directly validating the service's value in minimizing potential exposure.
• Validation of Just-in-Time Access: The project successfully demonstrated that a just-in-time (JIT) access model, where users request access only when needed, is not only feasible but highly effective in a large engineering organization.
• Power of SKIM-based Management: Leveraging SKIM (System for Cross-domain Identity Management) was a foundational decision. It allowed Chime to integrate with various applications without writing custom code for each, simplifying user and group management through Octa. This also pushed existing applications to adopt SKIM, leading to a more standardized and secure identity system.
• Importance of Product Metrics in Security: Rabczak emphasized that product metrics are crucial for security initiatives. Beyond simply tracking security posture, metrics like "permanent versus temporary access" (the "North Star metric"), approval times, and stale requests provided tangible evidence of the application's impact, helped monitor its health, and facilitated communication with leadership.
• Successful Cultural Shift: Despite initial resistance to moving away from persistent access, a carefully planned, iterative rollout, combined with leadership buy-in and features addressing user friction (like activity-based refresh and instant access for non-production environments), successfully fostered a culture of temporary access.
• Strategic Build vs. Buy Decision: Chime opted to build Access Service internally primarily due to security concerns regarding handing over Octa super admin API keys (the "keys to the castle") to third-party vendors. This decision allowed them to maintain full control over a highly sensitive system.
• Activity-Based Access Refresh: This feature, which automatically extends access if it's actively used, was a key innovation in reducing user friction. It addressed valid complaints from users who frequently needed access, preventing them from having to repeatedly request extensions.
• Emergency Access with Controls: The implementation of an emergency access feature, while providing rapid access during incidents, was designed with critical safeguards, including requiring a managed device, a secondary approval, and immediate notification to the Security Operations Center (SOC) via Tines.

Technical Deep Dive

▶ Watch: Metrics for Success: Using product metrics like permanent vs. temporary acces... (18:00)

The Access Service at Chime is a sophisticated yet elegantly designed solution built to address the complexities of temporary access management in a cloud environment. Its architecture is centered around Octa, the organization's single sign-on (SSO) provider, and leverages key industry standards and internal tooling.

Core Goals and Foundation:

The service was designed with several key goals:

1. Self-Service: Empowering engineers and non-security team members to request access independently.
2. Just-in-Time Access: Granting access only when needed and for a limited duration.
3. Streamlined Approval Flows: Making it easy for users to find and obtain necessary access quickly, while ensuring proper review.
4. Quick Access for Emergencies: Providing a "break glass" scenario for critical incidents outside of normal approval channels.

The entire system is built on top of Octa, which manages user identities, applications, and groups. Octa's ability to enforce multi-factor authentication (MFA) and its support for SKIM (System for Cross-domain Identity Management) were foundational to the Access Service.

SKIM (System for Cross-domain Identity Management):

SKIM is a crucial component. It's a specification that defines a set of REST CRUD APIs for managing users and groups within an application. By utilizing SKIM, Octa can integrate with various applications, performing user management operations (like adding or removing a user from a group) in the background without requiring Chime to write custom application-specific code. This means that when a user is added to an Octa group, the corresponding access changes are automatically propagated to the integrated application via SKIM, significantly reducing development overhead and improving consistency.

Access Service Architecture:

The Access Service itself is a web service that primarily interacts with the Octa API. Its foundational API calls are add a user to a group and remove user from a group. Key architectural components include:

• Database: A Postgres database backs the service, storing internal state related to access requests, users, and groups.
• SKIM Integration: As mentioned, the service relies on SKIM to manage access to integrated applications, eliminating direct connections from Access Service to individual application APIs for user management.
• Cron Jobs: Several scheduled tasks are critical for maintaining state and functionality:
• State Sync from Octa: Regularly syncs user information (including managers) and relevant Octa groups into the Access Service database. This ensures the service has up-to-date information on who can request what and who can approve. Rabczak noted this as one of the trickier parts, and they are exploring using SKIM webhooks from Octa to move away from polling cron jobs.
• Access Expiration: A cron job expires access requests, typically after 15 minutes if not approved, or after their granted duration.
• Usage Event Fetching: This job queries Panther, Chime's Security Information and Event Management (SIEM) system, for usage events from various applications. This data is crucial for the "activity refresh" feature, which extends access if it's actively being used.
• Notifications: The service sends notifications via Slack to both requesters and reviewers. Reviewers can approve or deny access requests directly through Slack, enhancing efficiency.
• Emergency Request Feature: For critical incidents, an emergency access request can be made. This triggers a notification to Chime's Security Operations Center (SOC) via Tines, a security orchestration, automation, and response (SOAR) platform, ensuring immediate awareness and follow-up for potentially suspicious activity.

Key Implementation Features:

• Idempotent Sync Tasks: Cron jobs are designed to be idempotent, meaning they can be run multiple times without causing unintended side effects or mangling the state, which is crucial for reliability.
• Asynchronous Third-Party Calls: All interactions with third-party services, especially the Octa API, are asynchronous. This is a necessity due to the "insane limits" on the Octa API, which apply at both the application and organization levels. Hitting these limits could block requests across the entire organization, so the service includes logic to back off before reaching rate limits.
• Minimizing Third-Party Dependencies: Given the sensitive nature of the application, minimizing external dependencies was a priority to reduce the attack surface and potential vulnerabilities.
• Conditional Approvals: Reviews are only required for production groups. Access to non-production environments is typically instant, reducing friction for development and debugging tasks.
• Terraform for Configuration: To promote a "shift left" mentality and reduce the security team's management burden, all group and approver configurations are managed in Terraform. Resource owners can submit Pull Requests (PRs) to onboard applications and groups themselves.
• A Terraform configuration specifies the Octa group name, a description, an approval_type (no approval, owner/manager, or owner only), and a list of approver emails.
• These properties are stored as custom Octa group profile attributes. A Boolean flag indicates if Access Service should manage the group, an enum defines the approval type, and approvers are housed in a separate Octa group that Access Service queries.

Emergency Access Requests:

Emergency access is designed for incident response. While it provides rapid access, it's not instantaneous for production groups. It requires the request to be made from a managed device (via VPN) and still needs approval from another person (e.g., an incident coordinator). The SOC is notified via Tines to monitor these requests.

Access Request State Diagram:

An access request progresses through several states:

• Pending: Initial state after submission.
• Cancelled: If the requester cancels.
• Review Decision: Awaiting approval or denial.
• Approved: Once approved by a reviewer.
• Processing: A transitory state added recently to handle Octa API rate limits. Requests move to a background async job for processing, preventing UI hangs.
• Active: Access has been granted.
• Expired: Access duration has ended.
• Revoked: Access manually removed by a user, manager, or resource owner.

Slack Integration:

The Slack integration is robust. Reviewers receive interactive messages to approve or deny requests, with denial requiring a reason. Once a request is reviewed, all related Slack notifications are updated to reflect the decision, preventing multiple reviewers from acting on the same request. Requesters receive notifications when access is granted, warnings before expiration, and confirmation upon expiration, with a link to re-request.

Audit Events:

Comprehensive audit events are captured for all actions within Access Service. These are crucial for compliance auditors and for debugging. Users can also view their own audit history, and the data can be exported to CSV.

Octa API Rate Limits:

A significant challenge encountered was the Octa API rate limits, which exist at both the application level and, critically, the organization level. Hitting the organization-level limit could impact any application using an Octa API key across the entire company. This necessitated careful design with asynchronous calls and proactive back-off logic to avoid service disruption.

Usage Events for Activity Refresh:

For the activity refresh feature, Chime initially aimed for role-level attribution (e.g., specific AWS AssumeRole events from CloudTrail). While this was achieved for some applications like AWS and StrongDM, and partially for Snowflake (despite its complex role hierarchy), Rabczak noted that simply tracking successful authentication events (e.g., "did you log into Zendesk?") proved sufficient and much easier to implement for many applications, providing significant value without the complexity of granular role-level tracking.

Demo / Proof of Concept

▶ Watch: Cultural Shift Strategy: Roadshows and premortem for organizational buy-in an... (22:00)

Tomas Rabczak presented a live demonstration of the Access Service application, showcasing its user interface and core functionality. The demo focused on the process of requesting and managing access to a non-production Kubernetes group.

The demonstration began with the user navigating to the Access Service application. The UI presented a search capability to find specific groups. For the purpose of the demo, a Kubernetes group was selected, which was configured for instant access because it was a non-production resource.

Upon initiating a new access request, the user was prompted to provide a business justification. The demo also highlighted the option to choose between "activity-based" access (which extends if used) or "standard" access, and an option for "emergency access" (though not selected in this instance).

Crucially, once the request for the non-production Kubernetes group was submitted, access was granted immediately, becoming active without any waiting period for approval. The speed of this process was visually emphasized by showing the user toggling StrongDM (SDM), which instantly reflected the newly acquired access to the Kubernetes cluster. This demonstrated the "instant access" feature for non-production environments, significantly reducing friction for engineers.

The demo concluded by showing the user's ability to revoke their own access. Upon revocation, the access to the Kubernetes cluster was immediately removed, as reflected in the SDM interface. The speaker also noted that resource owners and managers with appropriate permissions could also revoke access. The demonstration effectively illustrated the self-service, just-in-time, and rapid nature of access management for non-production resources within the Access Service.

Defensive Implications

▶ Watch: Key Challenges: Navigating OCTA API rate limits and their organizational impact. (28:00)

The implementation of Chime's Access Service carries profound defensive implications, fundamentally reshaping the organization's security posture and operational resilience.

1. Reduced Attack Surface: The most direct defensive benefit is the significant reduction in the attack surface. By transitioning from permanent to temporary access, the "gray area" of unused, persistent permissions is drastically minimized. The observed 50% drop-off in access for migrated applications directly translates to fewer potential entry points for attackers if an employee account is compromised. This limits the blast radius of a successful breach.
2. Just-in-Time (JIT) Access: Enforcing JIT access ensures that permissions are granted only for the duration they are actively needed. This drastically shrinks the window of opportunity for attackers to exploit compromised credentials, as access automatically expires.
3. Enhanced Auditability and Visibility: The Access Service centralizes all access requests and changes, providing a comprehensive audit trail. This is critical for compliance, incident response, and debugging. The ability to export audit events to CSV is invaluable for auditors, while the integration with Panther (the SIEM) for usage events ensures a centralized, queryable log store for all access-related activities. This also forced more applications to send their logs to Panther, improving overall log centralization.
4. Improved Incident Response with Emergency Access: The "break glass" emergency access feature, while designed for rapid response during incidents, is fortified with defensive controls. Requiring requests from a managed device, a secondary approval, and immediate notification to the Security Operations Center (SOC) via Tines ensures that even in urgent situations, there are checks and balances to prevent misuse and enable rapid detection of suspicious activity.
5. Shift Left Security: By making access management self-service and integrating it into developer workflows (e.g., Terraform configurations for group onboarding), the security responsibility is "shifted left." This empowers engineers to securely obtain the resources they need, reducing bottlenecks and fostering a more security-aware culture across engineering teams.
6. Standardization through SKIM Adoption: The reliance on SKIM for application integration not only simplified development but also pushed Chime to migrate existing applications to support SKIM. This standardization leads to a more consistent, manageable, and inherently more secure identity and access management (IAM) ecosystem, reducing the complexity and potential misconfigurations associated with disparate, custom integrations.
7. Cost Savings and Resource Optimization: While not strictly a defensive measure, the ability to automatically revoke unused access can lead to cost savings by reducing the number of unnecessary seat licenses for various applications. This also means fewer active accounts that could potentially be targeted.
8. Cultural Transformation: The iterative rollout and focus on user experience helped change a culture accustomed to persistent access. By demonstrating the benefits and addressing friction points, the security team fostered a positive perception of security initiatives, making employees more likely to adhere to secure practices.

Key Takeaways

• The "human element" remains a primary attack vector: A significant percentage of data breaches involve compromised employee access, highlighting the critical need to minimize the window and scope of permissions. Unused, permanent access represents a substantial and unnecessary attack surface.
• Just-in-Time (JIT) access is a powerful defensive strategy: Implementing a self-service, temporary access solution like Chime's Access Service can drastically reduce an organization's attack surface by ensuring access is granted only when needed and for a limited duration, with observed reductions of around 50% in unused access.
• Leveraging SKIM simplifies and secures identity management: Utilizing SKIM (System for Cross-domain Identity Management) with a central SSO provider like Octa is crucial for scalable and secure access management, eliminating the need for custom code integrations for each application and promoting standardization.
• Product metrics are indispensable for security initiatives: Tracking metrics beyond traditional security indicators, such as "permanent vs. temporary access," approval times, and application usage, provides tangible evidence of a security product's value, helps monitor its health, and facilitates communication with leadership and users.
• Cultural change requires strategic rollout and user-centric features: Transitioning from a culture of persistent access to temporary access demands an iterative rollout, dogfooding, leadership buy-in, and features that address user friction (e.g., activity-based refresh, instant access for non-prod environments) to ensure adoption and positive feedback.
• Build vs. Buy decisions must prioritize security control: For highly sensitive systems like access management, the decision to build internally, especially when concerns exist about handing over "keys to the castle" (like Octa super admin API keys) to third-party vendors, can be justified to maintain maximum control and minimize risk.

About the Speaker(s)

Tomas Rabczak is a Staff Software Engineer at Chime. He was the primary speaker for this presentation, detailing his experience and the work involved in building and rolling out Chime's internal "Access Service." His role involved leading the design, development, and implementation of this critical security application, including its architecture, integration with Octa, and the various features discussed. Rabczak expressed a particular interest in the application of product metrics within the security industry, advocating for their use to demonstrate value and monitor the health of security initiatives. He also highlighted his desire to open source the Access Service, following the example of Discord's similar project.