How to Secure Cloud Machine Identities

How to Secure Cloud Machine Identities

Speakers: Komal Dhull, Nathan Brahms

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=bCeGhoULnYE

Overview

This technical article delves into the critical and often overlooked domain of securing cloud machine identities. Presented by Komal Dhull, a Founding Backend Engineer, and Nathan Brahms, Co-founder and VP of Engineering at Plerion Security, the talk highlights the escalating challenges associated with managing and protecting the identities used by services within cloud environments. As organizations increasingly adopt cloud-native architectures, the sheer volume and sensitive access of these machine identities present a significant attack surface that demands dedicated security attention.

The speakers, whose company Plerion Security specializes in securing cloud access, shared insights gleaned from their work, emphasizing the complexities and potential pitfalls of mismanaged machine identities. The session was designed for security practitioners, particularly those relatively new to the subject, offering practical, actionable steps to mitigate risks. While the principles discussed are broadly applicable across cloud platforms, the presentation specifically focused on implementation details within AWS and GCP. Attendees were promised a clearer understanding of the risks and empowerment to take concrete steps to enhance their cloud security posture.

The core of the talk revolved around four main security best practices, categorized under credentials and permissions, providing a structured approach to tackling this pervasive problem. By illustrating common vulnerabilities with practical examples and offering specific cloud-native and open-source solutions, Dhull and Brahms underscored the urgency and feasibility of implementing robust machine identity security programs. The discussion aimed to equip security teams with the knowledge to move beyond traditional human identity management and address the unique challenges posed by their machine counterparts.

Background

▶ Watch: Defining machine identities and their growing scale problem (00:40)

A machine identity is defined as the identity a service uses to execute actions within a cloud service provider. This encompasses a wide range of computational entities, including processes running on virtual machines (VMs), Kubernetes workloads, and managed services like Cloud Functions or AWS Lambda functions. Unlike human identities, which are typically fewer in number and managed through traditional user accounts, machine identities are proliferating rapidly, creating a significant scale problem. Furthermore, these identities are often deeply embedded within applications and frequently require access to highly sensitive data and critical system functions, making them prime targets for attackers.

The speakers motivated the importance of securing machine identities by referencing real-world breaches. A notable example cited was the Okta 2020 breach. In this incident, an attacker gained access to an employee's account, which contained a saved credential for a service account. This service account, in turn, had access to sensitive files within Okta's customer support system. The breach underscored how an unsecured service account credential, a form of machine identity, could be a central point of compromise, leading to unauthorized access to critical data.

To set the stage for their best practices, Dhull and Brahms provided a quick refresher on IAM (Identity and Access Management) terminology specific to AWS and GCP:

• GCP: Human identities are represented by users and groups. Machine identities are service accounts. Authorization is managed by mapping identities to roles via bindings.
• AWS: Human identities are also users and groups. Machine identities are primarily IAM roles. Authorization is determined by mapping identities to allow and deny actions via policies.

The talk emphasized a practical, scalable approach. While demonstrating concepts using "ClickOps" or console-based recommendations, the speakers also provided guidance for implementing these practices in a real security program across numerous projects and accounts. For read actions, they suggested using scripts or HTTP commands, and for write actions, they recommended Terraform configurations, ensuring that the advice was applicable for both manual and automated security operations.

Key Findings

▶ Watch: Demonstration of long-lived credential compromise (04:00)

The core contribution of the talk was the articulation and detailed explanation of four main security best practices for securing cloud machine identities. These practices address both the credential management and permission aspects, providing a comprehensive framework for defenders.

1. Don't Use Long-Lived Service Keys/Credentials: The primary finding is that static, long-lived credentials pose a significant risk. These credentials, once compromised, can grant persistent, untraceable access to sensitive resources. The talk demonstrated how a developer could easily exfiltrate a service account key, use it to access sensitive data, and leave no direct audit trail linking the action back to their human identity. The recommendation is to avoid these wherever possible and leverage cloud-native, ephemeral alternatives.

1. Regularly Audit and Rotate Credentials: Acknowledging that completely eliminating long-lived credentials might not always be practical, the second key finding emphasizes the necessity of robust lifecycle management. Regular auditing helps discover unused or forgotten credentials, while rotation limits the window of opportunity for a compromised credential to be exploited. This practice is crucial for minimizing the impact of potential breaches.

1. Ensure Least Privilege Permissions: Excess privileges are a common vulnerability for machine identities. The talk illustrated how a VM instance with a default, overly permissive service account (e.g., editor role) could be leveraged by an attacker or an authorized user with SSH access to gain unintended access to sensitive data. The finding stresses that machine identities should only be granted the minimum permissions necessary to perform their intended functions, adhering strictly to the principle of least privilege.

1. Limit and Monitor User Access to Machine Identities: The final key finding addresses the risk of privilege escalation through human users gaining access to machine identities. Attackers can compromise a user account and then leverage that access to assume a powerful machine identity, effectively escalating their privileges. The talk highlighted the difficulty in auditing a user's effective access when it includes the ability to assume various machine roles. Therefore, strict controls and continuous monitoring of user interactions with machine identities are essential to prevent such escalation paths.

These four best practices form a foundational strategy for building a resilient machine identity security program in the cloud, moving beyond reactive measures to proactive risk mitigation.

Technical Deep Dive

▶ Watch: Demonstration of excess privilege leading to sensitive data access (11:00)

The talk provided a detailed technical exploration of each best practice, offering specific implementation guidance for both GCP and AWS.

Best Practice 1: Don't Use Long-Lived Service Keys

The speakers began by demonstrating the inherent danger of long-lived credentials. In a GCP example, a developer with access to Google Secret Manager could view and copy a service account key. Using this key, the developer could authenticate as the service account and access sensitive Cloud Storage data, such as listing and reading a "secret file." Crucially, the logs for these actions would show the service account as the principal, but not the original developer's identity, making forensic analysis challenging. The key, once downloaded, could be used indefinitely, even if the developer left the company.

To mitigate this, the talk proposed alternatives:

• GCP Alternatives:
• Service Identities: For services running directly within GCP, the cloud provider can automatically manage credentials. An example given was attaching a service account to a Cloud Function. When the function runs, Google automatically handles credential management, eliminating the need for developers to manage keys. Terraform configurations were suggested for setting this up.
• Workload Identity Federation: For third-party services (e.g., CI/CD systems like GitHub Actions), this mechanism allows external identities to assume GCP service accounts without long-lived keys. GitHub, for instance, can create a unique service identity for each action run, tied to the branch, job, repository, and organization. This external identity can then be mapped to a GCP service account with precisely the permissions needed for a specific deployment. While more complex to configure with Terraform, it eliminates static credentials.

• AWS Alternatives:
• Service Roles: Similar to GCP's service identities, AWS IAM roles can be assigned to AWS services. For many services, such as Lambda functions, AWS automatically creates and manages a service role specifically for that function, simplifying credential management.
• OIDC Federation: AWS supports OpenID Connect (OIDC) Federation, which is the same standard used for federating human users from identity providers like Okta. In a GitHub Actions context, GitHub can present itself as an OIDC login to AWS. An identity provider is configured in AWS, mapping the GitHub service identity to an AWS service role. This process is also more complex but avoids static IAM user access keys.

Once these alternatives are in place, organizations can enforce policies to prevent the creation of static credentials altogether:

• GCP: Use an Organization Policy named disable service account key creation. This policy can be applied at the organization, folder, or project level. Terraform was provided for this.
• AWS: Implement a Service Control Policy (SCP) to restrict IAM user key creation.

Best Practice 2: Regularly Audit and Rotate Credentials

Recognizing that some use cases might still necessitate long-lived credentials, the second best practice focuses on their diligent management. Regular auditing helps identify and remove unused credentials, while rotation limits the window of exposure for a compromised key.

• GCP:
• Enforce Expiry: An Organization Policy can be used to set a uniform expiry duration for all service account keys created within child projects.
• Auditing: The Cloud Monitoring API tracks service account key authentication events, providing details about the event and the APIs used during authentication.

• AWS:
• Auditing: The IAM service offers a straightforward way to download a credential report. This report contains comprehensive information about all credentials in the account, including their last usage, which is crucial for identifying stale or unused keys.

Best Practice 3: Ensure Least Privilege Permissions

The danger of excess privileges was demonstrated with a GCP example. A developer with SSH access to a Compute Engine VM instance could leverage the instance's attached service account, which by default might have the highly permissive editor role on the Google Cloud project. This allowed the developer to list and read sensitive files from a Cloud Storage bucket, actions far beyond what a VM typically needs.

Recommendations for enforcing least privilege:

• GCP Recommendations:
• Avoid assigning basic roles (editor, viewer, owner).
• Grant permissions only on the specific resources the machine identity needs.
• Utilize IAM conditions to further restrict actions based on context (e.g., time of day, source IP).
• Create custom roles that contain only the precise privileges required.
• Tools:
• Policy Intelligence: Built into Google Cloud IAM, it provides "security insights" on used and unused permissions, aiding in crafting custom roles. However, using it at scale requires a Security Command Center Premium tier subscription.
• Logs: For those without Premium tier, parsing admin activity logs (default) and enabling/parsing data access audit logs (disabled by default) can help identify the actual resources and permissions an identity uses.

• AWS Recommendations:
• Avoid wildcard grants for actions or resources (e.g., Action: "", Resource: "") in IAM policies.
• Explicitly enumerate all required actions and resources.
• Use conditions in policies for fine-grained control.
• Implement deny policies as an additional layer of defense for critical actions.
• Tools:
• IAM Access Analyzer: Can identify unused access, helping to refine policies. It needs to be explicitly created.
• CloudTrail events: AWS can automatically generate policies based on observed CloudTrail events, simplifying the process of creating least privilege policies.
• Repo Kid: An open-source tool that automates the continuous creation and assignment of least privilege roles.

Best Practice 4: Limit and Monitor User Access to Machine Identities

This practice addresses the risk of users (human identities) gaining excessive access to machine identities, which can lead to privilege escalation. Auditing such access is complex because a user's effective permissions include not only their direct grants but also the permissions of any machine identities they can assume.

• GCP Recommendations:
• Restrict Permissions on Service Accounts: Grant resource-level permissions on specific service accounts. Avoid granting service account permissions (like iam.serviceAccounts.actAs or iam.serviceAccounts.getAccessToken) at the project, folder, or organization level, as this applies to all current and future service accounts.
• Tools:
• gcp-iam-privilege-escalation: An open-source tool that scans GCP configurations for permissions that could allow privilege escalation, including those related to service account access.
• Monitor Suspicious Authentication: The Cloud Monitoring API tracks service account authentication events. If a service account was assumed via IAM permissions, these events will include the "original principle," providing a crucial audit trail. Usage logs for the service account can offer even more detailed information, including the contents of the request.

• AWS Recommendations:
• Restrict User Access to Assuming Roles:
• Avoid wildcards (* or :root) in role trust policies, as this can allow any identity in an AWS account to assume the role.
• Avoid granting the iam:PassRole permission, which allows a user to pass a role to a service and bypasses some AWS CloudTrail event logging for assume role events.
• Explicitly specify the principals allowed to assume a role in the trust policy.
• Configure SourceIdentity: For federated access, this can be set in the identity provider. AWS will then track the original identity that assumed a role, even through role chaining, enhancing auditability.
• Use deny policies in trust policies to prevent most users from assuming sensitive roles.
• Tools:
• Pmapper: An open-source tool that visualizes identities with access to roles as a graph, helping to understand complex access paths.
• Monitor Assume Role Events: AWS CloudTrail logs all assume role events. If SourceIdentity is configured, these events will include the original identity that performed the assumption, providing a complete audit trail.

Demo / Proof of Concept

▶ Watch: Implementing least privilege: custom roles, policy intelligence, log analysis (12:30)

While the talk did not feature a single, overarching demonstration of a defensive solution, it effectively utilized two illustrative examples to highlight the vulnerabilities that the best practices aim to address. These served as clear "proofs of concept" for the attack vectors.

1. Long-Lived Credential Compromise: The first demonstration showed a developer accessing a service account key stored in Google Secret Manager. The developer then copied this key to their local machine, used gcloud auth activate-service-account to authenticate as the service account, and subsequently executed gsutil ls and gsutil cat commands to list and read sensitive files from a Cloud Storage bucket. The critical part of this demonstration was showing how the Google Cloud logs recorded the service account as the principal, but provided no direct link back to the developer's human identity, illustrating the auditability gap and the persistence of access.

1. Excess Privilege Exploitation: The second demonstration involved a developer with SSH access to a GCP Compute Engine VM instance. The key vulnerability here was that the service account attached to this VM had been granted the highly permissive editor role on the entire Google Cloud project, often a default configuration. Once SSHed into the VM, the developer was able to leverage the VM's service account permissions to again access sensitive data in a Cloud Storage bucket using gsutil commands. This showcased how a seemingly innocuous access (SSH to a VM) could lead to significant privilege escalation due to overly broad machine identity permissions.

These demonstrations, though simple, powerfully conveyed the risks associated with mismanaged machine identities and provided concrete scenarios that security practitioners could relate to their own environments. They effectively set the stage for the defensive strategies proposed in the subsequent sections.

Defensive Implications

▶ Watch: Summary of the four best practices for machine identity security (19:00)

The insights and best practices presented in this talk offer a robust framework for cloud defenders to significantly enhance their security posture against machine identity-related threats. The implications are multi-faceted, requiring a shift in focus from solely human identity management to a comprehensive strategy that encompasses the unique lifecycle and access patterns of machine identities.

Firstly, defenders must prioritize the adoption of ephemeral, cloud-managed credentials over static, long-lived keys. This means actively migrating away from IAM user access keys in AWS and service account keys in GCP, leveraging OIDC Federation for external workloads (like CI/CD pipelines) and service roles/identities for internal cloud services. By doing so, the risk of credential exfiltration and persistent unauthorized access is drastically reduced.

Secondly, for any unavoidable long-lived credentials, implementing a rigorous credential lifecycle management program is paramount. This includes enforcing automatic expiration, regular rotation, and continuous auditing to identify and revoke unused or compromised credentials. Cloud-native tools like AWS IAM credential reports and GCP Cloud Monitoring API for authentication events are essential for this.

Thirdly, the principle of least privilege must be applied meticulously to all machine identities. Defenders should move away from broad, default roles (e.g., editor, owner) and wildcard permissions (*). Instead, custom roles with precisely enumerated actions and resources, coupled with IAM conditions, should be the standard. Tools like AWS IAM Access Analyzer, GCP Policy Intelligence (with Security Command Center Premium), and open-source solutions like Repo Kid can aid in identifying and enforcing least privilege. Furthermore, implementing deny policies for critical actions provides an additional layer of defense.

Finally, defenders need to strictly limit and monitor user access to machine identities. This involves restricting permissions that allow users to assume or act as service accounts (e.g., iam.serviceAccounts.actAs in GCP, iam:PassRole in AWS). Trust policies for roles should explicitly specify principals, avoiding broad grants. Crucially, configuring SourceIdentity for federated access in AWS and monitoring the "original principle" in GCP Cloud Monitoring API for service account authentications provides invaluable audit trails, allowing defenders to trace actions back to the originating human identity, even through role chaining. Tools like Pmapper can help visualize complex access graphs to identify potential privilege escalation paths.

In essence, defenders should embrace automation (e.g., Terraform for policy enforcement), leverage cloud-native security features (Organization Policies, SCPs, CloudTrail, Cloud Monitoring), and integrate open-source tools to build a proactive, continuous security posture for their cloud machine identities. This comprehensive approach is vital to prevent privilege escalation, data exfiltration, and other forms of compromise that stem from mismanaged machine identities.

Key Takeaways

• Prioritize Ephemeral Credentials: Always favor cloud-managed service identities (GCP Service Identities, AWS Service Roles) and OIDC Federation for both internal and external workloads to eliminate the need for long-lived static credentials.
• Implement Robust Credential Management: When static credentials are unavoidable, enforce strict policies for regular auditing, rotation, and expiration to minimize the window of exposure and identify unused keys.
• Enforce Least Privilege Rigorously: Grant machine identities only the absolute minimum permissions required for their function, avoiding broad roles, wildcard grants, and leveraging custom roles and IAM conditions.
• Control User Access to Machine Identities: Strictly limit and monitor human user permissions that allow assuming or acting as machine identities to prevent privilege escalation and maintain clear audit trails.
• Leverage Cloud-Native and Open-Source Tools: Utilize cloud provider features like Organization Policies, Service Control Policies, IAM Access Analyzer, Cloud Monitoring, and CloudTrail, alongside open-source tools like Repo Kid and Pmapper, for continuous enforcement, auditing, and visualization of machine identity security.

About the Speaker(s)

Komal Dhull is a Founding Backend Engineer at Plerion Security. Her work involves building tooling to help secure access to the cloud, which provided her with direct experience in the complexities and challenges of machine identity management that inspired this talk.

Nathan Brahms is one of the Co-founders and the Vice President of Engineering at Plerion Security. His leadership in engineering at Plerion Security focuses on developing solutions for cloud access security, giving him a deep understanding of the practical security implications of machine identities in cloud environments.

Plerion Security is a company dedicated to building tooling that helps secure access to the cloud, indicating their expertise and focus on cloud security challenges, particularly around identity and access management.