Evolving OpenID Connect and Observability in Keycloak - Ryan Emerson & Takashi Norimatsu

Evolving OpenID Connect and Observability in Keycloak

Speakers: Ryan Emerson, Principal Software Engineer, Red Hat; Takashi Norimatsu, Keycloak Maintainer, Hitachi Limited Japan

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=bC4xbBJs0CA

Overview

This talk, presented by Takashi Norimatsu and Ryan Emerson, delves into the significant advancements made in Keycloak, an open-source identity and access management solution. The presentation is divided into two core segments: the evolution of OpenID Connect (OIDC) security features and substantial improvements in Keycloak's observability story. Takashi Norimatsu, a Keycloak maintainer from Hitachi, focuses on how Keycloak is adopting cutting-edge OIDC specifications to bolster security and support emerging digital identity paradigms.

Following this, Ryan Emerson, a Principal Software Engineer on the Keycloak SRE team at Red Hat, illuminates the comprehensive enhancements to Keycloak's operational visibility, including refined metrics, robust tracing capabilities, and improved logging. This talk is crucial for anyone deploying or operating Keycloak, offering insights into preventing sophisticated attacks through new security protocols and empowering SREs and developers with the tools to proactively monitor, troubleshoot, and optimize their Keycloak instances. The combined focus on both security and operational excellence underscores Keycloak's commitment to being a secure and manageable identity platform in cloud-native environments.

Background

▶ Watch: Introduction to Keycloak security and observability updates (0:00)

Keycloak serves as a critical component in many modern application architectures, providing robust authentication and authorization services. As an OpenID Connect (OIDC) provider, it underpins the security of user access and application interactions. However, the evolving threat landscape and the increasing complexity of distributed systems necessitate continuous innovation in both security protocols and operational tooling. Traditionally, OIDC relies heavily on bearer tokens (as defined in RFC 6750), which, while simple to use, carry an inherent risk: anyone possessing a stolen token can use it to access protected resources. This fundamental vulnerability has driven the need for more secure token mechanisms.

Concurrently, the global push towards digital identity, exemplified by initiatives like the EU Digital Identity Wallet (EUDI wallet), has created a demand for standardized ways to issue and manage verifiable credentials (VCs). These digitalized certificates, ranging from national IDs to university degrees, require secure and interoperable protocols for issuance and verification. The OpenID Foundation has been instrumental in developing specifications like OID4VCI to address this need, providing a communication framework for conveying VCs between issuers and digital wallets.

On the operational front, managing and troubleshooting complex identity systems like Keycloak in production environments has always posed significant challenges. Historically, operators have relied on basic logs and custom metrics, often lacking the granular detail and correlation needed for rapid root cause analysis. The rise of cloud-native architectures, characterized by distributed services and dynamic scaling, further amplifies the need for sophisticated observability tools. To address these challenges, the Keycloak community, particularly through its Special Interest Groups (SIGs) like the OIDC SIG and the SRE SIG, actively contributes to integrating new security specifications and developing comprehensive observability features, aiming to improve both the security posture and the operational experience for Keycloak users.

Key Findings

▶ Watch: OID4VCI and Verifiable Credentials for EUDI Wallet (4:40)

The talk highlights several pivotal advancements across Keycloak's security and observability domains, largely driven by community contributions and the Keycloak SRE team.

In terms of security, Keycloak has made strides in supporting advanced OIDC specifications:

• DPoP (Demonstrating Proof-of-Possession, RFC 9449): Officially supported, DPoP addresses the fundamental security flaw of bearer tokens by making access tokens sender-constrained. This ensures that only the legitimate client application can use a token, even if stolen.
• OID4VCI (OpenID for Verifiable Credential Issuance): Keycloak now experimentally supports OID4VCI, covering a part of the specification. This enables Keycloak to act as both a token issuer (authorization server) and a credential issuer (resource server) for verifiable credentials, aligning with initiatives like the EUDI wallet.
• Future OIDC SIG Work: The OIDC SIG is actively working on supporting several upcoming specifications, including Workload Identity (transaction tokens, SPIFFE), FAPI (Financial-grade API) for browser-less authentication in native applications, Shared Signals Framework (SSF), and Open Federation for dynamic trust establishment between identity providers (IDPs) and relying parties (RPs).

On the observability front, Keycloak has undergone a significant transformation to provide operators with richer insights:

• Enhanced Documentation: Comprehensive guides have been introduced for Service Level Indicators (SLIs) and Service Level Objectives (SLOs), alongside an in-depth guide covering all Keycloak-provided metrics (core functionality, JVM, database, HTTP requests, clustering). These guides provide context for interpreting metrics and defining healthy system behavior.
• Out-of-the-Box Grafana Dashboards: Available from Keycloak 26.2, two official Grafana dashboards are provided: one for troubleshooting and another for capacity planning. These dashboards serve as a ready-to-use solution or a foundation for customization.
• Full Tracing Support: Introduced as a preview in Keycloak 26.0 and fully supported since Keycloak 26.1, tracing provides spans for all incoming and outgoing HTTP requests, identity provider brokerage, and outgoing database and LDAP calls. The Tracing Provider SPI allows for custom span creation in Keycloak extensions.
• Elastic Common Schema (ECS) Logging: Keycloak now supports the ECS format across all log handlers, facilitating standardized log ingestion and analysis in centralized logging systems.

These findings collectively demonstrate Keycloak's proactive approach to both enhancing its security posture against evolving threats and providing robust operational tools essential for managing complex identity deployments at scale.

Technical Deep Dive

▶ Watch: FAPI: Browserless authentication flow for native apps (9:00)

The technical advancements in Keycloak span two critical areas: strengthening its security protocols and significantly improving its operational observability.

Security Enhancements: Evolving OpenID Connect

Demonstrating Proof-of-Possession (DPoP) - RFC 9449

The primary security concern with traditional OIDC bearer tokens (RFC 6750) is their susceptibility to theft. If an attacker acquires a bearer token, they can impersonate the legitimate client and access protected resources. DPoP directly addresses this by introducing sender-constrained tokens. Instead of merely possessing a token, the client must cryptographically prove its possession of a corresponding private key when using the access token. This is achieved by binding the access token to a cryptographic key pair owned by the client. When the client makes an API call, it includes a DPoP proof JWT (JSON Web Token) that is signed with its private key and contains a hash of the access token. The resource server then validates this proof against the public key associated with the access token, ensuring that only the legitimate client (who holds the private key) can use the token. Keycloak's support for DPoP significantly mitigates the risk of stolen access tokens, making it a crucial enhancement for high-security applications.

OpenID for Verifiable Credential Issuance (OID4VCI)

With the advent of digital identity wallets like the EUDI wallet in the EU, the need for standardized protocols to issue and manage verifiable credentials (VCs) has become paramount. VCs are digitally signed and verifiable versions of real-world credentials (e.g., driver's licenses, passports, university degrees). OID4VCI, developed by the OpenID Foundation, specifies a communication protocol for a wallet to request and receive VCs from an issuer. Keycloak's experimental support for OID4VCI positions it as a key player in this emerging ecosystem. In this setup, Keycloak plays a dual role:

1. Token Issuer (Authorization Server): The wallet first obtains an access token from Keycloak.
2. Credential Issuer (Resource Server): The wallet then uses this access token to request a specific VC from Keycloak, which then issues and sends the VC back to the wallet.

While currently experimental and covering only a part of the specification, this integration highlights Keycloak's commitment to supporting the future of digital identity.

Future OpenID Connect SIG Activities

The Keycloak OIDC SIG is actively exploring and contributing to several other advanced specifications:

• Workload Identity: This includes support for transaction tokens and SPIFFE (Secure Production Identity Framework for Everyone), aiming to provide strong identities for workloads rather than just human users.
• FAPI (Financial-grade API): Designed for high-security environments, FAPI enables browser-less authentication flows for native applications. This is particularly relevant for first-party native applications (e.g., banking apps) where users input credentials directly into the app, bypassing browser redirects for authentication, thus enhancing user experience and security in specific contexts.
• Open Federation: This specification allows for the dynamic establishment of trust between IDPs and RPs without prior manual configuration. Operating at the application layer, it offers a similar trust-building capability to TLS or mTLS but for identity federation. A promising use case is dynamic client registration, where an IDP (like Keycloak) can automatically verify and accept registration requests from legitimate RPs. Italy's digital identity system, SPID, is cited as an adopter of Open Federation, demonstrating its real-world applicability.

Observability Enhancements: Gaining Deeper Insights

Comprehensive Documentation

Keycloak has significantly improved its documentation for observability, providing crucial resources for operators:

• Service Level Indicators (SLIs) and Objectives (SLOs) Guide: This guide introduces the concepts of SLOs (target goals, e.g., 95% of authentication requests faster than 250ms) and SLIs (measurable metrics to track progress towards SLOs). It provides concrete examples, such as using HTTP response times as an indicator for latency-related SLOs, and suggests foundational metrics (e.g., http_server_requests_seconds_bucket) for tracking.
• In-depth Metrics Guide: This extensive guide details all metrics exposed by Keycloak, categorized into core functionality, JVM, database, HTTP requests, and clustering (local or across availability zones). Each metric is presented with its raw name, a high-level description, and crucial contextual information (e.g., "on a healthy cluster the average replication time will be stable"). This context is invaluable for configuring effective monitoring and alerting systems like Prometheus.

Out-of-the-Box Grafana Dashboards

To streamline monitoring, Keycloak 26.2 introduces two official Grafana dashboards:

• Troubleshooting Dashboard: Designed to help quickly identify the root cause of issues, providing visualizations of key operational metrics.
• Capacity Planning Dashboard: Aids in right-sizing Keycloak deployments based on current load, helping to prevent performance bottlenecks.

These dashboards are available for direct use or as a customizable starting point, encouraging community feedback for further refinement.

Full Tracing Support

Tracing, introduced as a preview in Keycloak 26.0 and fully supported since 26.1, is a game-changer for understanding the flow and latency of requests through Keycloak. It provides spans for:

• All incoming and outgoing HTTP requests.
• Identity Provider (IdP) brokerage operations.
• Outgoing database calls (e.g., PostgreSQL).
• Outgoing LDAP calls.

A significant feature is the Tracing Provider SPI, which allows developers of Keycloak extensions to create their own custom spans, integrating their code into the overall tracing landscape. This provides end-to-end visibility across Keycloak's internal operations and its interactions with external systems.

Elastic Common Schema (ECS) Logging

To facilitate centralized log management and analysis, Keycloak now supports the Elastic Common Schema (ECS) format across all its log handlers. ECS provides a standardized way to structure log data, making it easier to parse, filter, and correlate logs from various sources within a logging solution like Elastic Stack (Elasticsearch, Kibana) or Loki. This standardization is crucial for large-scale deployments where logs from multiple Keycloak instances and other services need to be aggregated and analyzed efficiently.

These technical deep dives illustrate Keycloak's comprehensive strategy to not only secure the identity layer but also to provide the necessary tools for operators to maintain robust, performant, and observable deployments.

Demo / Proof of Concept

▶ Watch: Transition to Keycloak Observability updates (12:20)

Ryan Emerson demonstrated Keycloak's new observability features using a MiniKube cluster setup. The architecture comprised a single Keycloak pod interacting with a PostgreSQL database. Observability components included Prometheus for scraping metrics, Jaeger for collecting tracing spans, and Promtail pushing logs to Loki. All these data sources were visualized through Grafana, acting as a single pane of glass.

The core of the demo revolved around a defined Service Level Objective (SLO): "95% of all authentication related requests to be faster than 250 milliseconds within a given 5-minute range." To simulate a production environment, 100 users were created in a realm zero within Keycloak, and a Gatling script was configured to log a user in and out every second, providing a light load on the system.

Initially, the Grafana dashboard displayed a healthy system, with the SLO panel showing 99.44% of requests meeting the 250ms target and zero error responses. Ryan then navigated through the troubleshooting dashboard, showcasing:

• JVM metrics: Visualizations for average memory usage, CPU usage, and garbage collection time.
• Database metrics: Details on connection pool utilization and available connections.
• HTTP metrics:
• A panel showing total requests per Keycloak URI, highlighting auth and authenticate endpoints as the most frequently hit.
• A heatmap visualizing request latencies grouped into buckets. Initially, most requests were falling into the sub-100ms range.

The key highlight of the demo was the integration of exemplars with tracing. On the latency heatmap, specific "pink dots" (exemplars) represented individual requests. By clicking on an exemplar, Ryan could query Jaeger to retrieve the full trace spans associated with that specific request. A healthy trace showed a total request time of 170 milliseconds, with specific spans indicating time spent on Argon hashing (78ms) and database calls (e.g., a select statement taking 74ms). Furthermore, clicking on a database span within Jaeger allowed direct access to associated debug logs, demonstrating the powerful correlation between metrics, traces, and logs.

To illustrate the troubleshooting capabilities, Ryan introduced a problem using Chaos Mesh, injecting an arbitrary 100-millisecond delay into the PostgreSQL service. Immediately, the Grafana SLO panel updated, showing a significant drop to 56% of requests within the 250ms range, indicating a clear SLO violation. Revisiting the HTTP latency heatmap, a new, thicker band of requests appeared in the higher latency buckets. Selecting an exemplar from this problematic range revealed a total request time of 739 milliseconds. Analyzing the spans, it was evident that while Argon hashing time remained consistent, the database calls (both select statements and commits) had now increased to hundreds of milliseconds. This rapid diagnosis allowed Ryan to pinpoint the database as the root cause of the performance degradation, effectively demonstrating how Keycloak's enhanced observability features enable quick and efficient root cause analysis, saving valuable time for SREs.

Defensive Implications

▶ Watch: Improved Keycloak observability documentation and SRE work (13:10)

The advancements in Keycloak's security and observability features carry significant defensive implications for organizations leveraging the platform.

For Security Architects and Engineers:

• Adopt DPoP for Enhanced Token Security: The official support for DPoP (RFC 9449) provides a critical mechanism to mitigate the risk of stolen access tokens. Security teams should prioritize configuring Keycloak and client applications to utilize sender-constrained DPoP tokens, especially for high-value APIs and sensitive data access. This moves beyond basic bearer token security, significantly raising the bar for attackers.
• Prepare for Verifiable Credentials and Digital Wallets: Keycloak's experimental support for OID4VCI signals its readiness for the future of digital identity. Organizations should start evaluating how verifiable credentials and digital wallets (like the EUDI wallet) could integrate into their identity ecosystems, particularly for use cases requiring strong, verifiable assertions about user attributes (e.g., identity verification, age checks, professional certifications).
• Leverage FAPI for Native Applications: For first-party native applications, especially in regulated industries like finance, the upcoming FAPI support will be crucial. Security teams should plan for implementing browser-less authentication flows to enhance the user experience and potentially reduce certain attack vectors associated with browser redirects.
• Explore Open Federation for Dynamic Trust: For complex, multi-organization identity ecosystems, Open Federation offers a way to establish trust dynamically and securely. This can streamline onboarding of new relying parties and reduce the manual overhead and potential misconfigurations associated with static trust relationships.

For SREs and Operations Teams:

• Proactive Monitoring with SLOs/SLIs: The new documentation on SLOs and SLIs empowers SRE teams to define clear performance and availability targets for their Keycloak deployments. By translating these objectives into measurable indicators using Keycloak's comprehensive metrics, teams can set up proactive alerts (e.g., in Prometheus) that detect performance degradations or outages before they impact users.
• Utilize Grafana Dashboards for Immediate Insights: The out-of-the-box Grafana dashboards (troubleshooting and capacity planning) are invaluable tools. SREs should integrate these dashboards into their monitoring stacks to gain immediate visibility into Keycloak's health, resource utilization, and potential bottlenecks. These dashboards serve as a strong starting point, which can then be customized to fit specific operational needs.
• Accelerate Root Cause Analysis with Tracing: The full support for distributed tracing (via Jaeger or similar tools) is a game-changer for troubleshooting. When performance issues arise, SREs can use traces to quickly identify the exact component or operation causing latency (e.g., a slow database query, an unresponsive LDAP server, or an external identity provider). The ability to drill down from metrics (exemplars) to specific traces and correlated logs significantly reduces Mean Time To Resolution (MTTR).
• Standardize Logging with ECS: Adopting Elastic Common Schema (ECS) logging across all Keycloak instances and other services simplifies log aggregation and analysis. This standardization allows for more efficient querying, filtering, and correlation of logs in centralized logging platforms like Loki or Elastic Stack, making it easier to identify patterns, security incidents, or operational issues across the entire system.

By strategically adopting these security enhancements and fully leveraging the new observability toolkit, organizations can build more resilient, secure, and manageable Keycloak-based identity solutions, effectively defending against both external threats and internal operational challenges.

Key Takeaways

• DPoP Enhances Access Token Security: Keycloak's support for Demonstrating Proof-of-Possession (DPoP) (RFC 9449) significantly improves security by making access tokens sender-constrained, preventing their misuse even if stolen.
• Keycloak Embraces Digital Identity Evolution: Experimental support for OpenID for Verifiable Credential Issuance (OID4VCI) positions Keycloak at the forefront of digital identity, enabling it to issue and manage verifiable credentials for initiatives like the EUDI wallet.
• Future-Proofing with Advanced OIDC Specs: The Keycloak OIDC SIG is actively working on integrating advanced specifications like FAPI for secure native application authentication and Open Federation for dynamic, trustless identity provider-relying party relationships.
• Comprehensive Observability for Operational Excellence: Keycloak 26.1+ introduces full support for tracing, enhanced metrics documentation, Elastic Common Schema (ECS) logging, and out-of-the-box Grafana dashboards (Keycloak 26.2).
• Empowering SREs for Proactive Management: The new observability features, particularly the integration of exemplars linking metrics to traces and logs, enable SRE teams to define SLOs/SLIs, proactively monitor system health, and rapidly diagnose performance bottlenecks and root causes.

About the Speaker(s)

Takashi Norimatsu is a dedicated Keycloak maintainer and works for Hitachi Limited Japan. His contributions to the Keycloak community primarily focus on advancing the platform's security features, particularly within the realm of OpenID Connect specifications. He is deeply involved with the Keycloak OIDC Special Interest Group (SIG), driving the adoption of new security standards to make Keycloak more robust and secure.

Ryan Emerson is a Principal Software Engineer at Red Hat and a key member of the Keycloak SRE team. His expertise lies in improving the operational aspects of Keycloak, focusing on enhancing its observability, reliability, and ease of management for production environments. He plays a significant role in developing and documenting features related to metrics, tracing, and logging, empowering users and SREs to better understand and manage their Keycloak deployments.