Keycloak High Availability, Clustering, and Cache Replication

High Availability, Clustering, and Cache Replication

Deploying Keycloak behind a load balancer to handle increased traffic immediately exposes a fundamental state management problem: Keycloak is inherently stateful. When a user authenticates, a session is established on a specific node. If subsequent requests are routed to a different node, that node lacks the context of the initial authentication, potentially forcing re-authentication or returning 401/403 errors.

Achieving true horizontal scaling in Keycloak requires more than just adding servers; it demands a consistent view of authentication state across all nodes. This consistency is maintained through two primary mechanisms: JGroups-based clustering for inter-node communication and Infinispan cache replication for data synchronization. This article explores the mechanics of this architecture, configuration requirements, and operational tradeoffs for platform engineers and SREs.

The State Problem in Distributed Systems

In a monolithic deployment, all requests target a single server, where the session store resides in local memory or a local database. In a clustered environment, requests are distributed across multiple nodes. If Node A processes a login, Node B cannot verify the resulting JWT or SSO cookie without accessing the same session data.

There are two primary approaches to solving this state distribution problem:

1. Sticky Sessions: The load balancer consistently routes requests from a specific user to the same node. While simple, this is fragile. If the node hosting the session fails, the session is lost unless it is persisted to an external store.
2. Shared State (Clustering): All nodes share session data in real-time. This enables true stateless scaling, allowing any node in the cluster to handle any request regardless of which node established the session.

Keycloak employs the second approach. It leverages Infinispan, a distributed cache platform, to replicate session data across the cluster, ensuring that session state is available to all nodes.

Infinispan Cache Replication

Infinispan is embedded directly within Keycloak and manages several caches. For high availability, the critical caches are:

• default: Caches general-purpose data such as user federation caches and persistent user sessions if configured.
• sessions: Stores user, authentication, and login sessions. This is the primary store for active session state.
• realms: Stores realm configurations, clients, and roles.

When a node updates a user’s session, Infinispan replicates that update to all other nodes in the cluster. This replication occurs over a network channel managed by JGroups, which handles node discovery and message transport.

How Replication Works

1. Write: A user logs in. The originating node creates a session entry in the sessions cache.
2. Replicate: Infinispan sends a write command to all other nodes in the cluster via JGroups.
3. Commit: Other nodes apply the write to their local cache.
4. Read: When the user makes a subsequent request to a different node, that node checks its local cache. Because the session was replicated, the node has the data and can validate the token without contacting the originating node.

This process ensures that if any single node fails, the remaining nodes retain the full session state. However, this replication introduces network latency. As the cluster size grows, the volume of network traffic generated by every write operation increases proportionally.

Configuring the Cluster

Configuring Keycloak for clustering requires precise network bindings and cache mode settings. Simply running multiple instances is insufficient.

1. Network Bindings

Each node must be able to discover and communicate with others via JGroups. Keycloak uses the cache-stack option together with JGroups bind-address settings to define its identity within the cluster, while hostname and http-relative-path configure the externally facing URL used for issuing tokens and redirects.

# Node 1
kc.sh start \
  --hostname=node1.example.com \
  --http-relative-path=/auth \
  --cache-stack=tcp
 
# Node 2
kc.sh start \
  --hostname=node2.example.com \
  --http-relative-path=/auth \
  --cache-stack=tcp

The JGroups transport, configured via --cache-stack and the underlying bind-address settings, instructs Infinispan on what IP address or DNS name to advertise to other nodes. In containerized environments like Docker or Kubernetes, this must resolve to the pod IP or a service DNS name that targets the pod.

2. Cache Stack Selection

Keycloak supports various JGroups stacks for cluster communication. The default is often udp (UDP multicast), but in modern cloud environments, tcp (TCP unicast) is frequently preferred. UDP multicast is often blocked by firewalls and cloud provider networking restrictions.

• udp: Uses multicast addresses. Simple to set up but unreliable in NATted or cloud environments.
• tcp: Uses unicast TCP connections. More reliable, though it requires explicit configuration of initial hosts.
• kubernetes: Uses the Kubernetes API to discover pods. Ideal for K8s deployments.

For bare metal or VMs, tcp with a static list of nodes is common. For Kubernetes, the kubernetes stack is typically the best choice.

3. Synchronous vs. Asynchronous Replication

By default, Infinispan uses synchronous replication for the default cache. A write operation waits for acknowledgment from all nodes before returning success. This ensures strong consistency but increases latency.

In high-throughput scenarios, you might consider switching to asynchronous replication for non-critical caches. However, for user sessions, synchronous replication is recommended to prevent session loss. Asynchronous replication does not prevent session loss; it reduces latency at the cost of potential data loss. Therefore, async should only be used for non-session data where eventual consistency is acceptable.

Operational Tradeoffs

Clustering introduces complexity that must be managed carefully. Consider the following tradeoffs.

Latency and Bandwidth

Every authentication event triggers replication traffic. In a 5-node cluster, a single login generates traffic to 4 other nodes, significantly increasing network load compared to a single-node deployment. Monitor network bandwidth and latency between nodes closely. If nodes span different availability zones, consider using asynchronous replication for non-critical caches to reduce cross-AZ latency.

Split-Brain Scenarios

Network partitions can lead to "split-brain" scenarios where two groups of nodes cannot communicate, potentially leading to data divergence if both accept writes. Infinispan mitigates this using partition handling strategies. By default, Infinispan denies both reads and writes across all partitions until they merge; this behavior can be configured so that, for example, only the partition holding a majority of the cluster continues to accept writes, while other partitions become read-only or reject writes.

To handle these scenarios, configure the partition-handling and merge-policy settings in your Infinispan cache configuration. While Keycloak’s default configuration handles most of this, you should test network partition scenarios in a staging environment.

Load Balancer Health Checks

Your load balancer must perform health checks that go beyond simple HTTP 200 responses. Use Keycloak’s /health/live and /health/ready endpoints:

• /health/live: Checks if the process is alive.
• /health/ready: Checks if the node is ready to serve traffic, including verifying that the Infinispan cache is initialized and connected to the cluster.

If a node is still initializing its cache, it should not receive user traffic. The /health/ready endpoint ensures that only fully synced nodes are added to the load balancer’s pool.

Common Pitfalls

Before finalizing your cluster configuration, be aware of these common mistakes:

• Incorrect Hostname Resolution: If the --hostname flag points to a loopback address or an internal IP that other nodes cannot reach, the cluster will form but fail to replicate data. Always verify DNS resolution from within each container or VM.
• Firewall Blocking JGroups Ports: JGroups uses a range of ports for communication. Ensure that not only the HTTP port but also the JGroups discovery and communication ports are open between all nodes.
• Ignoring Cache Initialization Time: Adding nodes to the load balancer before they have joined the Infinispan cluster can cause initial request failures. Rely on the /health/ready endpoint to gate traffic.

Practical Takeaways

When designing for Keycloak high availability, keep these mental models in mind:

1. Statelessness is a Goal, Not a Default: Keycloak clusters are stateful by design. Your infrastructure must support state replication, not just load distribution.
2. Network is the Bottleneck: As cluster size grows, network bandwidth and latency become the primary constraints on performance. Design your network topology accordingly.
3. Consistency vs. Availability: Synchronous replication guarantees consistency but limits availability during network partitions. Asynchronous replication improves availability but risks data loss. Choose based on your business requirements.
4. Health Checks are Critical: Proper health checking is the only way to ensure that traffic is only routed to healthy, synced nodes. Never skip this step.

Conclusion

High Availability in Keycloak is achieved through Infinispan cache replication, which synchronizes session state across nodes via JGroups. This architecture allows for true horizontal scaling without relying on sticky sessions, though it comes at the cost of increased network traffic and latency.

To implement this successfully:

1. Use a TCP-based JGroups stack (tcp or kubernetes) for reliability.
2. Configure --hostname correctly to ensure nodes can discover each other.
3. Use /health/ready endpoints in your load balancer to prevent routing traffic to uninitialized nodes.
4. Monitor network latency between nodes, as replication traffic scales with cluster size.

By understanding the mechanism of cache replication, you can design a Keycloak cluster that is both highly available and performant.