Keycloak High Availability: Clustering and Scaling

Deploying Keycloak in production requires moving beyond the default single-node configuration where session data, authentication states, and user configurations reside solely in the local JVM heap. Achieving Keycloak HA is not a simple feature toggle; it necessitates a fundamental architectural shift to an embedded Infinispan cluster. This mechanism transforms a collection of Java Virtual Machine (JVM) instances into a single logical storage unit capable of maintaining state consistency and managing session affinity across network boundaries.

The Infinispan Mechanism

The core engine driving this architecture is Infinispan, an in-memory data grid embedded directly within the Keycloak server process. When clustering is enabled, Keycloak initializes an Infinispan instance that manages distinct caches for sessions (user login states), tokens (JWTs), and users (cached attributes). It is crucial to distinguish between a standalone instance and a clustered one: the NONE topology disables the cluster entirely, whereas enabling clustering via the --cluster flag typically defaults to SYNC or REPL_ASYNC depending on the specific configuration, ensuring replication occurs between nodes.

Consider a scenario with two Keycloak nodes, kc-node-1 and kc-node-2, running in a Kubernetes cluster. Configuring the cache mode to SYNC enforces a strict handshake protocol. When a user logs in at kc-node-1, the server creates a session object. Before responding to the client, kc-node-1 must serialize this object and transmit it to kc-node-2. kc-node-2 writes the object to its local cache and sends an acknowledgment back. Only then does kc-node-1 return a 200 OK to the browser. This synchronous handshake guarantees that if kc-node-1 crashes immediately after the user logs in, kc-node-2 holds the complete state, allowing the user to continue without re-authenticating.

# JVM flags to enable sync replication and disable rebalancing during startup
-Dinfinispan.cache.mode=SYNC
-Dinfinispan.cluster.rebalancing.enabled=false

In this configuration, the tradeoff is latency. Every write operation now incurs network round-trip time (RTT) to the peer node. If your nodes are in the same Availability Zone, this is negligible. If they are in different regions, the latency spike can cause timeouts for the user. This brings us to the critical decision point in multi-region deployments: choosing between consistency and availability.

Stateful Scaling Mechanics

Scaling Keycloak horizontally introduces a mechanism known as "rebalancing." When you add a third node to the cluster, Infinispan detects the change in topology. It calculates which cache entries need to move to balance the load evenly across all three nodes. During this rebalancing process, the cluster enters a "rebalancing" state where it performs heavy I/O operations to migrate data. If this happens while the cluster is under heavy write load, you risk performance degradation or even node crashes due to memory pressure.

For production multi-region scaling, the recommended strategy is to disable automatic rebalancing. You set -Dinfinispan.cluster.rebalancing.enabled to false. This forces the cluster to remain stable during scaling events. When you add a new node, it starts empty. You then manually trigger a rebalance or rely on a rolling restart strategy where you add nodes one by one and allow the cluster to stabilize before adding the next. This prevents the "thundering herd" problem where every node tries to migrate data simultaneously.

Multi-Region Topology

In a multi-region setup, you cannot simply spin up nodes in us-east-1 and us-west-2 and expect them to function as a single cluster with SYNC replication. The network latency between regions (often 50ms to 100ms) combined with the synchronization protocol will make the system unresponsive under load. The mechanism here dictates that you must switch the cache mode to ASYNC for cross-region traffic. In ASYNC mode, kc-node-1 commits the local write and acknowledges the client immediately, then fires the replication event to kc-node-2 in the background without waiting for an acknowledgment. This ensures that the user experience remains fast, but it introduces a window of potential data loss if both nodes fail before the asynchronous message is processed.

Another critical mechanism in multi-region environments is the handling of "split-brain" scenarios. If the network link between regions fails, the cluster might split into two independent partitions. Both partitions might accept writes, leading to data divergence. To mitigate this, Keycloak relies on JGroups/Infinispan cluster-level quorum mechanisms to mitigate split-brain scenarios, not a specific cluster_cache entity. However, in a geographically distributed setup, standard quorum mechanisms often fail because the network partition is inevitable. The effective solution here is to use an external Infinispan cluster or a dedicated cache service (like Redis or Memcached) that is configured with explicit failover policies, rather than relying on the embedded Infinispan instances to resolve the split.

When you scale the cluster, you must also consider the max-sessions setting in the jboss-infinispan.xml. If you do not increase this limit, the cluster will start evicting older sessions to make room for new ones, even if you have plenty of memory. This is because Infinispan uses a LRU (Least Recently Used) eviction policy by default. If your application has a high churn rate of sessions, you must tune the eviction_strategy to NONE or set a sufficiently high max_count to prevent premature eviction.

Operational Pitfalls

Finally, the health of the cluster depends on the underlying network discovery mechanism. Keycloak uses JGroups for node discovery. In a containerized environment, you must ensure that the JGROUPS_BIND_ADDR is set correctly so that nodes can find each other. If the DNS resolution fails or the firewall blocks the JGroups port (default 7800), the nodes will run in isolation, effectively creating a single-node cluster regardless of how many pods you run. This results in session data being lost every time a pod restarts, defeating the purpose of the HA setup.

Common failure modes also include cache entry eviction policies and memory pressure leading to OOM kills. Administrators must monitor heap usage closely, as the embedded Infinispan cache consumes memory alongside the application code. Without proper tuning of the max-sessions and eviction_strategy, the cluster may become unstable under load, causing unpredictable behavior.

Practical Takeaways

1. Consistency vs. Latency: Never use SYNC replication across regions. The latency penalty will degrade user experience; stick to ASYNC for geographically distributed nodes.
2. Controlled Scaling: Always disable automatic rebalancing (-Dinfinispan.cluster.rebalancing.enabled=false) in production. Manually manage data distribution to prevent thundering herd events during scaling.
3. Quorum Awareness: Understand that embedded clusters struggle with split-brain in multi-region setups. Rely on external caching layers or robust network partitioning strategies if split-brain mitigation is critical.

FAQ

Q: Can I use Infinispan NONE mode for High Availability? A: No. NONE mode disables clustering entirely, meaning each node operates as a standalone instance. Session data will be lost if a node restarts, which defeats the purpose of High Availability.

Q: How do I handle session loss in ASYNC multi-region mode? A: In ASYNC mode, there is a brief window where data exists on the primary node but not the replica. To minimize impact, ensure your application logic handles transient failures gracefully, or consider a hybrid approach where critical writes are routed to a local region with synchronous replication before propagating asynchronously.

Q: What is the impact of rebalancing on production performance? A: Rebalancing triggers heavy I/O as data chunks are migrated between nodes. In a high-write environment, this can cause significant latency spikes or even node crashes due to memory pressure. Disabling automatic rebalancing is the standard mitigation strategy.

Conclusion

The mechanism of Keycloak HA is a balancing act between data consistency, network latency, and operational complexity. By understanding that Infinispan is not just a cache but a distributed state machine, you can architect a system that survives node failures and scales horizontally without sacrificing performance. The key is to configure the cache topology to match your network reality: synchronous replication for single-region, asynchronous for multi-region, and manual rebalancing to control stability.