Keycloak Performance Tuning: JDBC Ping vs DNS_PING Cache Strategies

When tuning Keycloak for high throughput, the most common bottleneck is not the authentication logic itself, but the underlying distributed cache synchronization provided by Infinispan. Keycloak relies on JGroups to manage cluster membership, and the "ping" protocol determines how nodes find each other. The two primary strategies, JDBC_PING and DNS_PING, operate on fundamentally different mechanisms: one treats the database as the source of truth for topology, while the other treats the network infrastructure as the source of truth. Understanding the mechanism of data flow between these strategies is critical for avoiding cache starvation or split-brain scenarios during scale-up.

This article is Part 13 of the Keycloak Masterclass Series.

The Mechanism of DNS_PING

The DNS_PING strategy operates on the principle of decoupling cluster membership from application state. When a Keycloak node starts, it does not query a database table to find its peers. Instead, it performs a DNS lookup against a configured record (e.g., keycloak-cluster.example.com). The mechanism here is purely network-centric. The node receives a list of hostnames or IP addresses from the DNS resolver and attempts to establish TCP connections to those endpoints.

This approach assumes that the DNS record contains the complete and accurate set of available nodes. In a dynamic environment like Kubernetes, this is often achieved via a Headless Service or a dedicated ClusterIP service. The node resolves the service name, receives a list of Pod IPs, and joins the cluster.

The critical performance implication lies in the handling of node churn. If a node crashes, DNS_PING relies on the DNS TTL (Time To Live) or the underlying orchestration layer to update the record. If the DNS record is cached by the operating system or a local resolver with a long TTL, a new node joining the cluster may not see the crashed node immediately, or conversely, may attempt to connect to an IP that is no longer active. This leads to "stale" cluster views where nodes are attempting to replicate data to dead endpoints, causing write retries and increased latency.

Furthermore, DNS_PING does not inherently verify the health of the node it discovers; it only verifies network reachability. If a node is running but its Infinispan cache is hung, DNS_PING will still include it in the cluster view, propagating the hang to the entire cluster. This is a known limitation when using round-robin DNS or load balancers that do not support health-aware routing at the DNS level.

The Mechanism of JDBC_PING

In contrast, JDBC_PING embeds the cluster discovery logic directly into the database layer. When enabled, Keycloak nodes poll a specific database table configured via the JDBC_PING protocol (e.g., JGROUPS_PING table) rather than the INFRA_CACHE cache to retrieve the list of active cluster members. The mechanism is synchronous with the database transaction lifecycle regarding the refresh of the membership view. However, to optimize performance, the node maintains a local cached view of members between polls. This ensures that while the refresh of the membership view requires a database read, the node does not perform an immediate DB read for every internal state change or message, thereby reducing the frequency of database contention.

The performance trade-off here is immediate. Every time a node joins, leaves, or needs to refresh its view of the cluster, it must perform a database read. This creates a dependency chain: Cluster Membership -> Database Read -> Cluster View. Under normal load, this is negligible. However, during a "thundering herd" event—where a large number of nodes restart simultaneously—the database can become a bottleneck. All nodes will simultaneously attempt to read the membership table, potentially causing lock contention on the specific row or table used for discovery.

The advantage of JDBC_PING is consistency. Because the database is the single source of truth for the application state, the cluster view is guaranteed to reflect the actual state of the persisted data. If a node crashes and fails to commit its "heartbeat" or presence update to the database, it will eventually be purged from the table (based on the configured timeout), ensuring that the cluster view remains accurate without relying on external DNS propagation delays.

However, this strategy introduces a risk of circular dependency. If the database is slow or under heavy load, the cluster discovery mechanism slows down, which delays cache synchronization, which in turn increases the load on the database as nodes retry operations. This feedback loop can lead to a cascading failure if the database connection pool is exhausted by the discovery process itself.

Worked Scenario: Benchmarking the Trade-offs

To visualize the impact of these mechanisms, consider a benchmark scenario involving a 4-node Keycloak cluster handling 5,000 concurrent users performing login and token refresh operations. We compare two configurations:

Configuration A (DNS_PING): Running on Kubernetes with a Headless Service. Configuration B (JDBC_PING): Running on bare-metal VMs with a shared PostgreSQL instance.

Phase 1: Warm-up (Node Join) In Configuration A, when a new node starts, it resolves the DNS record. If the DNS TTL is set to 60 seconds, the new node might see an outdated list of IPs if a previous node was just terminated. The new node attempts to connect to the dead node, times out, and retries. This adds a 2–5 second delay per node to reach a stable cluster view. The cache is initially empty, and the first few requests trigger a full cache replication from the remaining healthy nodes. Latency spikes to 400ms for the first 100 requests.

In Configuration B, the new node queries the JGROUPS_PING table. The database responds instantly with the current list of 3 active nodes. The join happens in under 100ms. However, the initial query adds a small load to the PostgreSQL CPU. The cache view stabilizes faster, but the database transaction log grows slightly larger due to the heartbeat updates.

{
  "scenario": "node_join",
  "config_a_latency_ms": 400,
  "config_b_latency_ms": 100,
  "config_a_delay_source": "dns_ttl_cache",
  "config_b_delay_source": "db_transaction_overhead"
}

Phase 2: Write Storm (Token Refresh) Both clusters enter a state where users are rapidly refreshing tokens, generating high write traffic to the SESSION_CACHE.

In Configuration A, the cache invalidation messages are broadcast via the JGroups channel. Since the cluster view is stable (assuming DNS updates are handled by the orchestrator), the replication is efficient. However, if a node fails, the DNS record might not update immediately. The remaining 3 nodes continue to replicate to the dead node, wasting bandwidth and CPU cycles on failed RPCs. This manifests as a 15% increase in CPU usage on the remaining nodes compared to the baseline.

In Configuration B, the cache invalidation messages are also broadcast, but the cluster view is refreshed more frequently from the database. During the write storm, the database handles the heartbeat updates alongside the application traffic. If the database connection pool is not sized correctly, the JDBC_PING thread waits for a connection, delaying the node's ability to acknowledge cache updates. This causes the cache to temporarily hold stale data, leading to an increased risk of authentication failures as users hit the stale cache before the update propagates.

Phase 3: Node Failure Recovery When a node in Configuration A crashes, the remaining nodes detect the failure via JGroups' GMS (Group Membership Service) timeout. They then update their internal view. The DNS record remains unchanged until the orchestrator updates it, which might take 30–60 seconds. During this window, the cluster operates with a "ghost" node.

When a node in Configuration B crashes, it stops writing heartbeats to the database. After the configured ping.interval (default 2000ms) and ping.time_to_live (default 10000ms), the other nodes query the database, see the missing entry, and remove it from the cluster view. This recovery is deterministic but relies entirely on the database being responsive. If the database is slow, the recovery time is extended.

Strategic Recommendation

For most cloud-native deployments, DNS_PING is the superior choice for performance, provided the orchestration layer (Kubernetes, Docker Swarm) handles service discovery health checks correctly. The mechanism avoids the database dependency for cluster membership, reducing the risk of database-induced cluster instability. The latency penalty of DNS lookups is generally lower than the lock contention risk of JDBC_PING under high-frequency churn.

However, if you are running on bare metal or in a hybrid cloud environment where DNS records are not dynamically updated, JDBC_PING offers a more predictable consistency model. The trade-off is the increased load on the database, which must be accounted for in capacity planning.

In my opinion, the "best" strategy is not a binary choice but a function of your infrastructure's reliability. If your DNS layer is robust and your orchestration handles health checks, DNS_PING will yield better benchmark results for write-heavy workloads. If your infrastructure is fragile or your database is already heavily optimized for caching, JDBC_PING provides a safety net that prevents split-brain scenarios, albeit at the cost of higher database I/O.

Ultimately, the mechanism of data flow dictates the system's resilience. DNS_PING flows through the network; JDBC_PING flows through the database. Choose the path that aligns with your weakest link.

Conclusion

Selecting between JDBC_PING and DNS_PING in Keycloak is a decision that balances network topology visibility against database-driven state synchronization. While DNS_PING offers superior performance in dynamic, containerized environments by decoupling discovery from the database, JDBC_PING provides a deterministic consistency model for static or hybrid infrastructures. Understanding the specific failure modes and latency characteristics of each mechanism allows architects to optimize Infinispan cache strategies for their specific workload requirements, ensuring stable cluster behavior under both steady-state and high-churn conditions.

For further verification of these mechanisms, refer to the official Infinispan JGroups Ping Protocols documentation and the Keycloak Cluster Configuration Guide.

Common Pitfalls

When implementing these strategies, engineers frequently encounter specific configuration errors that degrade performance or cause instability:

1. JDBC_PING Table Mismatch: The discovery table name must match the configuration in jgroups.xml exactly. A mismatch between the configured table name and the actual database schema prevents nodes from discovering each other entirely, leading to a single-node cluster that cannot replicate data.
2. Connection Pool Exhaustion: In JDBC_PING setups, failing to size the database connection pool correctly for the number of Keycloak nodes can lead to exhaustion. If all nodes attempt to poll the table simultaneously during a restart, they may block waiting for connections, effectively halting cluster formation.
3. DNS_PING TTL Delays: Setting an excessively high TTL on the DNS record used by DNS_PING creates a lag between a node's actual state and the cluster's view. If a node dies, the cluster continues to send traffic to it until the TTL expires, causing significant latency spikes and failed transactions during the outage window.
4. Stale Records in Dynamic Environments: In Kubernetes, relying on static DNS records instead of Headless Services can result in DNS_PING resolving to IPs of terminated pods, causing persistent connection timeouts and resource waste.

Practical Takeaways

To select the right strategy for your environment, apply these mental models:

• The Database Dependency Rule: If your database is already the bottleneck for your application traffic, avoid JDBC_PING for cluster discovery to prevent compounding latency issues.
• The Orchestration Trust Model: If you are running on a platform like Kubernetes that guarantees service discovery health checks, trust the network layer (DNS_PING) over the database layer for faster convergence.
• The Consistency vs. Availability Trade-off: Choose JDBC_PING if you require the cluster view to strictly reflect the database state at all times, accepting higher latency. Choose DNS_PING if you prioritize low-latency writes and can tolerate brief periods of stale node information.

FAQ

Q: Can I mix JDBC_PING and DNS_PING in the same cluster? A: No. All nodes in a JGroups cluster must use the same discovery protocol to ensure they can find each other. Mixing protocols will result in a fragmented cluster where nodes cannot communicate.

Q: Does DNS_PING require a specific DNS provider? A: No, DNS_PING works with any standard DNS provider. However, for dynamic environments, it is critical that the DNS provider supports rapid updates or that the orchestration layer (e.g., Kubernetes) manages the DNS records automatically via Headless Services.

Q: How does JDBC_PING handle database failover? A: JDBC_PING relies on the database being highly available. If the database goes down, the cluster discovery mechanism fails, and the Keycloak cluster will stop functioning. It is recommended to use a highly available database cluster (e.g., PostgreSQL with streaming replication) when using JDBC_PING.