When a Spring Boot application running on Red Hat OpenShift 4 suddenly stops serving traffic due to exhausted or hung database connections, default HikariCP configurations are often the culprit. Here is how to configure, monitor, and troubleshoot Hikari connection pools to survive network drops and container CPU throttling.
If you are already running Spring Boot on Red Hat OpenShift, you might have tackled Java Virtual Machine (JVM) memory tuning and Red Hat Data Grid integration (see Integrate a Spring Boot application with Red Hat Data Grid and How to troubleshoot Spring Boot in OpenShift). This guide focuses on the layer in between: how Hikari manages database connections under container constraints.
Hikari use in Spring Boot
You can configure Hikari in Spring Boot through application.properties, for example:
# Database Connection Settings
spring.datasource.url=jdbc:postgresql://localhost:5432/db
spring.datasource.username=dbuser
spring.datasource.password=dbpass
spring.datasource.driver-class-name=org.postgresql.Driver
# HikariCP Specific Settings
# Fixed pool size (Recommended for high-load/SDN environments)
spring.datasource.hikari.minimum-idle=20
spring.datasource.hikari.maximum-pool-size=20
# Timeouts and Lifecycle
spring.datasource.hikari.idle-timeout=300000
spring.datasource.hikari.max-lifetime=1800000
spring.datasource.hikari.connection-timeout=30000
spring.datasource.hikari.validation-timeout=5000
# Monitoring & Debugging
spring.datasource.hikari.pool-name=InfinispanAuditHikariPool
spring.datasource.hikari.leak-detection-threshold=5000You can also configure a DataSource bean programmatically in Java:
@Configuration
public class DataSourceConfig {
@Bean
@ConfigurationProperties("spring.datasource.hikari")
public HikariDataSource dataSource() {
HikariConfig config = new HikariConfig();
// Setting values manually or letting @ConfigurationProperties bind them
config.setJdbcUrl("jdbc:postgresql://127.x.x.x:5432/db");
config.setUsername("admin");
config.setPassword("password");
// Optimization: Fixed Pool Size
config.setMaximumPoolSize(40);
config.setMinimumIdle(40);
// Crucial for the "Waiting" thread issue we saw
config.setConnectionTimeout(30000);
config.setValidationTimeout(5000);
// Helps identify which code "forgot" to close a connection
config.setLeakDetectionThreshold(5000);
return new HikariDataSource(config);
}
}Alternatively, you can define these settings in an application.yaml file:
params:
datasource:
driverClassName: com.mysql.jdbc.Driver
jdbcUrl: jdbc:mysql://localhost:8080/myDb
username: login
password: password
maximumPoolSize: 5To add HikariCP and a PostgreSQL driver to a Maven project, include the Java Database Connectivity (JDBC) Spring Boot starter dependencies, which is included via spring-boot-starter-jdbc started dependency:
<!-- Core Spring Boot Web-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!-- spring-boot-starter-jdbc includes HikariCP as default connection pool -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<!-- PostgreSQL Driver -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>Common settings for HikariCP deployment include:
# Basic Database Credentials
spring.datasource.url=jdbc:postgresql://localhost:5432/database_details
spring.datasource.username=your_user
spring.datasource.password=your_password
spring.datasource.driver-class-name=org.postgresql.Driver
# HikariCP Tuning
spring.datasource.hikari.maximum-pool-size=10
spring.datasource.hikari.minimum-idle=5
spring.datasource.hikari.connection-timeout=20000
spring.datasource.hikari.idle-timeout=300000
spring.datasource.hikari.max-lifetime=1200000
spring.datasource.hikari.pool-name=PostgresHikariPoolDeploying Spring Boot with Hikari on Red Hat OpenShift 4
You can deploy a Spring Boot application in Red Hat OpenShift Container Platform 4 using Source-to-Image (S2I) or by deploying a prebuilt container image directly.
Hikari properties
Consider the following useful properties:
| Property | Default | Details |
|---|---|---|
spring.datasource.hikari.minimum-idle | 20 | Minimum number of idle connections kept in the pool at any time |
spring.datasource.hikari.maximum-pool-size | 10 | Maximum number of connections kept in the pool at any time |
spring.datasource.hikari.idle-timeout | 300000 | Maximum time (in milliseconds) a connection can remain idle in the pool before closing |
spring.datasource.hikari.max-lifetime | 1800000 | Maximum lifetime (in milliseconds) of a connection in the pool before replacement |
spring.datasource.hikari.connection-timeout | 30000 | Maximum time (in milliseconds) an application waits for a connection from the pool before timing out |
spring.datasource.hikari.validation-timeout | 5000 | Maximum time (in milliseconds) Hikari waits for a connection validation check to succeed |
See the Troubleshooting section for details on spring.datasource.hikari.leak-detection-threshold and how to use it for leak detection.
Hikari threads
HikariCP uses 2 primary background threads to manage the connection pool:
- The
HikariPool-1:housekeeperthread runs every 30 seconds to perform routine pool maintenance. - The
HikariPool-2:connection-adderthread creates new connections. This thread can become a concern if a new connection waits indefinitely for a database response.
Here's an example of starting Hikari:
2025-11-10T16:24:09.391Z INFO 1 --- [hikari-example] [ main] [ ] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2025-11-10T16:24:09.546Z INFO 1 --- [hikari-example] [ main] [ ] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Added connection ConnectionID:1 ClientConnectionId: 503ec06d-9e4e-45be-a1b5-e3a45748a3d3
2025-11-10T16:24:09.548Z INFO 1 --- [hikari-example] [ main] [ ] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2025-11-10T16:24:09.583Z WARN 1 --- [hikari-example] [ main] [ ] org.hibernate.orm.deprecation : HHH90000025: SQLServer2012Dialect does not need to be specified explicitly using 'hibernate.dialect' (remove the property setting and it will be selected by default)
2025-11-10T16:24:09.584Z WARN 1 --- [hikari-example] [ main] [ ] org.hibernate.orm.deprecation : HHH90000026: SQLServer2012Dialect has been deprecated; use org.hibernate.dialect.SQLServerDialect instead
2025-11-10T16:24:09.611Z INFO 1 --- [hikari-example] [ main] [ ] org.hibernate.orm.connections.pooling : HHH10001005: Database info:
Database JDBC URL [Connecting through datasource 'HikariDataSource (HikariPool-1)']
Database driver: undefined/unknown
Database version: 11.0
Autocommit mode: undefined/unknown
Isolation level: undefined/unknown
Minimum pool size: undefined/unknown
Maximum pool size: undefined/unknown
2025-11-10T16:24:10.337Z WARN 1 --- [hikari-example] [ main] [ ] org.hibernate.mapping.RootClass : HHH000038: Composite-id class does not override equals()Hikari scenarios and challenges
When running Spring Boot on Red Hat OpenShift 4, connection pool issues typically surface in 5 common production scenarios.
1st scenario: Hikari has only 1 adder mechanism
As I explain in SpringBoot application datasource connection is stuck, under specific conditions, network or database disruptions can interrupt active connections, forcing Hikari to create new ones. Because Hikari uses a single thread to add connections, a network interruption can cause that thread to block indefinitely in SocketInputStream.read while waiting for a socket response:
"HikariPool-1:connection-adder" #27481 [281678] daemon prio=5 os_prio=0 cpu=1.23ms elapsed=129648.92s tid=0x00007f4e38009710 nid=281678 runnable [0x00007f4ef2cea000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.Net.poll(java.base@21.0.8/Native Method)
at sun.nio.ch.NioSocketImpl.park(java.base@21.0.8/NioSocketImpl.java:191)
at sun.nio.ch.NioSocketImpl.park(java.base@21.0.8/NioSocketImpl.java:201)
at sun.nio.ch.NioSocketImpl.implRead(java.base@21.0.8/NioSocketImpl.java:309)
at sun.nio.ch.NioSocketImpl.read(java.base@21.0.8/NioSocketImpl.java:346)
at sun.nio.ch.NioSocketImpl$1.read(java.base@21.0.8/NioSocketImpl.java:796)
at java.net.Socket$SocketInputStream.read(java.base@21.0.8/Socket.java:1099) <-----------------------This stalled thread isn't the root cause; it happens after an underlying failure disconnects the application from the database. When HikariCP tries to reconnect, the thread stalls while reading from the socket. Because the pod gets stuck in this adder loop and the JVM cannot self-heal, you must restart the pod, ideally through an automated Kubernetes liveness probe. Detecting the stall early and restarting affected pods immediately prevents the adder thread from blocking continuously. You can do this using actuator, jcmd, or jstack, as shown in the following example:
apiVersion: v1
kind: Pod
metadata:
name: my-app-pod
spec:
containers:
- name: my-app-container
image: my-app-image:1.0
ports:
- containerPort: 8080
livenessProbe:
exec:
command: ["/opt/probes/hikari-liveness.sh"]
initialDelaySeconds: 15
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
...
...
Example probe script:
...
...
#!/usr/bin/env bash
# socket-stall-check.sh - Kubernetes liveness probe.
# Exit 1 (unhealthy -> kubelet restarts pod) if the HikariCP connection-adder
# is stalled on a socket read; exit 0 otherwise.
# Catches the JDK21 NIO stall: Net.poll -> NioSocketImpl.read -> SocketInputStream.read
PID="$(pgrep -f 'java ' | head -n1)"
[ -z "$PID" ] && exit 0 # no JVM yet -> don't fail liveness
# Extract the connection-adder thread block, then check for a socket-read frame.
if jcmd "$PID" Thread.print 2>/dev/null \
| awk '/"[^"]*connection.adder"/{g=1} g{print} g&&/^$/{g=0}' \
| grep -qE 'sun\.nio\.ch\.Net\.poll|NioSocketImpl|socketRead0|SocketInputStream'; then
echo "hikari connection-adder stalled on socket read" >&2
exit 1
fi
exit 02nd scenario: Hikari database access from Red Hat OpenShift 4
When an application in Red Hat OpenShift cannot establish a database connection through Hikari, the breakdown usually stems from routing, firewall rules, or DNS resolution between the cluster and the database network. When connections fail to leave the pod, verify egress connectivity directly from the running container using oc exec with network utilities like nc, curl, or openssl (although nc is seldom installed in containers).
Also, check whether a Red Hat OpenShift NetworkPolicy restricts egress traffic to external database subnets or if DNS resolution fails for the database service endpoint.
3rd scenario: Concurrent access to Hikari
This challenge involves concurrent access across multiple applications, namespaces, and pod replicas. Multiplying pod replicas looks simple in your deployment manifest, but your database sees every new pod as an entirely new pool of connections. Without careful sizing, an autoscaling event intended to handle a traffic surge can accidentally take down the database back end. Hikari's maximumPoolSize setting is central to this balance, and its optimal value depends on benchmarking against database connection limits.
In this scenario, CPU throttling can also increase connection delays when the Linux kernel limits CPU time for running threads. Check the Red Hat OpenShift web console or Prometheus to verify whether throttling is occurring.
4th scenario: Hikari timeouts
Misaligned timeouts between Hikari, the database, and intermediate network hops create intermittent drops that are hard to debug. Configuring 2 key settings helps prevent these drops:
keepaliveTime: Available in HikariCP 4.0.1 or later, this feature prevents the network layer from dropping Hikari's database connection by sending a ping at the TCP level. If an intermediate load balancer idle timeout is shorter than the connection idle time, the network might silently drop the connection.maxLifetime: Set this value lower than other connection timeouts, such as the load balancer or network-level idle timeout. For example, a firewall or F5 timeout can drop an established connection without notifying the client. If the Red Hat OpenShift load balancer terminates an idle connection every 30 minutes, setmaxLifetimeto a shorter duration (such as 25 minutes).
5th scenario: Hikari threads are still Java threads
In a containerized JVM, Hikari connection threads don't run in isolation—they compete with garbage collection and application logic for the same Linux Completely Fair Scheduler (CFS) CPU quota slices defined in your pod limits. For example, in the context of container-aware Java, Java calculates thread counts based on the available CPUs spec.limits for CPU-bounded threads:
Hikari (total=14, active=14, idle=0, waiting=1)The number of CPU-bound threads varies according to the CPU limit. While idle threads don't use CPU cycles, active threads share kernel time.
If you benchmarked your application with a different thread count—or without simulating heavy traffic—the application can stall or slow down when its pod CPU limits (spec.limits) don't allocate enough kernel time for Hikari and your application threads. When the Linux kernel throttles container CPU under high load, it starves Hikari's internal management threads of execution time, reproducing the thread starvation documented in HikariCP issue 1688. This situation can worsen when combined with garbage collection pauses and memory allocation spikes.
Troubleshooting
Beyond these 5 scenarios, connection leaks can also exhaust your pool. A leak occurs when an application doesn't return a connection to the pool after finishing its database operation, preventing other threads from reusing it.
To detect and diagnose connection leaks, use these tools:
- Connection leak detection: Configure the
leakDetectionThresholdsetting using Spring Boot'sspring.datasource.hikari.leak-detection-thresholdproperty, which sets how many milliseconds a connection can remain out of the pool before Hikari logs a warning. - Hikari access logs: Use access logs to track when connections are borrowed, returned, or closed.
- Hikari Prometheus metrics: Review the HikariCP Prometheus metrics guide for queries that help you quickly troubleshoot issues based on the threads described earlier. For example, query
hikaricp_connections_pendingto immediately spot the stalled adder threads.
Conclusion
Tuning HikariCP on Red Hat OpenShift comes down to matching your pool sizes to database capacity, setting connection lifetimes shorter than network and load balancer timeouts, and ensuring container CPU limits give pooling threads enough quota to avoid connection stalls. Review your connection metrics before scaling pod replicas, and let health probes catch stalled pools early.
Next steps
Audit your application's leak-detection-threshold and maxLifetime settings against your cluster's load balancer idle timeouts, and import the Hikari Prometheus Grafana dashboard to monitor pool saturation before your next production rollout.
Have you encountered stalled adder threads or CPU throttling in your own clusters? Try out these metric thresholds in your staging environment, and share your tuning experiences in the Red Hat Developer community. If you run into persistent connection stalls in your Red Hat OpenShift clusters, you can also open a support case through the Red Hat Customer Portal.
Special thanks to Leticia Konno, TJ Cowhey, and Will Russell for collaborating to solve similar problems over the years, some of which I'm documenting in this article.