We observed RedisConnectionFailureException intermittently as mentioned in above stacktrace, quantum is around 2-3 per 15 mins, which is very low, but still it should not come ideally. As you can see we have set the connection-timeout = 1000ms & read-timeout=200ms, which is high compare to the response time we have from redis calls.
Stacktrace:
Error reading data from redis for key: key_name. Error: org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketTimeoutException: Read timed out; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:65) at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:42) at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44) at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42) at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:192) at org.springframework.data.redis.connection.jedis.JedisConnection.doWithJedis(JedisConnection.java:827) at org.springframework.data.redis.connection.jedis.JedisConnection.doInvoke(JedisConnection.java:165) at org.springframework.data.redis.connection.jedis.JedisConnection.lambda$new$0(JedisConnection.java:74) at org.springframework.data.redis.connection.jedis.JedisInvoker$Synchronizer.invoke(JedisInvoker.java:1018) at org.springframework.data.redis.connection.jedis.JedisInvoker.just (JedisInvoker.java:112) at org.springframework.data.redis.connection.jedis.JedisStringCommands.get(JedisStringCommands.java:57) at org.springframework.data.redis.connection.DefaultedRedisConnection.get(DefaultedRedisConnection.java:279) at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis (DefaultValueOperations.java:58) at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:61) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224) at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:191) at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:97) at org.springframework.data.redis.core.DefaultValueOperations.get (DefaultValueOperations.java:54) at com.company.service.CacheServiceImpl.getHashFromDsRedis(CacheServiceImpl.java:75) at com.company.service.CacheServiceImpl$$FastClassBySpringCGLIB$$b22ab586.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke (MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.company.instrumentation.aspect.impl.GenericInstrumentationAspectImpl.doInstrumentation(GenericInstrumentationAspectImpl.java:55) at jdk.internal.reflect.GeneratedMethodAccessor137.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at ...Our RedisConfig.java
@Bean("jedisFactory")public JedisConnectionFactory jedisFactory() { RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration(); redisStandaloneConfiguration.setHostName(redisProperties.getHostName()); redisStandaloneConfiguration.setPort(redisProperties.getPort()); redisStandaloneConfiguration.setPassword(redisProperties.getPassword()); JedisPoolConfig jedisPoolConfig = new JedisPoolConfig(); jedisPoolConfig.setMinIdle(redisProperties.getMinIdle()); jedisPoolConfig.setMaxIdle(redisProperties.getMaxIdle()); jedisPoolConfig.setMaxTotal(redisProperties.getMaxTotal()); jedisPoolConfig.setMaxWait(Duration.ofMillis(redisProperties.getMaxWaitTime())); JedisClientConfiguration jedisClientConfiguration = JedisClientConfiguration.builder() .connectTimeout(Duration.ofMillis(redisProperties.getConnectTimeout())) .readTimeout(Duration.ofMillis(redisProperties.getReadTimeout())).usePooling().poolConfig(jedisPoolConfig).build(); return new JedisConnectionFactory(redisStandaloneConfiguration, jedisClientConfiguration);}@Bean("redisTemplate")public RedisTemplate<String, Object> redisTemplate( @Qualifier("jedisFactory") JedisConnectionFactory jedisConnectionFactory) { RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>(); redisTemplate.setConnectionFactory(jedisConnectionFactory); redisTemplate.setKeySerializer(new StringRedisSerializer()); redisTemplate.setValueSerializer(new StringRedisSerializer()); redisTemplate.setHashKeySerializer(new StringRedisSerializer()); redisTemplate.setHashValueSerializer(new StringRedisSerializer()); redisTemplate.afterPropertiesSet(); return redisTemplate;}Config values:
REDIS_CONNECT_TIMEOUT_MS="1000"REDIS_POOL_MAX_IDLE="40"REDIS_POOL_MAX_TOTAL="55"REDIS_POOL_MAX_WAIT_TIME_MS="15"REDIS_POOL_MIN_IDLE="15"REDIS_READ_TIMEOUT_MS="200"Solution tried
We reduced REDIS_POOL_MIN_IDLE a little as we thought that it might happen that older ideal connections are getting dropped by redis server after few seconds, & when request comes spring assigns these thread which has a dropped connection attached to it, which could result in RedisConnectionFailureException. Errors count have reduced somewhat, but not completely.
Suspection
In DEBUG logs we could see that whenever there's a redis call made from application. These messages are getting logged.
DEBUG org.springframework.data.redis.core.RedisConnectionUtils: Fetching Redis Connection from RedisConnectionFactoryDEBUG org.springframework.data.redis.core.RedisConnectionUtils: Closing Redis Connection.Is there a way with which we can monitor/plot on grafana and see what's happeningexactly with the connections in the JedisPool, are they even workingor making connection and closing on every hit.