棘手的bug, 无从下手
使用redisson 的过程中, 我们的应用程序突然报错:
org.redisson.client.RedisConnectionClosedException: Command (EXISTS), params: [paypal_fee_in_account] succesfully sent, but channel [id: 0x812039d1, L:0.0.0.0/0.0.0.0:34182 ! R:172.16.34.14/172.16.34.14:6379] has been closed!
具体如下:
17 13:25:58 ERROR[-nio-9040-exec-2] ntThrowableUtils.throwableProceed:28 [202210171325580000009934]: Unexpected error occur during executing method "payByBalance", "SYS00001"
17 13:25:58 ERROR[-nio-9040-exec-2] ntThrowableUtils.throwableProceed:29 [202210171325580000009934]:
org.redisson.client.RedisConnectionClosedException: Command (EXISTS), params: [paypal_fee_in_account] succesfully sent, but channel [id: 0xdeee45c8, L:0.0.0.0/0.0.0.0:34208 ! R:172.16.34.14/172.16.34.14:6379] has been closed!
at org.redisson.client.handler.CommandsQueue.channelInactive(CommandsQueue.java:88)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:242)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:228)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:221)
at org.redisson.client.handler.ConnectionWatchdog.channelInactive(ConnectionWatchdog.java:84)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:242)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:228)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:221)
at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:242)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:228)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:221)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1403)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:242)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:228)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:912)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:827)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
从字面意思来看, 是redisson的链接被关闭, 那为什么redis 似乎没有错误? 为什么 会被关闭? 是因为空闲时间太长了吗?
各种查找资料, 发现可能是配置问题.
我增加 ping 参数试试:
pingConnectionInterval: 3000
会不会是因为空间机制出了问题, 导致链接失活了, 然后又去使用导致了这个错误?
但是过了几天, 发现问题还是存在..
各种尝试, 发现问题依旧.. 棘手的bug啊, 让人无从下手...
终于找到问题原因
经排查, 问题可能是因为redisson的版本问题, 在redisson 的issue中找到了问题 https://github.com/redisson/redisson/issues/1748 , 果然还是 github 好用! redisson 的问题还是蛮多的, 另一个方面也反应了redisson的更新速度之快..
仔细阅读issue, 发现其实就是官方的bug,
而且, 确实已经修复了, 修复版本号是: https://github.com/redisson/redisson/commit/1a08db75b72f8863f9e5dc2fa4e3de5780d521d7
从commit 中可以看出具体修改了什么.
而我司项目的 redisson 版本是 redisson-spring-boot-starter-3.9.1 , 确实比较低了! 问题修复的版本是: redisson-parent-3.11.6 或者 redisson-2.15.0
左思右想, 这个原因比较符合我们的现状, 因为, 我们的redis 其实是正常的, 我们的其他项目也用到redisson ,但是没有这个问题, 而他的 redisson的版本是
问题解决: 升级redisson
怎么办, 当然是升级咯. 不过, 升级也是很多问题的:
如何升级? 我开始是增加 compile('io.lettuce:lettuce-core:6.2.1.RELEASE') 试试, 发现还是原来的错误..
去掉导致问题的依赖试试: com.lktest:lk-framework-redis:2.4.01-RELEASE :
结果 发现他被很多其他第三方类库依赖, :
***************************
APPLICATION FAILED TO START
***************************
Description:
Parameter 0 of method idGenerator in com.lktest.lk.framework.idgenerate.config.IdGenConfiguration required a bean of type 'org.springframework.data.redis.core.RedisTemplate' that could not be found.
The following candidates were found but could not be injected:
- Bean method 'redisTemplate' in 'RedisAutoConfiguration' not loaded because @ConditionalOnMissingBean (names: redisTemplate; SearchStrategy: all) found beans named redisTemplate
- Bean method 'stringRedisTemplate' in 'RedisAutoConfiguration' not loaded because @ConditionalOnMissingBean (types: org.springframework.data.redis.core.StringRedisTemplate; SearchStrategy: all) found beans of type 'org.springframework.data.redis.core.StringRedisTemplate' stringRedisTemplate
Action:
Consider revisiting the entries above or defining a bean of type 'org.springframework.data.redis.core.RedisTemplate' in your configuration.
唉, 不能去掉啊..
升级到redisson-spring-boot-starter:3.11.6
升级redisson-spring-boot-starter:
dependency('com.l:lk-framework-core:2.4.01-RELEASE') {
exclude 'org.redisson:redisson-spring-boot-starter'
}
dependency 'org.redisson:redisson-spring-boot-starter:3.11.6'
还是有问题,
Description:
Parameter 0 of constructor in com.lktest.lk.framework.redis.config.RedisClusterConfig required a bean of type 'org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory' that could not be found.
The injection point has the following annotations:
- @org.springframework.beans.factory.annotation.Autowired(required=true)
The following candidates were found but could not be injected:
- Bean method 'redisConnectionFactory' in 'LettuceConnectionConfiguration' not loaded because @ConditionalOnMissingBean (types: org.springframework.data.redis.connection.RedisConnectionFactory; SearchStrategy: all) found beans of type 'org.springframework.data.redis.connection.RedisConnectionFactory' redissonConnectionFactory
Action:
Consider revisiting the entries above or defining a bean of type 'org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory' in your configuration.
坑爹的循环依赖
看来是 RedissonAutoConfiguration 和 LettuceConnectionFactory 冲突了, 而 RedissonAutoConfiguration 优先 ...
为什么没有升级之前好好的, 升级了 redisson就不行了呢? 各方查找, 终于发现原因. 原来 RedissonConnectionFactory 在3.9.1的时候是@AutoConfigureAfter(RedisAutoConfiguration.class) 后面变成了 AutoConfigureBefore !!
@Configuration
@ConditionalOnClass({Redisson.class, RedisOperations.class})
@AutoConfigureBefore(RedisAutoConfiguration.class)
@EnableConfigurationProperties({RedissonProperties.class, RedisProperties.class})
public class RedissonAutoConfiguration {
...
}
3.9.1
/**
*
* @author Nikita Koksharov
*
*/
@Configuration
@ConditionalOnClass(Redisson.class)
@AutoConfigureAfter(RedisAutoConfiguration.class)
@EnableConfigurationProperties(RedissonProperties.class)
public class RedissonAutoConfiguration {
...
}
而
/**
* {@link EnableAutoConfiguration Auto-configuration} for Spring Data's Redis support.
*
* @author Dave Syer
* @author Andy Wilkinson
* @author Christian Dupuis
* @author Christoph Strobl
* @author Phillip Webb
* @author Eddú Meléndez
* @author Stephane Nicoll
* @author Marco Aust
* @author Mark Paluch
*/
@Configuration
@ConditionalOnClass(RedisOperations.class)
@EnableConfigurationProperties(RedisProperties.class)
@Import({ LettuceConnectionConfiguration.class, JedisConnectionConfiguration.class })
public class RedisAutoConfiguration {
@Bean
@ConditionalOnMissingBean(name = "redisTemplate")
public RedisTemplate<Object, Object> redisTemplate(
RedisConnectionFactory redisConnectionFactory) throws UnknownHostException {
RedisTemplate<Object, Object> template = new RedisTemplate<>();
template.setConnectionFactory(redisConnectionFactory);
return template;
}
@Bean
@ConditionalOnMissingBean
public StringRedisTemplate stringRedisTemplate(
RedisConnectionFactory redisConnectionFactory) throws UnknownHostException {
StringRedisTemplate template = new StringRedisTemplate();
template.setConnectionFactory(redisConnectionFactory);
return template;
}
}
这样就导致了,
要是能够把那个 改成AutoConfigureAfter就好了, 可是.. 这样直接改第三方类库, 会不会引起其他问题?
怎么办? 修改一下己方的类库吧, 把己方RedisClusterConfig 里面的 LettuceConnectionFactory 改成 RedisConnectionFactory ( 其实这样是不合理的! 我们使用redis 不应该直接绑定是 lettuce 还是 jedis 的实现, 否则 底层就无法升级! ), 当然, 己方类库不能直接修改, 我们只能覆盖它, 就是把 这个修改后的java文件直接放置到src 的同package路径下, 然后就会覆盖! 因为它的优先级高于 jar 里面的同名类库!
放绝招, 覆盖己方类
我发现一个更好的解决方案, 那就是:
@Configuration
public class RedisClusterConfig {
private static final Logger logger = LoggerFactory.getLogger(RedisClusterConfig.class);
// private LettuceConnectionFactory lettuceConnectionFactory; // 不能直接依赖LettuceConnectionFactory
private RedisConnectionFactory lettuceConnectionFactory; // 这样就好了!!
@Value("${.cache.redis.defaultExpiration:1800000}")
private long defaultExpiration;
@Bean
public StringRedisTemplate stringRedisTemplate() {
return new StringRedisTemplate(this.lettuceConnectionFactory);
}
@Autowired
public RedisClusterConfig(RedisConnectionFactory lettuceConnectionFactory) {
this.lettuceConnectionFactory = lettuceConnectionFactory;
}
@Bean
public RedisTemplate<String, Long> stringLongRedisTemplate() {
RedisTemplate<String, Long> template = new RedisTemplate();
template.setConnectionFactory(this.lettuceConnectionFactory);
template.setKeySerializer(new StringRedisSerializer());
template.setHashValueSerializer(new GenericToStringSerializer(Long.class));
template.setValueSerializer(new GenericToStringSerializer(Long.class));
return template;
}
@Bean
public RedisTemplate<String, Object> lettuceRedisTemplate() {
Jackson2JsonRedisSerializer<Object> jackson2JsonRedisSerializer = this.getJackson();
RedisTemplate<String, Object> redisTemplate = new RedisTemplate();
redisTemplate.setConnectionFactory(this.lettuceConnectionFactory);
RedisSerializer<?> stringSerializer = new StringRedisSerializer();
redisTemplate.setKeySerializer(stringSerializer);
redisTemplate.setValueSerializer(jackson2JsonRedisSerializer);
redisTemplate.setHashKeySerializer(stringSerializer);
redisTemplate.setHashValueSerializer(jackson2JsonRedisSerializer);
redisTemplate.afterPropertiesSet();
return redisTemplate;
}
private Jackson2JsonRedisSerializer<Object> getJackson() {
Jackson2JsonRedisSerializer<Object> jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class);
ObjectMapper objectMapper = (new ObjectMapper()).registerModule(new ParameterNamesModule()).registerModule(new Jdk8Module()).registerModule(new JavaTimeModule());
objectMapper.setVisibility(PropertyAccessor.ALL, Visibility.ANY);
objectMapper.enableDefaultTyping(DefaultTyping.NON_FINAL);
jackson2JsonRedisSerializer.setObjectMapper(objectMapper);
return jackson2JsonRedisSerializer;
}
...
}
