It’s always a good experience to visit customer sites and talk to people. Sometimes I get to see things outside SQL world as well. There is a lot to learn and I believe that I can do that by sharing what I learned. In this blog post we will discuss Why Cluster Network is Unavailable in Failover Cluster Manager?
During my last visit to an India based company, I was talking to a windows admin during lunch and he was talking about a cluster issue. It was an interesting conversation where he told that sometimes a reboot is THE solution to solve a problem. He told me an incident where Cluster networks were shown as unavailable in failover cluster manager. After lunch, I went to his desk to get more details.
As we can see under box created around Nodes, this was only with one node.
When we look at cluster logs, we see below the messages.
========B02===========
00000648.00002464::2016/11/29-08:58:45.173 INFO [FTI][Initiator] This node (1) is initiator
00000648.00002464::2016/11/29-08:58:45.173 WARN [FTI][Initiator] Ignoring duplicate connection: usable route already exists
00000648.00002464::2016/11/29-08:58:45.173 INFO [CHANNEL 147.170.123.251:~3343~] graceful close, status (of previous failure, may not indicate problem) ERROR_SUCCESS(0)
00000648.00002464::2016/11/29-08:58:45.174 WARN cxl::ConnectWorker::operator (): GracefulClose(1226)’ because of ‘channel to remote endpoint 147.170.123.251:~3343~ is closed’
========B01============
00004090.00005db0::2016/11/29-08:58:45.157 INFO [FTI][Follower] This node (2) is not the initiator
00004090.00005db0::2016/11/29-08:58:45.157 DBG [FTI] Stream already exists to node 1: false
00004090.00005db0::2016/11/29-08:58:45.157 DBG [CHANNEL 147.170.123.252:~54783~] Close().
00004090.00005db0::2016/11/29-08:58:45.157 INFO [CHANNEL 147.170.123.252:~54783~] graceful close, status (of previous failure, may not indicate problem) ERROR_SUCCESS(0)
00004090.00005db0::2016/11/29-08:58:45.157 INFO [CORE] Node 2: Clearing cookie 63cfe37d-42be-4211-8cd8-6db6b3344b52
00004090.00005db0::2016/11/29-08:58:45.157 DBG [CHANNEL 147.170.123.252:~54783~] Not closing handle because it is invalid.
00004090.00005db0::2016/11/29-08:58:45.157 WARN mscs::ListenerWorker::operator (): GracefulClose(1226)’ because of ‘channel to remote endpoint 147.170.123.252:~54783~ is closed’
Based on cluster logs and highlighted message “Ignoring duplicate connection: usable route already exists”, we can say that this issue is caused due to stale information on network from rejecting node.
The only solution to fix the error was to reboot the active node.
I search on internet and found that this could be because of real network issue, some antivirus software as well. So, if above message is not shown in cluster log, then you can search further. Please share the solution if you find.
Reference: Pinal Dave (http://blog.sqlauthority.com)
First appeared on SQL SERVER – Why Cluster Network is Unavailable in Failover Cluster Manager?