Recently once by my client contacted via Skype to know my thoughts about a cluster failover failure issue. They were having two nodes SQL Clustered instance and it was running fine of Node1. As soon as they failover to Node2, it was not able to start and was failing with upgrade script. Let us learn about the script level upgrade for database ‘master’ failed because upgrade step ‘upgrade_ucp_cmdw.sql’.
2016-08-16 10:07:00.52 Logon Error: 18401, Severity: 14, State: 1.
2016-08-16 10:07:00.52 Logon Login failed for user ‘NT AUTHORITY\SYSTEM’. Reason: Server is in script upgrade mode. Only administrator can connect at this time. [CLIENT: 172.19.10.94]
2016-08-16 10:07:00.55 spid9s Restoring database to multi user mode before aborting the script
2016-08-16 10:07:00.55 spid9s Setting database option MULTI_USER to ON for database sysutility_mdw.
2016-08-16 10:07:00.55 spid9s Error: 14714, Severity: 21, State: 1.
2016-08-16 10:07:00.55 spid9s Attempting to upgrade a Management Data Warehouse of newer version ‘10.50.4042.0’ with an older version ‘10.50.4000.0’. Upgrade aborted.
2016-08-16 10:07:00.55 spid9s Error: 2745, Severity: 16, State: 2.
2016-08-16 10:07:00.55 spid9s Process ID 9 has raised user error 14714, severity 21. SQL Server is terminating this process.
2016-08-16 10:07:00.55 spid9s Error: 912, Severity: 21, State: 2.
2016-08-16 10:07:00.55 spid9s Script level upgrade for database ‘master’ failed because upgrade step ‘upgrade_ucp_cmdw.sql’ encountered error, state 2, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
2016-08-16 10:07:00.55 spid9s Error: 3417, Severity: 21, State: 3.
2016-08-16 10:07:00.55 spid9s Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.
2016-08-16 10:07:00.55 spid9s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.
I asked to share the complete ERRORLOG and here is what I saw.
Here is one old blog on the same topic.
Here is an interesting line of the error:
Attempting to upgrade a Management Data Warehouse of newer version ‘10.50.4042.0’ with an older version ‘10.50.4000.0’. Upgrade aborted.
Above line means upgrade is aborted because SQL is trying to upgrade from newer version to an older version. As per online documentation build 10.50.4042 came from MS15-058: Description of the security update for SQL Server 2008 R2 Service Pack 2 GDR: July 14, 2015 and 10.50.4000 is SQL Server 2008 R2 Service Pack 2 (SP2)
This means where was a version mismatch of SQL Server binaries on both the nodes.
Solution/Workaround
We looked into version of SQLServr.exe on both nodes and found that the problem node was having version 10.50.4000 whereas working node was having version of 10.50.4042.
As soon as we applied the same patch on both the nodes, failover was working like a charm!
Reference: Pinal Dave (http://blog.sqlauthority.com)
First appeared on SQL SERVER – Script level upgrade for database ‘master’ failed because upgrade step ‘upgrade_ucp_cmdw.sql’