During my last consulting engagement, I was pulled by my client to consider an issue which they were facing. They informed that they have applied service pack on one of their clustered environment and since than SQL Server is not coming online. I asked to share ERRORLOG from the SQL instance. SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location. Let us learn about how to fix error after cluster patching.
2016-11-20 21:09:49.44 spid9s Starting execution of PREINSTMSDB100.SQL
2016-11-20 21:09:49.44 spid9s —————————————-
2016-11-20 21:10:01.67 spid9s Error: 5184, Severity: 16, State: 2.
2016-11-20 21:10:01.67 spid9s Cannot use file ‘D:\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\temp_MS_AgentSigningCertificate_database.mdf’ for clustered server. Only formatted files on which the cluster resource of the server has a dependency can be used. Either the disk resource containing the file is not present in the cluster group or the cluster resource of the Sql Server does not have a dependency on it.
2016-11-20 21:10:01.67 spid9s Error: 1802, Severity: 16, State: 1.
2016-11-20 21:10:01.67 spid9s CREATE DATABASE failed. Some file names listed could not be created. Check related errors.
2016-11-20 21:10:01.67 spid9s Error: 912, Severity: 21, State: 2.
2016-11-20 21:10:01.67 spid9s Script level upgrade for database ‘master’ failed because upgrade step ‘sqlagent100_msdb_upgrade.sql’ encountered error 598, state 1, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
2016-11-20 21:10:01.67 spid9s Error: 3417, Severity: 21, State: 3.
2016-11-20 21:10:01.67 spid9s Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.
2016-11-20 21:10:01.67 spid9s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.
The start of the problem is Error: 5184, Severity: 16, State: 2.
If we look at error message is clear that the D drive is not having dependency with the SQL Server resource. We checked failover cluster manager and found below.
As we can see we have only cluster disk 4 which was E drive. We added by clicking on a highlighted area. Once we added the disk we found that issue was still not solved and SQL was not coming online. Checked ERRORLOG again and found a new problem.
2016-11-20 21:09:48.32 Logon Error: 18456, Severity: 14, State: 11.
2016-11-20 21:09:48.32 Logon Login failed for user ‘NT AUTHORITY\SYSTEM’. Reason: Token-based server access validation failed with an infrastructure error. Check for previous errors. [CLIENT: 100.168.11.171]
I asked them series of action and they informed that they have already attempted to rebuild the system databases – which was a news to me. So now the problem was that this login was not existing in SQL Server as System databases were rebuilt. Here were the steps to fix this issue.
- Start SQL using command prompt
NET START MSSQLSERVER /m
- Added ‘NT AUTHORITY\SYSTEM’ account
- Stopped SQL Server
NET STOP MSSQLSERVER
After this we could bring SQL Server online and issue was resolved. Have you seen a similar issue where rebuild was done in the cluster and it didn’t work?
Reference: Pinal Dave (http://blog.sqlauthority.com)
First appeared on SQL SERVER – Error After Cluster Patching – Error: 5184, Severity: 16, State: 2