I do apply patches to my SQL instances as and when they are released by Microsoft. This is important because I always feel keeping the bits up-to-date is essential because I don’t want to get infected as I travel quite a bit. Since I play a lot with my SQL Server, there are more chances that things are going to break sooner than your production server. Here is one such incident. Let us learn about how to fix script level upgrade. Let us learn how to fix the error related to SSISDB.
I was trying to apply the patch to my SQL instance and it went fine. But then I was NOT able to start the SQL Server service. Here are the messages from SQL ERRORLOG.
2018-03-31 11:44:51.63 spid6s Starting up database ‘tempdb’.
2018-03-31 11:44:54.93 spid5s Database ‘master’ is upgrading script ‘SSIS_hotfix_install.sql’ from level 184552836 to level 184555396.
2018-03-31 11:44:54.98 spid5s Error: 956, Severity: 14, State: 1.
2018-03-31 11:44:54.98 spid5s Database SSISDB is enabled for Database Mirroring, but has not yet synchronized with its partner. Try the operation again later.
2018-03-31 11:44:54.99 spid5s The failed batch of t-sql statements:
This again is an upgrade script failure issue. I have written few blogs having the same error.
Based on my previous experiences with such errors, I realized that this is a Script Upgrade failure. Below are the steps which occur during a patching process,
- Whenever any SQL Server patch is applied, the setup would patch the binaries first.
- During the restart of the instance, SQL Server startup would go through “script upgrade mode” during the recovery
- Script upgrade mode is the phase where objects inside the databases are upgraded based on recent patch applied/removed.
- Based on features installed and number of databases available, it would take a varying amount of time.
From above we can see that we got an error while executing the T-SQL Statement: SSIS_hotfix_install.sql’. We can also see that we have an additional error which gives us clarity about the problem caused here.
Database SSISDB is enabled for Database Mirroring, but has not yet synchronized with its partner.
When I checked further, I found that SSIS_hotfix_install.sql is a patch install script for the SSIS server catalog (SSISDB).
SOLUTION/WORKAROUND
As we have done every time on such issues, we had to use trace flag 902 to bypass script upgrade mode. We started SQL using trace flag 902 as below
NET START MSSQL$SQL2012 /T902
Refer: SQL SERVER – 2005 – Start Stop Restart SQL Server from Command Prompt
After this, I could connect to SQL Server because the problem script didn’t run due to trace flag. We could see that SSISDB was indeed present in this server. What is more interesting was that this server was acting as a mirrored server for the database SSISDB. Which means we cannot read/write in this database. If I go back to the basics of script upgrade mode, we try to update the metadata of such databases to match the build we have just installed.
So, in order to get the script upgrade completed for the SSISDB, to successfully run the script SSIS_hotfix_install.sql we had to get this database to a read/write mode which means we had to get this database online. Which also means we had to make this database a Principal copy.
I failed over this database as Principal copy database. We then re-started the instance removing TF902.
2018-03-31 15:30:58.33 spid5s Schema build in SSISDB has been updated to 11.0.6020.0
2018-03-31 15:30:58.41 spid31s Database mirroring is active with database ‘DB1’ as the mirror copy. This is an informational message only. No user action is required.
2018-03-31 15:31:05.73 spid30s Database mirroring is active with database ‘DB2’ as the mirror copy. This is an informational message only. No user action is required.
2018-03-31 15:31:56.95 spid5s ——————————————————
2018-03-31 15:31:56.95 spid5s Execution of SSIS_HOTFIX_INSTALL.SQL completed
SQL Server services started fine, and we could see that the “SSIS_hotfix_install.sql” also got applied successfully.
Have you encountered such errors in production where upgrade script failed? How did you fix it?
Reference: Pinal Dave (https://blog.sqlauthority.com)
First appeared on SQL SERVER – Unable to Start SQL After Patching – Database SSISDB is Enabled for Database Mirroring