This was an interesting scenario I encountered when configuring one of the clusters. I had finished installing SQL Server on Node1 and had started running ADD NODE on Node2. We got this error in the setup wizard before reaching the Service Account Password page.

SQL Server Setup has encountered the following error:
Could not find subkey System\CurrentControlSet\Services\MSSQLFDLauncher
Error code 0x851B0001.

I searched on the internet and there were various users facing same error. Most of them didn’t share the solution and those solutions, which were available didn’t work for me. In this blog, we would learn how to fix error “could not find subkey” with code 0x851B0001

It is important to note that actual add node installation didn’t start yet and we were in the wizard. We got this error in the setup wizard just before the Service Account Password page.

MY ANALYSIS

Based on my setup experience, I can tell that during the ADD Node procedure, SQL Server setup performs a remote registry query to get the details of the configurations which were set in Node 1. Looks like the setup expected this key to be present there, but unfortunately, it did not find one and thus gave the error. The service and key MSSQLFDLauncher is for SQL Server Full-Text Search. We went to Node 1 and found Full-Text Search was not installed!

Now customer told us an interesting thing. They did not see Full-Text Search option while installing on the first node. So, we decided to remove SQL Server from Node 1 and try to re-install.  While re-installing, on the Features Page we were not able to see the Full-Text Search option.

SOLUTION/WORKAROUND

After a bit of testing on my lab server, I was able to reproduce the issue! We will not see a feature listed on the Features Page if the respective MSI file is missing from the Setup Media. Full-Text Search using the MSI called sql_fulltext.msi and is present in the setup path MediaRoot\x64\Setup. I was not able to see the sql_fulltext.msi file in the client’s setup media. I was suspecting other files to be missing too so we extracted a fresh copy of the setup media (from new downloaded ISO) and then ran the “New Failover Cluster” setup. This time, we were able to see Full-Text Search option. Let us learn how to fix the error related to unable to add node.

We selected the components and went further with the setup. Setup completed fine on the first Node. Then we used the same “good” media to run Add Node and that too completed successfully.

Have you encountered such issue? Was it something other than setup media issue? Please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Add Node – Could not find subkey System \ CurrentControlSet \ Services \ MSSQLFDLauncher. Error code 0x851B0001

While playing with my lab cluster, I ran into a situation. In this blog, we would learn about a situation where failover was not working from one node to another node. Everything used to work fine on Node1. Let us fix the error related to failover clustered.

Errors from System Events logs:

01/04/2018 04:02:16 PM
Error
1069
Microsoft-Windows-failoverclustering
Cluster resource ‘SQL Server (InstA)’ of type ‘SQL Server’ in clustered role ‘Instance_A’ failed.

01/04/2018 04:02:19 PM
Information
7036
Service Control Manager
The SQL Server (InstA) service entered the running state.

01/04/2018 04:04:04 PM
Information
7036
Service Control Manager
The SQL Server (InstA) service entered the stopped state.

System Events logs pretty much showed generic errors so, I moved on to the cluster logs and isolated to the timeline when I tested the failover. Here is my earlier blog to learn how to generate cluster logs.

SQL SERVER – Steps to Generate Windows Cluster Log?

Errors from cluster log. Remember that by default time is in UTC time-zone. In Cluster log, ERR and WARN are generally the interesting messages. The above-highlighted error was very interesting. And when I spent more time in the cluster log, I unearthed these below errors:

0000173c.000029f4::2018/04/01-11:32:16.564 ERR [RHS] s_RhsRpcCreateResType: (126)’ because of ‘Error loading resource DLL sqagtres.dll.’
00000ffc.000027d8::2018/04/01-11:32:16.564 INFO [RCM] result of first load attempt for type SQL Server Agent: 126
0000173c.000029f4::2018/04/01-11:32:16.604 ERR [RHS] s_RhsRpcCreateResType: (126)’ because of ‘Error loading resource DLL sqsrvres.dll.’
00000ffc.000027d8::2018/04/01-11:32:16.604 INFO [RCM] result of first load attempt for type SQL Server: 126

Both the errors found above are talking about a possible faulty resource DLLs for SQL Server resources. We can see number 126 which is the error code. As we can see below, error number 126 = The specified module could not be found.

Since the error above talked about the DLL not being loaded, I tried to register the DLL in question and got the below error:

The module “SQSRVRES.dll” failed to load.
Make sure the binary is stored at the specified path or debug it to check for problems with the binary or dependent .DLL files.
The specified module could not be found.

This is the same what we saw in cluster log earlier.

WORKAROUND/SOLUTION

By looking at various errors so far, I was concerned if the SQL cluster related DLLs are actually present. So, I tried to verify the same in the path C:\windows\system32 folder. Surprisingly, I was not able to find sqsrvres.dll and sqagtres.dll. Instead, both were renamed as,

sqsrvres2.dll

sqagtres2.dll

I renamed the DLLs to the required name sqsrvres.dll and sqagtres.dll. After that, we were able to start SQL resource from Failover Cluster Manager. I also rebooted the affected node so that all the resources come online on their own so that all the DLLs loading happening automatically.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Event ID 1069 – Unable to Failover Clustered Instance to Another Node. Resource is not Loaded

In the current era where companies’ mergers and take over is a common phenomenon, it is possible to run into a situation where there are two domains which are connected but not trusted. I was engaged in consulting for a client who had a similar merger in the past. The reporting application was in a different domain and they wanted data to be synchronized in SQL Server database on parent domain. Essentially, they had two windows clusters and they want to have one AG across them. The only solution which came to my mind was Distributed Availability Group in SQL Server. To make things little more complicated, they did not want to use certificates to configure this setup. As per them, certificates will add an overhead of maintenance. Here is the oversimplified diagram of their deployment.

SOLUTION/WORKAROUND

After contemplating and testing for a while, we decided to make use of NTLM Pass-Through Authentication between the domains and were also successful in doing the same. Below were high-level steps followed to achieve the same.

• Create a local windows user in each of the nodes (N1, N2, N3, N4) across domains using the same name and password. Note: Only the NodeName will change here. Username and password must the same.

For example N1\DAGUser, N2\DAGUser, N3\DAGUser and so on.

• Add this account to Local Administrators group on all the nodes across domains
• Change SQL Server startup account to this newly created account on all the nodes across domains (via SQL Server Configuration Manager).
• Create the Primary Availability Group (HA_AG) with a corresponding listener name (HA _LIST)
• Grant the SQL Server service account CONNECT permissions to the endpoint. Run on ALL nodes — GRANT CONNECT ON ENDPOINT::Hadr_endpoint TO [<nodename>\UserName]
• Each node should have “connect” permission from all other three nodes.
• Create the secondary Availability Group (DR_AG) with a corresponding listener name (DR_LIST)
• Join the secondary replicas to the secondary Availability Group
• Add entries to the HOST File with “IPAddr and Listener FQDN” info of the other domain. i.e. Domain1 nodes will contain [1.1.xxx DR_LIST.MyDomain2.com] Domain2 nodes will contain [192.168.1.xxx HA_LIST.MyDomain.com]
• Create Distributed Availability Group (HA_DR_AG) on the primary Availability Group (HA_AG)
• Join the Secondary Availability Group (DR_AG) to the Distributed Availability Group

There were some hiccups in deployment but overall, above steps worked and we were able to fulfill their requirement. If you have such consultancy requirements for Always On, feel free to contact me.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Steps to Deploy Distributed Availability Group – Windows Clusters in Different Domains

I do apply patches to my SQL instances as and when they are released by Microsoft. This is important because I always feel keeping the bits up-to-date is essential because I don’t want to get infected as I travel quite a bit. Since I play a lot with my SQL Server, there are more chances that things are going to break sooner than your production server. Here is one such incident. Let us learn about how to fix script level upgrade. Let us learn how to fix the error related to SSISDB.

I was trying to apply the patch to my SQL instance and it went fine. But then I was NOT able to start the SQL Server service. Here are the messages from SQL ERRORLOG.

2018-03-31 11:44:51.63 spid6s Starting up database ‘tempdb’.
2018-03-31 11:44:54.93 spid5s Database ‘master’ is upgrading script ‘SSIS_hotfix_install.sql’ from level 184552836 to level 184555396.
2018-03-31 11:44:54.98 spid5s Error: 956, Severity: 14, State: 1.
2018-03-31 11:44:54.98 spid5s Database SSISDB is enabled for Database Mirroring, but has not yet synchronized with its partner. Try the operation again later.
2018-03-31 11:44:54.99 spid5s The failed batch of t-sql statements:

This again is an upgrade script failure issue. I have written few blogs having the same error.

SQL SERVER – Script level upgrade for database ‘master’ failed because upgrade step ‘sqlagent100_msdb_upgrade.sql’

SQL SERVER – Script level upgrade for database ‘master’ failed – There is already an object named ‘DatabaseMailUserRole’ in the database

SQL SERVER – Script level upgrade for database master failed – Error: 4860, Severity: 16, State: 1 – Cannot bulk load. SqlTraceCollect.dtsx does not exist

Based on my previous experiences with such errors, I realized that this is a Script Upgrade failure. Below are the steps which occur during a patching process,

• Whenever any SQL Server patch is applied, the setup would patch the binaries first.
• During the restart of the instance, SQL Server startup would go through “script upgrade mode” during the recovery
• Script upgrade mode is the phase where objects inside the databases are upgraded based on recent patch applied/removed.
• Based on features installed and number of databases available, it would take a varying amount of time.

From above we can see that we got an error while executing the T-SQL Statement: SSIS_hotfix_install.sql’. We can also see that we have an additional error which gives us clarity about the problem caused here.

Database SSISDB is enabled for Database Mirroring, but has not yet synchronized with its partner.

When I checked further, I found that SSIS_hotfix_install.sql is a patch install script for the SSIS server catalog (SSISDB).

SOLUTION/WORKAROUND

As we have done every time on such issues, we had to use trace flag 902 to bypass script upgrade mode. We started SQL using trace flag 902 as below

NET START MSSQL$SQL2012 /T902

Refer: SQL SERVER – 2005 – Start Stop Restart SQL Server from Command Prompt

After this, I could connect to SQL Server because the problem script didn’t run due to trace flag. We could see that SSISDB was indeed present in this server. What is more interesting was that this server was acting as a mirrored server for the database SSISDB. Which means we cannot read/write in this database. If I go back to the basics of script upgrade mode, we try to update the metadata of such databases to match the build we have just installed.

So, in order to get the script upgrade completed for the SSISDB, to successfully run the script SSIS_hotfix_install.sql we had to get this database to a read/write mode which means we had to get this database online. Which also means we had to make this database a Principal copy.

I failed over this database as Principal copy database. We then re-started the instance removing TF902.

2018-03-31 15:30:58.33 spid5s Schema build in SSISDB has been updated to 11.0.6020.0
2018-03-31 15:30:58.41 spid31s Database mirroring is active with database ‘DB1’ as the mirror copy. This is an informational message only. No user action is required.
2018-03-31 15:31:05.73 spid30s Database mirroring is active with database ‘DB2’ as the mirror copy. This is an informational message only. No user action is required.
2018-03-31 15:31:56.95 spid5s ——————————————————
2018-03-31 15:31:56.95 spid5s Execution of SSIS_HOTFIX_INSTALL.SQL completed

SQL Server services started fine, and we could see that the “SSIS_hotfix_install.sql” also got applied successfully.

Have you encountered such errors in production where upgrade script failed? How did you fix it?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Start SQL After Patching – Database SSISDB is Enabled for Database Mirroring

Whenever we try to use Configure Distribution option in SSMS in a SQL Server instance which is configured on a UNC Path, the wizard will not allow us to configure the Distribution database and we will get the below error:

The paths to the distribution database folder and transaction log folder must refer to drives that are local to ‘SQLNODE1’. The paths must begin with a local drive letter and colon (for example, C:). Mapped drive letters or network paths are not valid.

Don’t be surprised. SQL Server instances can be configured on a UNC Share. Starting with SQL Server 2012, system databases and user databases can be installed with Server Message Block (SMB) file server as a storage option. This applies to both SQL Server stand-alone and SQL Server failover cluster installations. The only exception is Filestream which is not currently supported on an SMB file share.

In this scenario, when we are trying to configure Distribution database, the Configure Distribution wizard was not happy with the UNC shares provided and we got an error. I’m not sure if this is a supported model, but it looks like it should be supported as SQL Server is supporting all other system databases.

WORKAROUND/SOLUTION

I adopted the T-SQL method to get the Distribution Database successfully configured. Below are the steps I followed.

In the Configure Distribution wizard, provide an existing path, C:\Temp (or any other valid path)

Then in the Wizard Actions page, uncheck “Configure Distribution” and check “Generate a Script …”

Go ahead and finish the wizard. Then open the script which was saved. It will look like the one below (partial script)

use master
exec sp_adddistributor @distributor = N'SQLAuthorityServer', @password = N'aVerY$tr0nggP@ssw00rd'
GO
exec sp_adddistributiondb @database = N'distribution', @data_folder = N'C:\Temp', @log_folder = N'C:\Temp', @log_file_size = 2, @min_distretention = 0, @max_distretention = 72, @history_retention = 48, @security_mode = 1
GO

We need to change the values “C:\Temp” above to the UNC path where we need to get the Distribution Database configured. Now the script will look like,

use master
exec sp_adddistributor @distributor = N'SQLAuthorityServer', @password = N'aVerY$tr0nggP@ssw00rd'
GO
exec sp_adddistributiondb @database = N'distribution', @data_folder = N' \\dc\ShareOnDC', @log_folder = N' \\dc\ShareOnDC', @log_file_size = 2, @min_distretention = 0, @max_distretention = 72, @history_retention = 48, @security_mode = 1
GO

Note: Make sure this UNC Path has been given all the required permissions. Refer here

Once the values have been changes, please go ahead and run this script in a Query Analyzer window. After the script has executed, you will see the Distribution database created. You can also verify the Distribution database by running,

USE [distribution]
GO
SELECT name
,physical_name
FROM   sys.database_files

Here is the output which confirms the file location.

Hope this helps someone who wants to have such special requirement and configuration.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Unable to Create Distribution Database on a UNC Share – Configure Distribution

I encountered this error while installing KB3045311. In fact, you might get such errors when installing any patch for one of my customers. The error was the very popular system error 5.

At first, I thought I am already aware of this error and in fact, I had already blogged about the same.

SQL SERVER – FIX – Error 1402. Could not open key: UNKNOWN\Components. System error 5

If the solutions given the above article does not help, you may try following the solution we found in this case. To follow the steps, you need to first look at SQL Setup Logs. In my case, there were 3 components that had failed which I found by looking into the setup log files. I always search for “return value 3” in MSI logs to find the error.

Product: Microsoft SQL Server VSS Writer — Error 1402. Could not open key: UNKNOWN\Components\7ABFE44842C12B390AF18C3B9B1A1EE8\65E5D8C621AC2B24095740B4C407769A.  System error 5.  Verify that you have sufficient access to that key, or contact your support personnel.

Once you have identified that long strange number after “components”, make a note as we need them later.

WORKAROUND/SOLUTION

Notice carefully, every error seen above is pointing to the same registry path: UNKNOWN\Components\7ABFE44842C12B390AF18C3B9B1A1EE8\65E5D8C621AC2B24095740B4C407769A

What is UNKNOWN there? Well, in the registry check and we should able to navigate until: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Components\7ABFE44842C12B390AF18C3B9B1A1EE8

But when I tried to access any GUID under this we get an Access Denied. We followed the below steps to get rid of Access Denied message

Follow the below steps on each GUID

• Right Click the GUID –> go to Permissions
• Click on Advanced –> go to Owner –> Choose <hostname>\administrators –> Click Ok –> Click Apply –> Click Ok –> Click Ok
• Right Click the GUID –> go to Permissions –>
• Click on Advanced –> Click Add –> Type <hostname>\administrators and click Ok –> Click Ok –> Click Ok

After this we were able to access the all the subkeys under, HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Components\7ABFE44842C12B390AF18C3B9B1A1EE8

After following all the above steps, we no longer had issues installing KB3045311. If you find any other solution for above error, please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX – Error 1402 – Could not Open Key: UNKNOWN\Components – System Error 5 (Another Solution)

Anything that can go wrong will go wrong while Murphy is out of town. While I was onsite to help a client, one such issue had cropped up in their SharePoint environment where SQL Server was used as backend and was a part of Windows failover cluster. When SQL Server was failed over from Active to the Passive node, users were not able to connect to SharePoint site and they were not able to open any SharePoint page. Since it was critical for their business, I got engaged to provide suggestions.

When they failed over, we could see that the SQL Server group had failed over successfully and all the SQL related resources were up and running. It’s just that the SharePoint users, who were not able to do anything. I started with basics and started to look in the SQL Server Errorlog. The below error was continuously getting logged.

2018-04-10 18:49:09.47 Logon       Error: 18456, Severity: 14, State: 38.
2018-04-10 18:49:09.47 Logon       Login failed for user ‘Domain\UserName’. Reason: Failed to open the explicitly specified database. [CLIENT: xx.xx.xx.xx]

Unfortunately, above error message didn’t tell us the database name. We run a profiler to get this information and from the profiler output, we could see that the above error was getting logged for the SharePoint content database – MySite_ContentDB

Then we tried to access this database from SSMS –

USE [MySite_ContentDB]

It failed with below error message.

Msg 945, Level 14, State 2, Line 1
Database ‘MySite_Content’ cannot be opened due to inaccessible files or insufficient memory or disk space.

I have double checked and made sure the database files are available at their desired location and made sure the proper permissions were set on these files. Still, we were just not able to access this database via GUI in SSMS or using any T-SQL commands.

Suspecting corruption and just thinking this database might not have come online, I tried to set the ONLINE status for this database

ALTER DATABASE [MySite_Content] SET ONLINE

This gave us some meaningful error(s).

Msg 5591, Level 16, State 5, Line 1
FILESTREAM feature is disabled.
Msg 5105, Level 16, State 14, Line 1
A file activation error occurred. The physical file name ‘:\’ may be incorrect. Diagnose and correct additional errors, and retry the operation.
Msg 945, Level 14, State 2, Line 1
Database ‘MySite_Content’ cannot be opened due to inaccessible files or insufficient memory or disk space. See the SQL Server errorlog for details.
Msg 5069, Level 16, State 1, Line 1
ALTER DATABASE statement failed.

Now everything made sense in this world. As the error said, filestream feature was not enabled on this node. Surprisingly, it was enabled on Node1. The documentation mentions that SharePoint application uses the RBS feature to store binary large objects (BLOBs) outside the content database. For this, FileStream feature needs to be enabled on the database server hosting the content database. And now because this is a clustered SQL instance, this feature needs to be enabled on all the participating nodes.

I asked my client to enable FileStream on this node. Refer Enable FileStream

After that SharePoint sites started working and we could also run the commands which were failing above.

ALTER DATABASE [MySite_Content] SET ONLINE
GO
USE [MySite_Content]
GO

My client also tested multiple failovers and made sure the SharePoint sites were working fine. I always thought that changes to clustered SQL Server instance, it would be done automatically on other nodes but looks like this didn’t happen for this client. I didn’t get time to investigate the cause. Have you come across analogous situation? Do you know the cause? Please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – SharePoint not Working After Failover to Another Node

One of my blog readers contacted me and informed that he is getting the same error as mentioned in below blog SQL SERVER – Service Pack Failure: MissingRegistryKeyException. Note that it had the same error message, but that did not help him here. In this blog, we would learn how to fix MissingRegistryKeyException while installing SQL Server service pack. Let us fix the error registry key is missing.

Below is the trimmed version of log (Detail.txt)

Microsoft.SqlServer.Configuration.SqlEngine.MissingRegistryKeyException: The registry key System\CurrentControlSet\Services\MSSQLSERVER\Performance is missing.
Microsoft.SqlServer.Configuration.SqlEngine.Globals.GetSqlRegistryKeyHKLM
Microsoft.SqlServer.Configuration.SqlEngine.PerfmonRegistration.GetDLLNameFromRegistry
Microsoft.SqlServer.Configuration.SqlEngine.PerfmonRegistration.NeedToPatch
Microsoft.SqlServer.Configuration.SqlEngine.SqlEngineSetupPrivate.PatchPerfmonCounters
Microsoft.SqlServer.Configuration.SqlEngine.SqlEngineSetupPrivate.Patch
Microsoft.SqlServer.Configuration.SqlConfigBase.SlpConfigAction.ExecuteAction
Microsoft.SqlServer.Configuration.SqlConfigBase.SlpConfigAction.Execute
Exception: Microsoft.SqlServer.Configuration.SqlEngine.MissingRegistryKeyException.
Source: Microsoft.SqlServer.Configuration.SqlServer_ConfigExtension.

Since he already tried following my earlier blog, I was wondering what to do next. A thought of running a REPAIR of SQL Server instance came to my mind. I quickly checked what all actions do a REPAIR take care.

One of the points mentioned in the above article was, And, this brought a smile on my face.

Here is the repair option from LandingPage in setup. Below image is taken from SQL Server 2017 setup media bit option remains same (SQL 2008 onwards)

SOLUTION/WORKAROUND

I suggested him to go ahead and use SQL Server REPAIR using the RTM Media and asked to re-run the patch. He sent me an email that he again tried to run the service pack after repair and it was completed without any errors.

Have you used Repair option in SQL Server setup? When did you use, and did it help? Please let me know via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Service Pack Failure: MissingRegistryKeyException. The Registry Key is Missing

SQL installation and parching are the processes which every DBA has to do. Some smart DBAs do this via script and some go with a manual approach. When it fails, it is fun to debug it because you have to take help from internet to find possible causes and solution. In this blog, we would learn about how to fix error Unable to open Windows Installer file -2147286960.

One day I was trying to install a standalone instance of SQL Server 2016, on the setup wizard I encountered the below error

Here is the text of the error message.

The following error has occurred:
Unable to open Windows Installer file ‘C:\SQL2016\SQLServer2016SP1-FullSlipstream-x64-ENU\x64\setup\sql_as.msi’.
Windows Installer error message: 1: 2203 2: C:\SQL2016\SQLServer2016SP1-FullSlipstream-x64-ENU\x64\setup\sql_as.msi 3: -2147286960.
Click ‘Retry’ to retry the failed action, or click ‘Cancel’ to cancel this action and continue setup.

I directly went to the location mentioned above and saw that the file in question was already present.

Let me make it more interesting. I saw something strange. See below,

As we could see the subject description for all other MSI files, but it was blank for this MSI file. Then I decided to view the info present in all the columns of this MSI file. And I saw the below:

The size of this file was 0KB. This was very bizarre and interesting. This could have happened when I was extracting the ISO image. This was a new image I had downloaded.

WORKAROUND/SOLUTION

I re-extracted the ISO image after which I got the file in the correct state:

You can also copy only this file from a different image and replace here.  After doing above steps, I was able to fix the error and setup completed without any issues.

I generally prefer mounting ISO to the OS and then install rather than using tools to unzip the files. Have you ever seen issues with the extraction of ISO?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Installation Error – Unable to Open Windows Installer File – 2147286960

Just the other day one of the readers sent me following email after reading my earlier blog post: STRING_SPLIT Function Performance Comparison. He really liked the new feature of SQL Server 2016 and decided to start using this new feature. However, as soon as he started to use this feature, he received following error. He was very much confused why he was not able to use the STRING_SPLIT function even though he had SQL Server 2016.

He attempted to run following command in SQL Query Editor.

DECLARE @VarString NVARCHAR(400) = 'Mike,John,Miko,Matt';
SELECT value
FROM STRING_SPLIT(@VarString, ',');

When he tried to run the command he got following error:

Msg 208, Level 16, State 1, Line 7
Invalid object name ‘STRING_SPLIT’.

He immediately wrote to me asking if I know the reason for the error and if there is any way I can help him to fix the error. Honestly, I immediately knew the reason what was the problem for him as many of my customers often faces similar issues.

WORKAROUND/SOLUTION

The primary reason for the error to show up is the compatibility level. The new function STRING_SPLIT is introduced in SQL Server 2016 and if your database compatibility is of SQL Server 2014 or earlier version, you may end up with this error. If you want to use this function, you need to set your compatibility of the level to SQL Server 2016 or later version of SQL Server.

Here is the quick script which can help you to change your compatibility to SQL Server 2016.

ALTER DATABASE [AdventureWorks2014]
SET COMPATIBILITY_LEVEL = 130 -- For SQL Server 2016
GO

You should change the compatibility level value to 140 if you want to set your database compatibility to SQL Server 2017.

Once you change your database compatibility level to either SQL Server 2016 or SQL Server 2017, the script just worked fine and produced following results.

Please note that you should set your database compatibility to the level which is appropriate for your database. Changing the compatibility of the database without proper testing may result in errors.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Fix Error: Invalid object name STRING_SPLIT

My virtual machines are my playground and I do install/uninstall a lot of stuff there (of course related to SQL Server learning). When things break, I get to learn something new. In this blog we would talk about how to troubleshoot and fix Could not load file or assembly error. This particular error can come in many flavors. Here is another form of the same error.

Could not load file or assembly ‘Microsoft.SqlServer.BatchParser, Version=11.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91’ or one of its dependencies. The system cannot find the file specified

WORKAROUND/SOLUTION

You need to notice few things in the error message. I have shown the values as per the error message

1. Assembly Name: Microsoft.SqlServer.BatchParser
2. Version: 11.0.0.0
3. Error Message: The system cannot find the file specified

If you check C:\WINDOWS\ASSEMBLY folder, you would see them like below.

Here is the version table:

Now, read the error message again and it would start making sense. We can see that 11.0.0.0 is missing from assembly folder. All we need is to install the right version of SMO (Shared Management Objects) and SQLCLR from feature pack. You can refer below article to know more about feature pack. Since 11.0.0.0 is missing, we need SQL 2012 version of the feature pack.

SQL SERVER – How to Download SQL Server Native Client?

In the article, I talked about Native Client but for current error, we need to download these files:

• ENU\x64\SQLSysClrTypes.msi for the X64 version
• ENU\x86\SQLSysClrTypes.msi for the X86 version
• ENU\x64\SharedManagementObjects.msi for the X64 version
• ENU\x86\SharedManagementObjects.msi for the X86 version

Hopefully, this blog would help some developer who has no idea about SQL Server versions jargon.

Reference: Pinal Dave (https://blog.sqlauthority.com) 

First appeared on SQL SERVER – Error: Could not Load File or Assembly Microsoft. SqlServer. management. sdk. sfc Version 12.0.0.0

If you are a SQL DBA then this blog might be a quick refresher for you. I am writing this for Data Protection Manager (DPM) Administrators who are not aware of the cause of the issue. In this blog, we would learn about how to fix error raised by DPM – “The backup failed because of a gap in the LSN the Log files”

One of my client who was not an expert in SQL contacted me for below error which was reported in DPM log.

Description: Last 1 recovery points not created. DPM tried to do a SQL log backup, either as part of a backup job or a recovery to a latest point in time job. The SQL log backup job has detected a discontinuity in the SQL log chain for database since the last backup. All incremental backup jobs will fail until an express full backup runs. For more information, open DPM Administrator Console and review the alert details in the Monitoring task area.

He had no idea about what needs to be done. So, he contacted me for On Demand (55 minutes) services. Even if I didn’t know much about DPM, I took this request and learned something new about DPM.

WORKAROUND/SOLUTION

In SQL Server full recovery model transaction log file gets truncated after transaction log backup. It looks like if the DPM server finds that there is a discontinuity in the log backup and it raises an error. This also means that some other backup application (mostly SQL Server Native Backup) is taking the log backup LSN is not in sequence for backups taken by DPM. If you are an SQLDBA, you can think it as LSN chain break which might happen in log-shipping when we take out-of-band backup.

SQL SERVER – Log Shipping Restore Job Error: The file is too recent to apply to the secondary database

So, I asked him to use the script from the following blog to find out if there are other backups running.

SQL SERVER – Get Database Backup History for a Single Database

Sure enough, SQLDBAs were not aware of DPM and they configured native SQL Server backup. Since DPM was already protecting the server, we disabled the job which was taking T-Log backups. After this issue was resolved and same error never came back.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – DPM Backup Failure – E_COORD_LOG_CHAIN_BROKEN – The Backup Failed Because of a Gap in The LSN the Log Files

Recently, I was playing with some settings in SQL Server Agent and encountered an interesting issue. In this blog we would learn about how to fix error Msg 22004 – The log file is not using Unicode format.

When I tried to open SQL Server Agent Log via SQL Server Management Studio, I was getting an error.

TITLE:.Net SqlClient Data Provider
——————————
The log file is not using Unicode format.

Here is how it looks like.

If we run sp_readerrorlog to read SQL Agent Logs, we get same error which is shown in the UI.

Interesting enough, if we open “SQLAGENT.OUT” file from Windows side, using notepad or any other text editor, we were able to open it and read it as well.

WORKAROUND/SOLUTION

When I tried to compare SQL Server Agent properties from my server, there was a difference in highlighted section (Write OEM file)

As per documentation on MSDN:

Write OEM error log: Writes the error log file as a non-Unicode file. This reduces the amount of disk space consumed by the log file. However, messages that include Unicode data may be more difficult to read when this option is enabled.

Now, how do I change it? It’s disabled in above screen! There is another place.

• Connect to Object Explorer to SSMS > Expand “SQL Server Agent” > Right Click on “Error Logs” and choose “Configure”
• Uncheck the box “Write OEM error log”

If you like T-SQL then you can run below T-SQL to fix it.

USE [msdb]
GO
EXEC msdb.dbo.sp_set_sqlagent_properties @oem_errorlog=0
GO

Once done, we need to restart SQL Agent service and then we should be able to read newly generated logs.

Have you encountered this issue? Please comment and let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Msg 22004 – The Log File is Not Using Unicode Format

One of my famous offerings is Comprehensive Database Performance Health Check where we always go over various SQL Server and database level properties. Lots of people do not know that there are quite a few SQL Server and database level settings which can help us improve our SQL Server performance many folds. Let us learn in this blog post how to fix the error about Databases Going to Recovery Pending State Randomly.

While doing this for one of my clients, they had an interesting issue. The informed that they noticed that sometimes, very randomly, one of their databases go to recovery pending state. As a usual troubleshooting step, I checked SQL Server ERRORLOG to see if there is anything interesting about the database. SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

Here is the filtered information for one of the database HRFINDB

2018-05-16 11:16:00.38 spid58 Starting up database ‘HRFINDB’.
2018-05-16 11:16:01.40 spid58 Error: 17204, Severity: 16, State: 1.
2018-05-16 11:16:01.40 spid58 FCB::Open failed: Could not open file T:\MSSQL12.MSSQLSERVER\MSSQL\DATA\HRFINDB_log.ldf for file number 2. OS error: 32(The process cannot access the file because it is being used by another process.).
2018-05-16 11:16:01.42 spid58 Error: 5105, Severity: 16, State: 1.
2018-05-16 11:16:01.42 spid58 A file activation error occurred. The physical file name ‘T:\MSSQL12.MSSQLSERVER\MSSQL\DATA\HRFINDB_log.ldf’ may be incorrect. Diagnose and correct additional errors, and retry the operation.
2018-05-16 11:16:01.45 spid58 Error: 5170, Severity: 16, State: 1.
2018-05-16 11:16:01.45 spid58 Cannot create file ‘T:\MSSQL12.MSSQLSERVER\MSSQL\DATA\HRFINDB_log.ldf’ because it already exists. Change the file path or the file name, and retry the operation.

I also found many starting up messages like below.

2018-05-14 15:27:03.80 spid61 Starting up database ‘HRFINDB’.
2018-05-14 15:27:08.23 spid54 Starting up database ‘HRFINDB’.
2018-05-14 15:28:20.02 spid52 Starting up database ‘HRFINDB’.
2018-05-14 15:29:46.88 spid54 Starting up database ‘HRFINDB’.
2018-05-14 15:29:48.20 spid54 Starting up database ‘HRFINDB’.

I immediately recalled message as I blogged about it long ago.

SQL SERVER – Starting Up Database – Why Multiple Times in Errorlog?

WORKAROUND/SOLUTION

The issue is a combination of Auto Close and Antivirus real-time scanning of the file. Read the link 1 and link 2.

So, have changed the Auto_Close to OFF for database ‘HRFINDB’ and we didn’t see same behavior again.  I also help them to configure antivirus settings as per above article.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Databases Going to Recovery Pending State Randomly

This was another interesting situation where client reported that their SQL Server Cluster resource, sometimes, fail to come online when they perform failover of SQL Server from one node to another. In this blog, we would learn how to fix ” SQL Server resource not coming to an online state ” issue.

You would agree with me that it is easy to fix something which is happening continuously. In this case, it was a random behavior. I have looked into ERRORLOG for “good” and “bad” scenario and found something interesting in the non-working situation.

2018-04-30 02:29:47.85 spid7s Error: 1807, Severity: 16, State: 3.
2018-04-30 02:29:47.85 spid7s Could not obtain exclusive lock on database ‘model’. Retry the operation later.
2018-04-30 02:29:47.86 spid7s Error: 1802, Severity: 16, State: 4.
2018-04-30 02:29:47.86 spid7s CREATE DATABASE failed. Some file names listed could not be created. Check related errors.
2018-04-30 02:29:47.86 spid7s Could not create tempdb. You may not have enough disk space available. Free additional disk space by deleting other files on the tempdb drive and then restart SQL Server. Check for additional errors in the event log that may indicate why the tempdb files could not be initialized.
2018-04-30 02:29:47.86 spid7s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.

From above it is clear that someone grabbed the connection to the model database before TempDB database could get created. If we have a connection to the model database, we can’t create a new database. Here are my few earlier blogs explaining the solution to the same error.

• SQL SERVER – FIX: Error: 1807 Could not obtain exclusive lock on database ‘model’. Retry the operation later.
• SQL SERVER – FIX: Error 1807 Could not obtain exclusive lock on database ‘model’. Retry the operation later – Part 2

But in this situation, we are not trying to create a database. Its SQL Server trying to create TempDB using the model database. My client said it is an intermittent issue. When I again bought SQL online, it was successful. I attempted failover/failback – successful. After around 20 tests, one of the tests failed again with the same behavior.

WORKAROUND/SOLUTION

After a lot of trial and errors, we concluded that most would be some application or automated service trying to connect to SQL continuously. But why would they go to the model database?  I was checking sys.syslogins and I saw “model” in dbname column and that was the big hint. We found that few accounts had a model database as their default database.

We changed it to master and observed many failovers, the issue never resurfaced.

Have you seen such issue earlier? Hopefully, some of my blog readers would be benefited with this blog. Please comment and let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – SQL Server Cluster Resource Doesn’t Come Online Randomly

This was one of an interesting issue reported by my client. They informed that after some hardening by the security team, they are not able to bring SQL resource online in the cluster. In this blog, we would learn about how to fix the issue where SQL cluster is not coming online.

In my client’s environment, there were two nodes (Node1 and Node2) They noticed that in “Failover Cluster Manager” and found that SQL resource was in “Online Pending” state.

After 3 minutes, it went to “failed” state. This happens when the cluster is not able to connect to SQL and showing in online pending even if service is running. Pending timeout in the cluster was set to 3 minutes. (default value is 3 minutes). Which means that we would see “online pending” for 3 minutes before goes to the failed state. After this SQL Service goes to stopped automatically.

I checked ERRORLOG and found below

2018-04-28 03:05:47.43 spid5s Recovery is complete. This is an informational message only. No user action is required.
2018-04-28 03:05:50.07 Logon Error: 18470, Severity: 14, State: 1.
2018-04-28 03:05:50.07 Logon Login failed for user ‘Domain\Node1$’. Reason: The account is disabled. [CLIENT: 128.xxx.xxx.xxx]

We also checked Cluster log and found the same error there as well.

SOLUTION/WORKAROUND

Now, we know that there is no issue with SQL startup, it is the SYSTEM account which is not able to connect to SQL Server to show the cluster resource online.

To fix the issue, we followed steps outlined below.

1. Started SQL via command line.

NET START MSSQLSEVER

2. Connected to SQL via SSMS
3. Found that “NT AUTHORITY\SYSTEM” account was disabled. We enabled the account by using below T-SQL.

ALTER LOGIN [NT AUTHORITY\SYSTEM] ENABLE

Note: you can also do it via SSMS by going to Login > Properties > Status tab (as shown below).

Enable the login and hit OK.

4. We stopped SQL via command line

NET STOP MSSQLSEVER

After following above steps, we attempted to bring SQL online and it came online. We also tested failover to another node and it worked like a charm.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Bring SQL Cluster Online – Reason: The Account is Disabled

One of my clients had an interesting situation where none of the SQL Maintenance plans were working. The example what I am using here is a SQL Server backup maintenance plan failing. I also saw a behavior where all the individual jobs were working fine as expected.

THE PROBLEM

Here is the error we were seeing in the job history

Description: Failed to acquire connection “Local server connection”. Connection may not be configured correctly or you may not have the right permissions on this connection.  End Error  Warning: 2018-01-21 15:48:36.32     Code: 0x80019002     Source: OnPreExecute     

(A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible.

From above it is clear that the maintenance plan was not able to connect to this instance of SQL Server. As we know, the Maintenance Plans uses SSIS engine to execute the plans. Every plan which gets created has a connection defined, which gets used during the plan execution. If the connection defined is not correct, the plan execution will fail with the above errors. I suspected something like this to be happening here.

So, I asked my client “the most famous question” in the troubleshooting world – What was changed recently?

and, they replied – “We changed the SQL Server port from default port 1433 to a non-default port 2433”

And I think that was a major clue.

WORKAROUND/SOLUTION

Based on the public documentation, all the jobs created via Maintenance Plans are executed via DTExec.exe which is an external 64-bit client process. This process tried to connect to the instance of SQL Server before executing the assigned step. Here is the place to see the connections.

As the SQL Port number was changed to a non-default port, either we need to put that in connection (as shown above) or create SQL Server client-side alias for this instance, locally, using the new port number.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Maintenance Plan Error: Failed to Acquire Connection “Local Server Connection”.

My client contacted me for a quick assistance in a setup trouble. She informed me that it’s a brand-new install and its failing at the end. She was installing SQL Server 2016 on Windows Server 2016. Let us learn about how to fix Installation Error.

Below was the error message which popped up during the installation.

Updating permission setting for file ‘H:\System Volume Information\ResumeKeyFilter.Store’ failed. The file permission setting was supposed to be set to ‘D: P(A;OICI;FA;;;BA)(A;OICI;FA;;;SY)(A;OICI;FA;;;CO)(A;OICI;FA;;;S-1-5-80-3880718306-3832830129-1677750213-2598158969-1042248003)’.
Click ‘Retry’ to retry the failed action, or click ‘Cancel’ to cancel this action and continue setup.

Your file name might vary but steps would remain same.

WORKAROUND / SOLUTION

We can fix above-mentioned error using steps. But you may want to note that this is not a clever idea to keep the data in the root of a drive. We should create a folder and keep database files inside the folder.

1. First to view this hidden folder (System Volume Information) click on the view tab and select Hidden Items and go to options tab click on the view tab and unselect Hide Protected Operating System File (Recommended).
   
2. Go to “G:\System Volume Information” folder and right click on file ResumeKeyFilter.Store click on properties and in security tab add your installation user. In my client’s case, it was a domain user called “Susan” so we added her account under permission (full control).

After the permission was given, we click on “Retry” option and setup completed without any further error. I hope this will help you to fix installation error.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Setup / Installation Error – Updating Permission Setting for File ResumeKeyFilter.Store Failed

In recent past, I was assisting a client in installing SQL Server clustered instance in a Windows Cluster. There were many errors encountered and I learned a lot from this experience. In this blog, we would learn about Microsoft Cluster Service (MSCS) cluster verification errors which might appear during installation or AddNode.

We got the below error and as we can see, it was a setup “rule” failure rather than an installation error.

Setup wizard failed with the rule — Microsoft Cluster Service (MSCS) cluster verification errors. Once we have this error, we would not be able to proceed next unless we fix it or follow KB to skip this rule.

From the Detail.txt we found the below error:

<DateTime>Slp: Rule evaluation message: The cluster either has not been verified or there are errors or failures in the verification report. Refer to KB953748 or SQL Server Books Online for more information.

Now, everyone might think we already have a KB Article 953748. It also has a workaround and almost all the client with whom I have interreacted, just use that method. But this client was interested to know why such error is occurring.

SOLUTION/WORKAROUND

From what I have seen, this error can occur due to one the following reasons,

• Cluster Validation was never run on the cluster
• Cluster Validation report contains errors
• Unable to access Cluster Validation report due to Admin$ share access blocked
• Cluster Validation report not present in the cluster nodes
• Unable to find the Cluster Validation report.

It is very important you understand why these errors have generated and try to resolve them first before going to the workaround step given in the above-mentioned article. Especially when you have found errors/warnings in the cluster validation report. If you have not run the report, I would highly recommend you run it before making any cluster-related installation.

Cluster Validation was never run on the cluster

Run Windows Cluster Validation using Failover Cluster Manager, read here.

Cluster Validation report contains errors

Fix the errors found in the report before you begin installing SQL Server.

Unable to access Cluster Validation report due to Admin$ share access is blocked

Unblock the Admin$ share before you begin installing SQL Server. Below is an example

Slp: Init rule target object: Microsoft.SqlServer.Configuration.Cluster.Rules.ClusterServiceFacet
Slp: Validation Report not found on Node1.DomainName.com
Slp: Validation Report not found on Node2.DomainName.com

I would write another blog in the same series as I found multiple reasons for such behavior. Stay tuned! Please comment and let me know if you found some more causes and ways to fix it.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Install Error: Microsoft Cluster Service (MSCS) Cluster Verification Errors – Part 1

In recent past, I was assisting a client in installing SQL Server clustered instance in a Windows Cluster. There were many errors encountered and I learned a lot from this experience. In this blog, we would learn about Microsoft Cluster Service (MSCS) cluster verification errors which might appear during installation or AddNode.

We got the below error and as we can see, it was a setup “rule” failure rather than an installation error.

Setup wizard failed with the rule — Microsoft Cluster Service (MSCS) cluster verification errors. Once we have this error, we would not be able to proceed next unless we fix it or follow KB to skip this rule.

From the Detail.txt we found the below error:

Slp: Initializing rule : Microsoft Cluster Service (MSCS) cluster verification errors
Slp: Rule applied features : ALL
Slp: Rule is will be executed : True
Slp: Init rule target object: Microsoft.SqlServer.Configuration.Cluster.Rules.ClusterServiceFacet
Slp: Validation Report not found on Node1.
Slp: Validation Report not found on Node2.DomainName.com.
Slp: Rule ‘Cluster_VerifyForErrors’ detection result: Is Cluster Online Results = True; Is Cluster Verfication complete = False; Verfication Has Warnings = False; Verification Has Errors = False; on Machine Node1
Slp: Evaluating rule : Cluster_VerifyForErrors
Slp: Rule running on machine: Node1
Slp: Rule evaluation done : Failed
Slp: Rule evaluation message: The cluster either has not been verified or there are errors or failures in the verification report. Refer to KB953748 or SQL Server Books Online for more information.

Now, everyone might think we already have a KB Article 953748. It also has a workaround and almost all the client with whom I have interreacted, just use that method. But this client was interested to know why such error is occurring.  We had discussed a few issues in Part 1 of this blog and you can find it here SQL SERVER – Install Error: Microsoft Cluster Service (MSCS) Cluster Verification Errors – Part 1.

In this blog, we will discuss another scenario which produces the same error, but the solution is different.

Cluster Validation report not present in the cluster nodes

If the respective file is not found in the given location, we would see it clearly mentioned in Detail.txt file as shown below:

<DateTime>Slp: Validation Report not found on Node1.DomainName.com
<DateTime>Slp: Validation Report not found on Node2.DomainName.com

If you use one of my favorite tool and a procmon trace and you will notice entries like below. The user might have deleted/moved the given file. The report is usually located under – C:\Windows\Cluster\Reports

Date & Time: <DateTime>
Event Class: File System
Operation: CreateFile
Result: NAME NOT FOUND
Path: \\Node1\admin$\Cluster\Reports\Validation Data For Node Set 83ADA46DDCA6617CB8C1C06A947DF511909FFD34.xml

Date & Time: <DateTime>
Event Class: File System
Operation: CreateFile
Result: NAME NOT FOUND
Path: \\Node2.DomainName.com\admin$\Cluster\Reports\Validation Data For Node Set 83ADA46DDCA6617CB8C1C06A947DF511909FFD34.xml

SOLUTION/WORKAROUND

If you have moved this file, please move it back to the location. Else re-run Windows Cluster validation to generate a new report and this new report will be used by the setup the next time when it runs.

I would write another blog in the same series as I found multiple reasons for such behavior. Stay tuned! Please comment and let me know if you found some more causes and ways to fix it.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Install Error: Microsoft Cluster Service (MSCS) Cluster Verification Errors – Part 2