Recently, while deploying a hybrid AlwaysOn availability group for a client, I faced this error. Since it was something I was not able to find many hits on internet search, I thought of sharing this via this blog. I am sure it would help others.

Topology

The client was trying to deploy hybrid cluster with both on-premise instances and instances hosted in Microsoft’s Azure cloud. All machines were domain-joined and it they were part of multi-subnet network connected via Express-route.

Error message

Here was the error message when they were trying to add a replica in Azure VM.

Msg 19456, Level 16, State 1, Line 3
None of the IP addresses configured for the availability group listener can be hosted by the server ‘AZURESQL-1’. Either configure a public cluster network on which one of the specified IP addresses can be hosted, or add another listener IP address which can be hosted on a public cluster network for this server.

Msg 41158, Level 16, State 3, Line 3
Failed to join local availability replica to availability, group ‘HR_AG’. The operation encountered SQL Server error 19456 and has been rolled back. Check the SQL Server error log for more details. When the cause of the error has been resolved, retry the ALTER AVAILABILITY GROUP JOIN command.

Workaround / Solution

I looked at the IP addresses under the network name in the “Failover Cluster Manager” and found that there were two IPs: one in 10.150.xx.xx and one 10.160.xx.xx range. We saw that the replica that we were attempting to add was in 10.140.xx.xx. So, we then added an IP address in the appropriate subnet as a dependency of the network name.

In short, this error can be resolved by adding a right IP to the listener. The IP address for all subnets cannot be an IP address already in use, i.e. The IP address of one of the nodes. After adding above, we again attempted to join the replica. And as expected, the operation succeeded.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Error 19456: None of the IP Addresses Configured for the Availability Group Listener can be Hosted by the Server

While installing Service Pack 3 for SQL Server, I was welcomed by a strange error about UNKNOWN\Components.

Line 2421: MSI (s) (C8:98) [22:31:05:835]: Product: Microsoft SQL Server System CLR Types (x64) – Update ‘Service Pack 3 for SQL Server System CLR Types (64-bit) (KB2979597)’ could not be installed. Error code 1603. Additional information is available in the log file C:\Program Files\Microsoft SQL Server\100\Setup Bootstrap\Log\20160105_195913\TFSDEV01\SQLSysClrTypes_Cpu64_1.log.

When I checked SQLSysClrTypes_Cpu64_1.log file, I found below error.

Line 2063: Error 1402. Could not open key: UNKNOWN\Components\0F02D6F45ECC29410AF18C3B9B1A1EE8\EDED107488810E94AB5E58875729C42A. System error 5. Verify that you have sufficient access to that key, or contact your support personnel.

Before attempting solutions, please take a backup of the registry. Microsoft’s built-in backup program allows for the ‘system state’ to be selectively backed up. You can also backup the Windows\System32\Config’ directory while taking backup.

SOLUTION

Step 1

Try KB https://support.microsoft.com/en-us/kb/313222

This is a long KB and take a lot of time to find what exactly we need to run.

SECEDIT /CONFIGURE /CFG %WINDIR%\INF\DEFLTBASE.INF /DB DEFLTBASE.SDB /VERBOSE /AREAS REGKEYS

If above doesn’t work, then go to step 2

Step 2

I have used the psexec tool to grant permission on the components folder for logged in account, after granting the permission SP3 installation completed successfully. Here are the steps:

1. Close registry editor, if opened.
2. Download and install psexec utility from the following link http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx
3. Open a command prompt using ‘run as administrator’
4. In the command prompt Navigate to the folder where PSEXEC is installed.
5. Execute the following from the command prompt.

psexec -i -s cmd.exe

6. The above command will open another command prompt running under ‘Local System’
7. On the new command prompt, open Registry Editor by typing “regedit” and hitting the enter key.
8. In registry editor, navigate to the following hive HKLM\Software\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\Components

9. Add the user account used to run the installation under permissions for the above hive and grant ‘Full control’ for that user.
10. Click on Advanced.

11. Under the Advanced security settings for Components choose the user and check the “Replace all existing inheritable permissions…………” check box as shown below and click OK

I could move forward by following step # 2.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX – Error 1402. Could not open key: UNKNOWN\Components. System error 5

In SQL Server, we can create linked server to many other RDBMS. The providers to connect would generally publish by destination RBMS. In this client scenario, they were dealing with linked server to Sybase. Let us learn about the error The OLE DB provider “ASEOLEDB” for linked server “SYBASESERVER” supplied inconsistent metadata for a column.

Here is the query which was tried by the client.

SELECT *
FROM SYBASESERVER.prodxc.dbo.sysobjects

It was failing with below error

Msg 7356, Level 16, State 1, Line 1
The OLE DB provider “ASEOLEDB” for linked server “ALTAIR” supplied inconsistent metadata for a column. The column “versionts” (compile-time ordinal 20) of object “prodxc.dbo.sysobjects” was reported to have a “DBCOLUMNFLAGS_ISFIXEDLENGTH” of 16 at compile time and 0 at run time.

Linked Server setup was as below:

• Destination Server: Sybase Server:  SYBASESERVER:5000
• Provider: SAP ASE OLE DB Provider

To troubleshoot, I asked them to execute the above query by changing the table name with some other tables within the system catalog.  Interestingly, we were able to successfully execute those queries with no issues.  So, the issue seems to be related to the specific table sysobjects.

Instead of four-part naming, we asked to try below OPENQUERY format of the linked server query.

SELECT *
FROM OPENQUERY([SYBASESERVER], 'SELECT * FROM PRODXC.DBO.SYSOBJECTS')

Later we found that It appears that the Sybase OLE DB MDA provider on the Sybase server were not up to date.

WORKAROUND/SOLUTION

1. Instead of four-part naming, use OPENQUERY
2. Make sure that drivers on the client and server are up-to-date.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Msg 7356, Level 16 – The OLE DB provider “ASEOLEDB” for linked server “SYBASESERVER” supplied inconsistent metadata for a column

One of my blog readers pinged me on Skype and asked a simple question. I always thought I had a blog written about this error, but I was wrong. So, this blog is the outcome of a short interaction with my blog reader. Let us learn about how to fix blocked access error.

Hi Pinal,
I am getting below error in SQL.

Msg 15281, Level 16, State 1, Procedure xp_cmdshell
SQL Server blocked access to procedure ‘sys.xp_cmdshell’ of component ‘xp_cmdshell’ because this component is turned off as part of the security configuration for this server. A system administrator can enable the use of ‘xp_cmdshell’ by using sp_configure. For more information about enabling ‘xp_cmdshell’, search for ‘xp_cmdshell’ in SQL Server Books Online.

Here is the command I am running.

xp_cmdshell 'dir c:\Windows'

We can easily reproduce the error if the setting is OFF.

As we can see that the error message is useful and tells what exactly needs to be done. But for someone who is new to SQL Server, above may not be enough.

SOLUTION / WORKAROUND

Here is the query we need to run in SQL Server Management Studio.

-- We need to have this ON because xp_cmdshell is an advanced option.
EXEC sp_configure 'show advanced options', 1
GO
-- To update the currently configured values for sp_configure
RECONFIGURE WITH OVERRIDE
GO
-- Now, enable the feature.
EXEC sp_configure 'xp_cmdshell', 1
GO
-- To update the currently configured values for sp_configure
RECONFIGURE WITH OVERRIDE
GO

After completing above query, we should be able to use xp_cmdshell without above error.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Msg 15281- SQL Server Blocked Access to Procedure ‘sys.xp_cmdshell’ of Component ‘xp_cmdshell’

There might be various reasons for above error, but this case is very specific to a situation where SQL Server was installed on a machine which was in Workgroup whereas the backup destination was the server located in the domain.  Here is the error message seen in SQL Agent job history. Let us learn about how to fix Operating system error 1326.

Executed as user: NT Service\SQLSERVERAGENT. Microsoft (R) SQL Server Execute Package Utility Version 11.0.2100.60 for 64-bit Copyright (C) Microsoft Corporation. All rights reserved.
Started: 12:23:24 PM Progress: 2016-12-27 12:23:25.00 Source: {229AB1E3-E7E5-4185-B305-BF0F7657122A}
Executing query “DECLARE @Guid UNIQUEIDENTIFIER EXECUTE msdb..sp…”.: 100% complete End Progress Error: 2016-12-27 12:23:25.35 Code: 0xC002F210 Source: Back Up Database Task Execute SQL Task
Description: Executing the query “BACKUP DATABASE [master] TO DISK = N’\\192.168.3….” failed with the following error: “Cannot open backup device ‘\\192.168.3.55\backupShare\master_backup_2016_12_27_122325_2407734.bak’. Operating system error 1326(Logon failure: unknown user name or bad password.). BACKUP DATABASE is terminating abnormally.”. Possible failure reasons: Problems with the query, “ResultSet” property not set correctly, parameters not set correctly, or connection not established correctly. End Error

Generally, when the backup is taken SQL Agent Service account is used to take backup. In this case since SQL was on Workgroup, it was NT Service\SQLSERVERAGENT. This account is not a valid account to validate for a domain.

VERIFICAITON

First, let’s verify that we are hitting the same issue which I experienced. From Workgroup machine, we would try to query remote share using xp_cmdshell. Here is the command.

xp_cmdshell 'dir \\192.168.3.55\backupShare'

This command should fail with below error.

Logon failure: unknown user name or bad password.

SOLUTION

Since we are talking about authentication issue. So, we need to make sure that we map a drive to SQL Server (not windows). As per SQL Server Books-On-Line:

“For a network share to be visible to SQL Server, the share must be mapped as a network drive in the session in which SQL Server is running”

Here is the command to map the drive.

EXEC XP_CMDSHELL 'net use Z: \\192.168.3.55\backupShare password@123 /User:domain\user'

Once the above is completely successful, we should be able to see content on the Z drive by running below command.

EXEC XP_CMDSHELL 'dir Z:\'

After this we should be able to take backup on the Z drive as its mapped within SQL Server as a drive.

Backup database master to disk = 'Z:\master.bak'

The above command should work and create the backup file in \\192.168.3.55\backupShare

Have you encountered this error earlier? Did you find this solution interesting? Please comment and let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Cannot open backup device. Operating system error 1326 (Logon failure: unknown user name or bad password.)

I was consulting a client for migration from SQL 2014 to SQL 2016. They were using methodology discussed in books online here. https://msdn.microsoft.com/en-us/library/dn178483.aspx  (Upgrading AlwaysOn Availability Group Replica Instances). Let us learn about how to fix error 946.

While doing that, they noticed that once they added SQL Server 2016 as secondary replica everything was looking great, but availability database on SQL Server 2016 were stuck in Synchronized / In Recovery in SSMS.

I have asked them to check SQL Server ERRORLOG on the secondary replica (SQL Server 2016) and found below the message.

Error: 946, Severity: 14, State: 1.
Cannot open database ‘ProdDB’ version 782. Upgrade the database to the latest version.

I asked to check sys.sysdatabases using below

SELECT name, version
FROM sys.sysdatabases

Since databases are part of availability group and have not been recoveredvered yet, it’s expected to see that version and state as “In recovery”. Once we have failed over the databases to secondary replica and made it as primary everything was looking good and database on SQL 2012 were shown as Not Synchronizing because SQL Server doesn’t allow database secondary replicaplica to be of lower version.

One thing which I noticed is that once I upgraded old primary to SQL 2016 then also databases were not getting synchronized. To fix that I must run below command on current secondary replica which is now upgraded to SQL 2016.

ALTER DATABASE [SQLAuthority] SET HADR RESUME;

Have you ever done upgrade of SQL in AlwaysOn availability group? How was your experience?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Fix: Error 946, Severity: 14 – Cannot open database ‘DB’ version 782. Upgrade the database to the latest version

While playing with my SQL Server startup parameter, I made some mistakes and came to a situation where SQL Server was not getting started. In this blog post, let us learn about error 17113.

Whenever you have a situation where SQL is not getting started, what you would normally do?

1. Look at ERRORLOG, if generated.
2. Look at System Event Log.
3. Look at Application Event log.

In my case, I was lucky enough that the SQL ERRORLOG file was getting generated. SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

Here is what I have in Errorlog.

2016-12-28 01:20:42.27 Server Registry startup parameters:
-m C:\Program Files\Microsoft SQL Server\MSSQL13.SOFTWARE\MSSQL\DATA\master.mdf
-e C:\Program Files\Microsoft SQL Server\MSSQL13.SOFTWARE\MSSQL\Log\ERRORLOG
-l C:\Program Files\Microsoft SQL Server\MSSQL13.SOFTWARE\MSSQL\DATA\mastlog.ldf
2016-12-28 01:20:42.27 Server Command Line Startup Parameters:
-s “SOFTWARE”
2016-12-28 01:20:42.28 Server Error: 17113, Severity: 16, State: 1.
2016-12-28 01:20:42.28 Server Error 3(The system cannot find the path specified.) occurred while opening file ‘master.mdf’ to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.
2016-12-28 01:20:42.28 Server SQL Server shutdown has been initiated

Do you find something interesting? If you are thinking that master.mdf file is not present in C:\Program Files\Microsoft SQL Server\MSSQL13.SOFTWARE\MSSQL\DATA\ then you are incorrect.

If the file was not present, here is how the ERRORLOG would look like.

2016-12-28 01:26:49.12 Server Registry startup parameters:
-d C:\Program Files\Microsoft SQL Server\MSSQL13.SOFTWARE\MSSQL\DATA\master.mdf
-e C:\Program Files\Microsoft SQL Server\MSSQL13.SOFTWARE\MSSQL\Log\ERRORLOG
-l C:\Program Files\Microsoft SQL Server\MSSQL13.SOFTWARE\MSSQL\DATA\mastlog.ldf
2016-12-28 01:26:49.12 Server Command Line Startup Parameters:
-s “SOFTWARE”
2016-12-28 01:26:49.13 Server Error: 17113, Severity: 16, State: 1.
2016-12-28 01:26:49.13 Server Error 2(error not found) occurred while opening file ‘C:\Program Files\Microsoft SQL Server\MSSQL13.SOFTWARE\MSSQL\DATA\master.mdf’ to obtain configuration information at startup. An invalid startup option might have caused the error. Verify your startup options, and correct or remove them if necessary.

Do you see the difference?

SOLUTION

In my case, the file was present in the location, but the parameter name was wrong. SQL Server uses -d for the master database mdf file. As I hinted earlier, I changed the parameter and now if you notice again the parameter is -m which is not a valid parameter for the master database data file. We can modify the startup parameters via configuration manager. Step is already listed here.

Hope you have learned something new.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Error 17113, Severity: 16 – Error Occurred While Opening File ‘master.mdf’ to Obtain Configuration Information at Startup

It is very common for people to move databases now a days to the cloud. The very first question I often hear from them is – How to Improve Application Performance on Cloud While Reducing Bandwidth Cost?

However, once the data is moved to any cloud there are two major challenges I see.

Challenge 1: Network Latency / Congestion and Slow Performance

Challenge 2: Network Bandwidth Congestion and High Cost

Every time when I go to performance tuning consultation for cloud hosted database, I hear about slow performance and high cost. The matter of the fact, it is so common that when I know the database is hosted on a cloud, I tell my customer that I already know their problem a) Slow Performance due to network latency and b) High cost due to high consumption of the data bandwidth.

If you have a very small database, you may not face issues with the high cost of bandwidth, but if you are running enterprise grade application where you move quite a lot of data from one server to another server, I am very confident that you are facing above two issues.

For example, Microsoft charges egress or data transfer out cost per 250MB around USD 21.32 and Amazon charges around USD 23.27. Now look at your database and find the size of the largest table, next think if you want your table to be exported to our in-premises system due to any reason, how much will it cost if you try to transfer out that much data. Trust me, in no time, you can almost cross the limit of your credit card. It is indeed a big big challenge and a REAL ONE!

Smart Solution

Though the challenge is quite big the solution is very simple. Actually the solution requires just a couple of clicks of the button and four minutes of your time.

I have installed NitroAccelerator in my cloud machine and local machine. Once I installed NitroAccelerator, I did two different tests of transferring around 25,000 rows of the data. When I saw the results, I was indeed very much surprised and happy.

Here is the less than 4 minutes long video where I narrate the entire story.

You can see that when I had turned ON NitroAccelerator, the query time was reduced 1/4 from the original time by giving me around 400% percentage of performance improvements and data size was reduced from 2267 KB to just 31 KB, that is around 98% of the compression of the data.  These tests do not include using the NitroAccelerator caching features for applications with redundant queries or using multiple active result sets (MARS) where you would get even more performance and cost saving benefits.

Here is my question to you – answer me honestly.

Do you want to gain 4 times more performance by reducing your cost by 98%?

If the answer is YES. Watch the above video and download NitroAccelerator.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Improve Application Performance on Cloud While Reducing Bandwidth Cost

A very old client of mine yesterday pinged me with this question about error related to QUOTED_IDENTIFIER.

“Pinal,

When I run my query it works just fine in SSMS but when I run it via SQL Server Agent Job, it gives me following error. Can you give me solution?

Here is the error he was facing:

Msg 1934, Level 16, State 1
UPDATE Failed Because the Following SET Options have Incorrect Settings: ‘QUOTED_IDENTIFIER’.

Solution / Workaround

The problem he was facing was very simple to fix it. He was running an update and which was working just fine in SSMS, because the SET options of the SSMS windows were different than the one for the query he was running.

Here is the solution of this simple error:

SET QUOTED_IDENTIFIER ON
GO
-- Write Your Query

When you write turn on the settings for the quoted identifier it will automatically remove the error for you. I hope this simple fix will help you.

If you want to read more about what is quoted identifier, you can read my following blog where I have explained the same issue in the detail.

• SQL SERVER – QUOTED_IDENTIFIER ON/OFF Explanation and Example – Question on Real World Usage
• SQL SERVER – QUOTED_IDENTIFIER ON/OFF and ANSI_NULL ON/OFF Explanation

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Fix: Error: 1934, Level 16, INSERT or UPDATE Failed Because the Following SET Options have Incorrect Settings: ‘QUOTED_IDENTIFIER’.

While trying to start SQL Service for a named instance, I got an error message related to event id 26 – Your SQL Server Installation is Either Corrupt or has Been Tampered With (Error Getting Instance Name).

Here is the text of the error which is related to event id 26.

Windows could not start the SQL Server (SOFTWARE) service on Local Computer. Error 1067: The process terminated unexpectedly.

When I checked event log I found following error messages:

Error Messages 1: The SQL Server (SOFTWARE) service terminated unexpectedly.  It has done this 4 time(s).

Error Message 2: Application popup: SQL Server: Your SQL Server installation is either corrupt or has been tampered with (Error getting instance name.).   Please uninstall then re-run setup to correct this problem.

The error gives hints about the issue, but I was not very sure what I should do. I search on the internet and found that we get Error getting the instance name when mapping between instance ID and name is incorrect. So, I looked into registry key

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\Instance Names\SQL

As we can see above, the SOFTWARE is mapped to MSSQL13.SOFTWARE. so I looked for this key

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL13.SOFTWARE

Interestingly, I found that someone renamed MSSQL13.SOFTWARE to MSSQL13.SOFTWARE1 which was causing the issue.

As soon I renamed the key back, the issue was resolved.

Another way to find a cause of such errors would be to start SQLServr.exe via command prompt and you might get a more meaningful error.

I don’t know who renamed the key and that is still a mystery.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Event ID 26 – Your SQL Server Installation is Either Corrupt or has Been Tampered With (Error Getting Instance Name)

One of my clients contacted me and informed that after patching of SQL Server, they noticed that their maintenance plan were failing. I asked them to share the complete error message about the Maintenance Plan failing with 0x80131904.

End Warning Error: 2017-01-09 17:22:27.10 Code: 0xC0024104 Source: Back Up Database Task Description: The Execute method on the task returned error code 0x80131904 (A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 – Error Locating Server/Instance Specified)). The Execute method must succeed, and indicate the result using an “out” parameter. End Error

I have removed some part of a message which is not relevant. In above long error information, interesting piece is “error: 26 – Error Locating Server/Instance Specified”.

Whenever we create a Maintenance Plan, SQL Server Management Studio always takes the name of the connection manager as “Local server connection” by default.

The important thing to note is, it uses the server name that we have used in SSMS to connect to the instance and puts this exact name in the server name property of the connection manager “Local server connection”

Does this give some hint?

SOLUTION / WORKAROUND

It is quite possible that the port of SQL might have been changed. When we connected earlier, when maintenance plan was created. We need to check on what is the server name, value in the connection manager “Local server connection” mentioned. To see that we can click “Manage Connections” after we click modify the Maintenance Plan.

Now, we can use the SQL Server Configuration manager and create an alias with the exact same name, provide the TCP and manual port number. Once we created alias, maintenance plan ran without an error?

Hopefully you know how to create TCP aliases in SQL Server Configuration Manager. If not, refer to books online documentation Create or Delete a Server Alias for Use by a Client (SQL Server Configuration Manager)

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Maintenance Plan failing with 0x80131904 – A network-related or instance-specific error occurred while establishing a connection to SQL Server.

There is a growing demand for database encryption nowadays and large companies offer alternative solutions for Microsoft TDE. But what if you want to protect the database from local Administrator?

One of solutions called DbDefence that I found recently offers interesting features that are not available in other products. DbDefence goes far beyond simple file encryption: it can hide database structure and data from local DBA. It works on all editions of SQL Server and can potentially be an alternative for TDE. That’s an interesting feature and I would like to check it and see if it is really does what it says.

I know my readers like technical examples.

Installation was simple and fast. After encrypting some test database I logged as sa and tried to access system catalog and here is what I got:

The same error I got when queried other tables. Here is what I saw when tried to open the database from SSMS:

Database files were indeed encrypted. Vendor claims files are encrypted with AES-128.

As you see it is not only encrypted, but also protected from privileged user. How would the owner access encrypted database? You need to add your application to the list of approved applications and provide database name and the password:

After the application is being added to the list it may access the database absolutely transparently, but only from a computer where it was configured. All other client applications from different computers will be rejected. An authorized application can’t be substituted with another application. Digital signature of authorized application is checked before granting access. There are different options to configure access, but I haven’t checked them all.

Even more, there is nothing in SQL Profiler related to protected database. Not a single statement visible!

There are many other features that I have not tested:

• Encrypted backups
• API for .NET and native apps. Using this API you can manage encryption from your application or command line.
• Encryption with FIPS 140-2 validated module
• Encryption with certificate.

If you are a software vendor and would like to protect your database from local administrator I think you might be interested.

The software is free for SQL Server 2005 and 2008. For newer SQL Server versions prices start from $698 per server.

Download link: https://www.database-encryption.com/

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – How to Protect Database from DBA and Hide Database Schema

In this blog post we will discover how to fix an Add Failover Cluster Node Fails With Error. If you have installed SQL Server cluster, then it would be easy for you to remember that it’s a two-step process.

1. InstallFailoverCluster
2. AddNode

My client completed step 1 successfully, but while adding a second node, he was getting below error in Detail.txt.

(13) 2017-01-08 14:33:01 Slp: Executing rules engine…
(13) 2017-01-08 14:33:01 Slp: Start rule execution, total number of rules loaded: 18
(13) 2017-01-08 14:33:01 Slp: Initializing rule : Number of cluster nodes supported for edition
(13) 2017-01-08 14:33:01 Slp: Rule is will be executed : True
(13) 2017-01-08 14:33:01 Slp: Init rule target object: Microsoft.SqlServer.Configuration.SetupExtension.NumberOfNodesFacet
(13) 2017-01-08 14:33:01 Slp: Rule ‘Cluster_NumberOfNodes’ edition Invalid allows 0 cluster nodes.
(13) 2017-01-08 14:33:01 Slp: Rule ‘Cluster_NumberOfNodes’ detected 1 cluster nodes.
(13) 2017-01-08 14:33:01 Slp: Evaluating rule : Cluster_NumberOfNodes
(13) 2017-01-08 14:33:01 Slp: Rule running on machine: SQLNODE02
(13) 2017-01-08 14:33:01 Slp: Rule evaluation done : Failed
(13) 2017-01-08 14:33:01 Slp: Rule evaluation message: This SQL Server edition does not support the installed number of cluster nodes. To continue, remove nodes and then complete cluster installation.

Initially, my thoughts, it’s because of a standard edition. SQL SERVER – Add failover cluster node fails with “number of cluster nodes supported for edition”

So, I asked to check SELECT SERVERPROPERTY (‘Edition’) and it was the enterprise! We should note that they were using Enterprise edition, which doesn’t have the limitation of number of nodes in the cluster. The error was very strange but when I looked at line by line, I found something interesting as below.

(13) 2017-01-08 14:33:01 Slp: Rule ‘Cluster_NumberOfNodes’ edition Invalid allows 0 cluster nodes.

From above it looks SQL setup is not able to get edition and it’s marked as “Invalid”. I checked further and found below message as well.

(04) 2017-01-08 14:33:01 Slp: Loading rule: AddNodeEditionBlock
(04) 2017-01-08 14:33:01 Slp: Creating rule target object: Microsoft.SqlServer.Configuration.SetupExtension.AddNodeEditionBlock
(04) 2017-01-08 14:33:01 Slp: Rule applied features : ALL
(04) 2017-01-08 14:33:01 Slp: ———————————————————————-
(04) 2017-01-08 14:33:01 Slp: Skipping rule AddNodeEditionBlock
(04) 2017-01-08 14:33:01 Slp: Rule will not be evaluated due to the following failed restriction(s):
(04) 2017-01-08 14:33:01 Slp: Condition “Is requested input setting is set to PID” did not pass as it returned false and true was expected.
Returning false as an unhandled exception was caught:
Microsoft.SqlServer.Chainer.Infrastructure.ChainerInvalidOperationException: The input ‘PID’ requested by the StringInputSettingExistsCondition is not of string type.

Based on my search on the internet, it looks like sqlboot.dll is used to get version using checksum value. The DLL is located in the path “SharedCode” stored in “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\120” and “checksum” is located in “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL12.<InstanceName>\Setup”

In my client’s case the file was missing from C:\Program Files\Microsoft SQL Server\120\Shared I am not sure how that happened. They did tell me that setup was incomplete on Node1 and they did manual hack to fix that.

Have you ever seen such weird error? Thanks to the internet to provide internals.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Add Failover Cluster Node Fails With Error – This SQL Server Edition Does Not Support the Installed Number of Cluster Nodes

This is one of the common question which is asked via emails to me. “How to apply the patch in AlwaysOn Availability Group configuration?” OR “What are the steps we should do and things to take care while patching availability replica?”

There might be many articles which would provide same details, but I want to make it short and crisp.

I am going to pick step by step for an Availability Group with one secondary replica.

1. Make sure that we have taken good recent OS backup with system state (or VMware snapshot with SQL services stopped), a good recent backup of all databases and a successful completion of a checkdb on the primary node. {This is not mandatory, but to avoid “Ouch” moment}
2. From the node acting as the primary replica (SQL1), change the failover mode to manual
3. Refresh the affected databases on the secondary replica (SQL2) and make sure that everything is green on the dashboard.
4. Apply the patch (service pack of CU) on SQL2.
5. Repeat the Windows Update and/or software updates until all available patches are applied. Do not move on with the patching steps until all patches and post patch reboot and configuration tasks are completed.
6. Double check that patches have been applied, the cluster is healthy and AlwaysOn Availability Groups are functional.
7. Make sure that synchronization state is SYNCHRONIZED.
8. Fail over the availability group to the secondary replica (SQL2).
9. Refresh the affected databases on secondary Replica (former primary = SQL1) until the synchronization state is synchronized.
10. Apply the patch (service pack of CU) on SQL1.
11. Repeat the Windows Update and/or software updates until all available patches are applied. Do not move on with the patching steps until all patches and post patch reboot and configuration tasks are completed.
12. Double check that patches have been applied, the cluster is healthy and AlwaysOn Availability Groups are functional.
13. Make sure that synchronization state is SYNCHRONIZED.
14. Fail over the availability group to the primary node (back to SQL1).
15. Change the failover mode to Automatic now (which we changed in Step b)

In case things do not go as planned, you have followed step a) so you know what needs to be done.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – How to Apply Patch in AlwaysOn Availability Group Configuration?

Today’s blog is originated from a blog interaction. After hardening a server, my friend was not able to start SQL Browser service. When we try to start from services, we get below standard error, which is not very helpful. Let us fix the error related to SQL Configuration.

The SQL Server Browser service on Local Computer started and then stopped. Some services stop automatically if they are not in use by other services or programs.

Then we looked into Event Logs and found below messages in application event log.

The SQL Server Browser is closing. There are no instances of SQL Server or SQL Server Analysis Services.

and below…

The SQL configuration for SQL is inaccessible or invalid.

Initially we thought that our SQL configuration is corrupted so we thought that we need to uninstall SQL Server. But NO, it was something else.

WORKAROUND/SOLUTION

We started SQLBrowser from command line and found error there also. Later, we found below registry key which is needed to start the SQL Browser.

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Microsoft SQL Server\90\SQL Browser

In our case, due to hardening, the key was deleted, so we created key and added two values under that.

They keys are AnalysisServiceListener and SsrpListener and value should be 1.

Have you ever seen issues due to server level hardening by Windows team?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – SQL Browser not Starting. Error – The SQL Configuration for SQL is Inaccessible or Invalid

In this blog post we are going to learn about Installation Error – The Specified Driver is Invalid.

SYMPTOMS

SQL Server installation failed with errors in setup logs. Summary.txt shows below

Troubleshooting information for those features:
Next step for SQLEngine: Use the following information to resolve the error, uninstall this feature, and then run the setup process again.
Next step for FullText: Use the following information to resolve the error, uninstall this feature, and then run the setup process again.
Next step for Replication: Use the following information to resolve the error, uninstall this feature, and then run the setup process again.

Above means that we need to:

1. Find cause and resolved the error.
2. Uninstall SQL Server.
3. Run setup process again to install SQL Server.

Fine, now let’s look further down to know what was the cause of those three components install failure.

Feature: Database Engine Services
Status: Failed: see logs for details
Reason for failure: An error occurred during the setup process of the feature.
Next Step: Use the following information to resolve the error, uninstall this feature, and then run the setup process again.
Component name: SQL Server Database Engine Services Instance Features
Component error code: 0x84BB0001
Error description: The specified driver is invalid.

This tells us that the error is “The specified driver is invalid.” I found below KB

https://support.microsoft.com/en-us/kb/933945 (Issues that you may encounter when you install a named instance of one or more SQL Server 2005 components and you specify different installation locations for one or more of the components)

But above is not applicable because we are install SQL 2012 not SQL 2005. As per books online documentation, I looked further into Detail.txt

lodctr: ‘E:\Program Files\Microsoft SQL Server\MSSQL11.SQLCAP\MSSQL\Binn\perf-MSSQL$SQLCAPsqlctr.ini’

(01) 2017-01-11 11:06:49 Slp: PerfCounter calling lodctr: ‘E:\Program Files\Microsoft SQL Server\MSSQL11.SQLCAP\MSSQL\Binn\perf-MSSQL$SQLCAPsqlctr.ini’

This failed with exact same error.When I opened below files, they were having some unreadable characters.

• E:\Program Files\Microsoft SQL Server\MSSQL11.SQLCAP\MSSQL\Binn\sqlctr.ini’

• E:\Program Files\Microsoft SQL Server\MSSQL11.SQLCAP\MSSQL\Binn\perf-MSSQL$SQLCAPsqlctr.ini’

When I checked from my laptop’s SQL instance, it was clearly readable. The path would vary based on instance name.

WORKAROUND/SOLUTION

We tried using the same media on another machine and it failed with the same error. I asked them to re-download the media as it looks like setup media is corrupted. After re-downloading media and we could install SQL Server on this machine.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Installation Error – The Specified Driver is Invalid

There are many wait types in SQL Server and not all of them are documented. Long ago I wrote blog on PREEMPTIVE and Non-PREEMPTIVE waits. SQL SERVER – PREEMPTIVE and Non-PREEMPTIVE – Wait Type – Day 19 of 28. In this blog post we will take a look at how to solve wait PREEMPTIVE_OLEDB_RELEASE.

Once I was contacted when there was wait of PREEMPTIVE_OLEDB_RELEASE seen in SQL Server. Here are the few columns from sys.dm_exec_requests.

There was no documentation available for this wait type. From the name, it looks like some kind of cleanup (release) task.  Here was the behavior which I saw.

1. All above were in RUNNING state.
2. ONLY wait duration and elapsed time column values were increasing.
3. There was a network outage earlier, which caused few SPID to run continuously so they were killed and now sitting in KILLED/ROLLBACK state. They were running UPDATE statements earlier.

WORKAROUND/ SOLUTION

As on the internet, there was no other help on this subject and there is no further documentation available from Microsoft as well. The customer was really keen on removing this wait stats. The good thing for us that their maintenance window was coming up on the very same day.

I explained to my client that they should restart the SQL instance to remove those sessions. And as expected, restarted to clear those SPIDs.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Queries are in KILLED/ROLLBACK State with Wait PREEMPTIVE_OLEDB_RELEASE

Once, while doing a load testing with my client, I was shown below error while taking log backups on secondary replica.

Msg 35295, Level 16, State 1, Line 1
Log backup for database “MyAppDB” on a secondary replica failed because the last backup LSN (0x0016ff11:00032258:0001) from the primary database is greater than the current local redo LSN (0x0016ffec:0006ee28:0001). No log records need to be backed up at this time. Retry the log-backup operation later.
Msg 3013, Level 16, State 1, Line 1
BACKUP LOG is terminating abnormally.

This error was interesting because as per documentation we can take backup on the secondary replica in AlwaysOn configuration. I quickly checked AlwaysOn Dashboard from Management studio, all were looking green and databases were synchronized. Then I checked transaction log file (LDF) size and it was very big which made me think in different directions. Went to dashboard again and added Redo queue size. We found that REDO thread was blocked by a user query. As per Sys.dm_exec_requests we found that SPID 28 (which was showing DB STARTUP) was blocked by a user SPID 255. Then we used conventional way and found that SPID 255 was running SELECT INTO query for many hours. This was from a job which had a complex join with huge tables.

WORKAROUND/SOLUTION

If you find the same error while taking log backups on secondary replica, then check REDO THREAD and check if its blocked. Based on criticality you need to take a decision to KILL the SPID which is blocking system SPID.

As soon as we killed user query, the “Recovery Queue” for that database started coming down. Once it came down to zero, we could take a transaction log backup on the secondary.

Have you seen REDO blocked in production environment? Do you monitor this by monitoring tool?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX : Msg 35295, Level 16 – Log Backup For Database ‘DBName’ on a Secondary Replica Failed

One should be very careful while working with system objects in SQL Server. Below is one of the situation where some permissions were given on system objects which was causing SQL startup failure. I have written various blog on script upgrade mode of SQL Server.

Whenever a patch is applied to SQL Server, one of the steps after first start is to run upgrade scripts. Those  scripts make changes to system objects. If there is any fatal error in running those scripts, SQL Server would not be able to start.

Below is the ERRORLOG snip from one of the failures.

2017-01-30 17:18:22.44 spid3s Error: 15173, Severity: 16, State: 1.
2017-01-30 17:18:22.44 spid3s Server principal ‘##MS_PolicyEventProcessingLogin##’ has granted one or more permission(s). Revoke the permission(s) before dropping the server principal.
2017-01-30 17:18:22.44 spid3s Error: 912, Severity: 21, State: 2.
2017-01-30 17:18:22.44 spid3s Script level upgrade for database ‘master’ failed because upgrade step ‘msdb110_upgrade.sql’ encountered error 15173, state 1, severity 16. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
2017-01-30 17:18:22.44 spid3s Error: 3417, Severity: 21, State: 3.
2017-01-30 17:18:22.44 spid3s Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.

The last message is very scary! We need to look back and look at meaningful error which is 15173 in this case. We should note that ##MS_PolicyEventProcessingLogin## is a certificate-based SQL Server login for internal system use only. It’s created from certificates when SQL Server is installed.

WORKAROUND/SOLUTION

In script failure, we always need to start SQL with trace flag 902, which would cause SQL to skip running those scripts and we get a chance to fix the issue. Here are the steps we followed to start SQL with trace flag.

• Open SQL Server Configuration Manager
• Select the SQL server instance in SQL Server Services,
• Right-click the instance, and then click Properties.
• Click the Startup Parameter tab.
• Add “-T902”

Then we started the SQL Server service. Since we have issue with ##MS_PolicyEventProcessingLogin##

SELECT * FROM   sys.server_permissions
WHERE grantor_principal_id = (SELECT principal_id
                               FROM sys.server_principals
                               WHERE NAME = N'##MS_PolicyEventProcessingLogin##')

Once we ran above query, we found that there was a login having permissions on this login and due to that SQL setup was unable to drop it.

Once we cleared dependency (using the REVOKE command), we were able to remove trace flag and start SQL without any error.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Script Upgrade – Server Principal ‘MS_PolicyEventProcessingLogin’ Has Granted One or More Permission(s)

In this blog post we are going to learn about PRINTing Status Using RAISERROR With NOWAIT Option.

The Problem 

There are many situations where you want to know which line of code is completed in the batch. Mostly this is a series of commands in SSMS and we want to know where it has reached so far.

I have seen many using PRINT statement after each statement. But the problem with PRINT statement is that it is not to SSMS unless output buffer is full. I think the buffer size is around 8 KB.

Here is the example of what I mean. You can copy below code in SQL Server Management Studio and run it in text mode to see that output.

PRINT 'SQLAuthority'+ REPLICATE(' ',8000)
PRINT 'Pinal'+ REPLICATE(' ',116)
WAITFOR DELAY '00:00:05'
PRINT 'Final Message'

When we run the code, we can notice that SQLAuthority is printed as soon as we execute the whole code. The Print, which is there is next line, is not printed immediately. The output can only be seen after 5 seconds, when WAIT is complete. I have selected 116 because if I use 115 then both would not be printed because buffer 8K is not full.

Now imagine the same piece of code within a batch where we are doing an insert in tables and we want to know where exactly the code is executing. In that case, we can’t rely on print statement.

The Solution

Here is one of the solution of the above problem.  We can use RAISERROR.

DECLARE @msg1 varchar(max) = 'SQLAuthority'+ REPLICATE(' ',8000)
DECLARE @msg2 varchar(max) = 'Pinal'+ REPLICATE(' ',116)
DECLARE @msg3 varchar(max) ='Final Message'

RAISERROR (@msg1, 0, 1) WITH NOWAIT
RAISERROR (@msg2, 0, 1) WITH NOWAIT
WAITFOR DELAY '00:00:05'
RAISERROR (@msg3, 0, 1) WITH NOWAIT

If we run above code, we can see that first two are printed immediately and then delay of 5 seconds and finally the last message is printed.

As we can see that the RAISERROR method is a much more sophisticated of returning status messages to the client instead of using PRINT.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – PRINTing Status Using RAISERROR With NOWAIT Option