My virtual machines are my play grounds for any research. A few days ago, on my lab machine, I was having difficulties applying latest service pack. I followed the documentation and looked at the files. In this blog post we will take a look at the service pack error. Here is what I found in Summary.txt

Instance MSSQLSERVER overall summary:
Final result: The patch installer has failed to update the shared features. To determine the reason for failure, review the log files.
Exit code (Decimal): -2068578304
Exit facility code: 1204
Exit error code: 0
Exit message: Attributes do not match. Present attributes (Directory, Compressed) , included attributes (0), excluded attributes (Compressed, Encrypted).
Start time: 2017-04-06 08:05:41
End time: 2017-04-06 08:06:04
Requested action: Patch

The above messages “Attributes do not match. Present attributes (Directory, Compressed) , included attributes, excluded attributes (Compressed, Encrypted).”clearly states that the folder of SQL Server is compressed.

WORKAROUND/SOLUTION

This is because I was trying to patch SQL where I compressed SQL related folders, which is not supported. To fix this I did below.

• Navigated to C:\Program Files folder
• Right click on properties on Microsoft SQL Server
• Under Advanced option in General tab, look for compress contents to save disk space, uncheck it.
• Also, uncheck encrypt contents to secure data. To avoid another error which might come due to this.
• Same for C:\Program Files (x86) related SQL folders as well.

It looks like there are others who have reported this https://connect.microsoft.com/SQLServer/feedback/details/704250/installation-failed-when-c-drive-compressed

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Service Pack Error – Attributes do not Match

In the recent few days, I have found many clients having various script upgrade failures while applying service packs or cumulative updates. Here are few of them. Let us learn about  Script Upgrade Failure.

SQL SERVER – Script level upgrade for database ‘master’ failed – CREATE SCHEMA failed due to previous errors

SQL SERVER – Script level upgrade for database ‘master’ failed – There is already an object named ‘DatabaseMailUserRole’ in the database

Here is one more ERRORLOG where SQL Server was not able to come online after failover.

2017-04-06 23:50:18.04 spid5s Configuration option ‘allow updates’ changed from 1 to 1. Run the RECONFIGURE statement to install.
<{81E6B8A0-60E9-4A9F-82E3-FD5326CC3663}>RsFxFt.Dll::RsFxMgmtInitialize failed: Error 0x80070002 (-2147024894)
<{C580416B-A13E-4ECD-B61B-AAFAE39E5E35}>Failed to initialize the CFsaShareFilter interface
<{1038F43D-3391-45F7-B1B3-BADF26459429}>Failed to initialize CFsaShareFilter: Error 0x80070002 (-2147024894)
2017-04-06 23:50:18.05 spid5s Error: 5597, Severity: 16, State: 1.
2017-04-06 23:50:18.05 spid5s FILESTREAM feature could not be initialized. The Windows Administrator must enable FILESTREAM on the instance using Configuration Manager before enabling through sp_configure.
2017-04-06 23:50:18.05 spid5s Error: 912, Severity: 21, State: 2.
2017-04-06 23:50:18.05 spid5s Script level upgrade for database ‘master’ failed because upgrade step ‘msdb110_upgrade.sql’ encountered error 5597, state 1, severity 16. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
2017-04-06 23:50:18.05 spid5s Error: 3417, Severity: 21, State: 3.
2017-04-06 23:50:18.05 spid5s Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.
2017-04-06 23:50:18.05 spid5s SQL Server shutdown has been initiated

WORKAROUND/SOLUTION

The error message 0x80070002 means ERROR_FILE_NOT_FOUND. The system cannot find the file specified.

I am not sure, but it looks like something went wrong with file-stream related files/DLLs. Since my client as not using file-stream so I suggested them to disable it and try. They Disabled filestream from configuration manager and then SQL could come online.

Have you even seen such upgrade issues related to Script Upgrade Failure?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Script Upgrade Failure – RsFxFt.Dll::RsFxMgmtInitialize failed: Error 0x80070002

Have you come across a situation where setup is taking many hours while navigating from one screen to another in clustered environment?  Have you ever waited for many hours watching following message related to slow installation wizard on the cluster?

Please wait while Microsoft SQL Server 2016 Setup process the current operation

When I looked into Detail.txt, I found something like below

(01) 2017-04-08 16:14:46 Slp: Running Action: RunRemoteDiscoveryAction
(01) 2017-04-08 16:14:46 Slp: Running discovery on local machine
(01) 2017-04-08 16:14:47 Slp: Discovery on local machine is complete
(01) 2017-04-08 16:14:47 Slp: Running discovery on remote machine: CRM-SQLNODE1
(01) 2017-04-08 21:13:35 Slp: Discovery on CRM-SQLNODE1 is complete
(01) 2017-04-08 21:13:35 Slp: Completed Action: RunRemoteDiscoveryAction, returned True

Notice the time in each row. There is a HUGE gap of time between two lines and that’s the time SQL setup is doing a remote discovery to find out what is there on a other node of the cluster.

When I asked, they informed that they have done hardening of the server as per the company standards. While checking their hardening document, we found that Admin shares were disabled.

WORKAROUND/SOLUTION

The remote discovery issue was resolved by setting the following registry key to 1.

HKLM\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters\AutoShareServer

Reference https://technet.microsoft.com/en-us/library/aa997392(v=exchg.80).aspx

• Open a registry editor, start > Run > Regedit.exe.
• Navigate to: HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters
• In the right pane, locate and double-click AutoShareServer.
• Change the value from 0 to 1.
• Close the registry editor, and restart the Server service for the change to take effect.

Have you seen this issue earlier? Do you know any other such issues due to hardening?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Slow Installation Wizard on Cluster – Please Wait While Microsoft SQL Server 2016 Setup Process the Current Operation

Sometimes DBA do something which they are not aware of and end up in looking at the logs to see what went wrong. Here is one of the articles I wrote about the changing service account from configuration manager. Why to Use SQL Server Configuration Manager Over Services applet (services.msc)? – Interview Question of the Week #112. In this blog post, let us learn how to fix logon failure error.

Now the situation here was not same, but kind of similar. SQL Server service was not getting started and ERRORLOG was also not getting generated. I looked into System event logs and found below the message.

The MSSQLSERVER service was unable to log on as SQLAuthority\SQLFarmService with the currently configured password due to the following error:
Logon failure: the user has not been granted the requested logon type at this computer.
This service account does not have the required user right “Log on as a service.”

WORKAROUND / SOLUTION

We can see the permission in the error message. In an attempt to fix above error, we did below.

1. Click on Start > Run
2. Open SecPol.msc to open local security policy.
3. Go to Security Settings > Local Policies > User Rights Assignment
4. Locate Log on as a Service (This is what we had in error message)
5. Make sure SQL Service account is added in this. In our case, it was missing so we added it manually.

We attempted to start again and it failed with new error.

A fatal error occurred while creating an SSL server credential. The internal error state is 10013.
This is generally due to incorrect TLS settings. We took backup of below registry key
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\protocols\TLS1.0

We set the value of enabled as 1 and restarted the machine to take effect. Once restarted, we were able to start SQL Service.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Logon Failure: The User has not Been Granted the Requested Logon Type at This Computer

During one of my consulting engagement about Always On availability group configuration, I found that listener network name was not coming online. As a rule of thumb, I always generate cluster log for any error related to cluster. SQL SERVER – Steps to Generate Windows Cluster Log?

Here is the information in the cluster log. I have removed the timestamp column to make it readable.

DBG [RHS] Resource SQLAUTH-PRODAG_SQLAUTH-LISTENER called SetResourceStatusEx: checkpoint 2. Old state OnlinePending, new state OnlinePending, AppSpErrorCode 0, Flags 0, nores=false
ERR [RES] Network Name: [NNLIB] Unable to add server name SQLAUTH-LISTENER to transport \Device\NetBt_If4, status 2
INFO [RES] Network Name : Netbios: Registered Name: SQLAUTH-LISTENER into Netbios (Type: Singleton), result: 2 (0 transports registered)
INFO [RES] Network Name : Netbios: Slow Operation, FinishWithReply: 2
INFO [RES] Network Name : Netbios: End of Slow Operation, state: Initializing/Idle, prevWorkState: Idle
INFO [RES] Network Name: Agent: OnInitializeReply, Failure on (1dac30da-a46a-4c44-923b-d428c6329d6d,Netbios): 2

I look for ERR in cluster log and here is the error which we can see above.

Unable to add server name SQLAUTH-LISTENER to transport \Device\NetBt_If4, status 2

Status 2 indicates: The system cannot find the file specified”. My search for “NetBt” was pointing to issue with “NetBIOS over TCPIP” setting

WORKAROUND/SOLUTION

When we checked Network Name resource properties, we saw “The system cannot find the file specified.” for NetBIOS status. Now to clear that, I looked at IP address resource and unchecked NetBIOS related setting as shown below.

If you want to change that setting a NIC level, then you can follow my earlier blog.

SQL SERVER – FIX Error – Cluster Network Name showing NETBIOS status as “The system cannot find the file specified”

After fixing above, we were able to bring listener online.

Let me know if you have ever faced similar error with windows cluster log. I would like to know your opinion and workaround as well. Remember sharing is the caring.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Unable to Add Server Name SQLAUTH-LISTENER to Transport Device NetBt_If4 Status 2 – Windows Cluster

One of my clients contacted me to see if I can provide any help in doing data recovery. Since my expert area is performance tuning, I offered them quick 15-minute time to see what the errors are and whether I have some techniques to recover the data for invalid OS error 87.

Here was the situation:

1. They were not able to restore from the backup.

Msg 3203, Level 16, State 1, Line 2
Read on “K:\SQLBackups\AuditDB_backup_2017_03_04_000038_1441415.bak” failed: 13(The data is invalid.)
Msg 3013, Level 16, State 1, Line 2
RESTORE DATABASE is terminating abnormally.

The error: “The data is invalid” is more of an operating system way of telling that file is not in a good shape

2. When I looked into the ERRORLOG I found failures as below.

Error: 18210, Severity: 16, State: 1.
BackupIoRequest::ReportIoError: read failure on backup device ‘ K:\SQLBackups\AuditDB_backup_2017_03_04_000038_1441415.bak’. Operating system error 87(The parameter is incorrect.).

The error: “The parameter is incorrect” is again of an operating system way of telling that file is not in a good condition.

3. Another error in ERRORLOG

Error: 3266, Severity: 16, State: 1.
The backup data at the end of ” K:\SQLBackups\AuditDB_backup_2017_03_04_000038_1441415.bak” is incorrectly formatted. Backup sets on the media might be damaged and unusable. To determine the backup sets on the media, use RESTORE HEADERONLY. To determine the backup sets on the media, use RESTORE HEADERONLY. To determine the usability of the backup sets, run RESTORE VERIFYONLY. If all of the backup sets are incomplete, reformat the media using BACKUP WITH FORMAT, which destroys all the backup sets.

WORKAROUND/SOLUTION

All error explained above are very dangerous and a SQL expert may not be able to help much. There are data recovery companies who might help. So, your best bet at this point would be either to contact hardware related company or data recovery company to help. If you have some other good backup, go with that.

Do let me know if you have any other suggestions in the comment area for the ODS error.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – OS Error 13 – The Data is Invalid OS Error 87 – The Parameter is Incorrect – Various Errors due to Faulty Media or Hardware

In SQL Server, there are error few messages which can be caused due to multiple reasons. One of the famous example is “Login Failed for User” error message. If you have seen this earlier, you would know that this message can come due to incorrect user name, incorrect password, incorrect database and many more other reasons. Whenever I see login failed, I look at SQL Server ERRORLOG to see the exact cause.

Here is one of the error where I was not clear about the cause.

Here were the messages in ERRORLOG.

Login failed. The login is from an untrusted domain and cannot be used with Windows authentication. [CLIENT: 169.111.227.120] SSPI handshake failed with error code 0x8009030c, state 14 while establishing a connection with integrated security; the connection has been closed. Reason: AcceptSecurityContext failed. The Windows error code indicates the cause of failure. The logon attempt failed [CLIENT: 169.111.227.120]

This was coming from the application server which was same domain. Based on my search on the internet, there is some kind of loopback, check taking place which causes trusted connections via the loopback adapter to fail.

WORKAROUND/SOLUTION

Loopback check can be removed by adding a registry entry as follows:

• Edit the registry using regedit. (Start –> Run > Regedit )
• Navigate to: HKLM\System\CurrentControlSet\Control\LSA
• Add a DWORD value called “DisableLoopbackCheck”
• Set this value to 1

If above doesn’t solve the issue, we need to create SPNs for SQL Service account. While searching for SETSPN.exe command I came across this nice tool which can help a lot of pain like syntax error etc. is called as “Microsoft® Kerberos Configuration Manager for SQL Server®” which can be downloaded from https://www.microsoft.com/en-us/download/details.aspx?id=39046

The best piece about this tool is that it can help in finding missing SPN and provide script to run or fix it directly, if you have permission. Basically, it can

• Gather information on OS and Microsoft SQL Server instances installed on a server.
• Report on all SPN and delegation configurations on the server.
• Identify potential problems in SPNs and delegations.
• Fix potential SPN problems.

Have you used this tool earlier?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Login Failed. The Login is From an Untrusted Domain and Cannot be Used with Windows Authentication

If we look at the error in the subject line, it is very clear that a full backup is needed so that the differential backup chain can be initiated. If this happens to a newly created database, then it makes sense and there is no need for me to write a blog. There was an interesting finding about the cause of this error, other than obvious one, so sharing it.  Let us see another error related to differential backup.

My client contacted me saying that they are having a database in production for which they have full and differential backups configured using maintenance plan. Sometimes they see below failure message for the differential backup plan under the history.
Failed:(-1073548784) Executing the query “BACKUP DATABASE [HRMS] TO DISK = N’O:\\Backup\\HRMS…” failed with the following error: “Cannot perform a differential backup for database “HRMS”, because a current database backup does not exist. Perform a full database backup by reissuing BACKUP DATABASE, omitting the WITH DIFFERENTIAL option. BACKUP DATABASE is terminating abnormally.”. Possible failure reasons: Problems with the query, “ResultSet” property not set correctly, parameters not set correctly, or connection not established correctly.

They also showed me successful run on the full backup plan and it was not having any errors. So, why is SQL confused and thinks there is not full backup. I asked them to try the same using T-SQL in management studio and it failed with the same error.

Msg 3035, Level 16, State 1, Line 1
Cannot perform a differential backup for database “HRMS”, because a current database backup does not exist. Perform a full database backup by reissuing BACKUP DATABASE, omitting the WITH DIFFERENTIAL option.
Msg 3013, Level 16, State 1, Line 1
BACKUP DATABASE is terminating abnormally.

I looked into SQL Server ERRORLOG and found something interesting for this database.

2017-04-13 06:39:50.10 spid52 I/O is frozen on database HRMS. No user action is required. However, if I/O is not resumed promptly, you could cancel the backup.
2017-04-13 06:40:50.11 spid52 I/O was resumed on database HRMS. No user action is required.
2017-04-13 06:40:50.24 Backup Database backed up. Database: HRMS, creation date(time): 2016/09/29(00:37:12), pages dumped: 426, first LSN: 55:395:37, last LSN: 55:414:1, number of dump devices: 1, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {‘{1A8E69AB-E97B-4CCE-8B3C-7E62573B8FD4}4’}). This is an informational message only. No user action is required.

If we take SQL native backups, we would never see “I/O is frozen” and “I/O was resumed” messages. Moreover, the TYPE of backup would be FILE not VIRTUAL_DEVICE as shown in 3^rd message. We used query from my blog to check the backups for this database. https://blog.sqlauthority.com/2010/11/10/sql-server-get-database-backup-history-for-a-single-database

We also check the report https://blog.sqlauthority.com/2014/08/04/sql-server-ssms-backup-and-restore-events-report/

At this point, we were clear that these are backups happening from some 3^rd party tool. These tools use VSS to take backup of the machine. This is not a normal full database backup carried directly by SQL Server but through the Virtual Device Interface (VDI). I search and found there’s a similar known issue below:

https://support.microsoft.com/en-us/help/903643/the-ntbackup-exe-operation-may-break-the-differential-backup-chain-of-a-sql-server-database-and-may-invalidate-the-differential-backups-when-you-perform-a-snapshot-backup-of-the-database

WORKAROUND/SOLUTION

In this case, we figured out that these were Azure Virtual Machines and recently they configured VM level backup from Azure Portal. I looked into Microsoft’s documentation and I did find a solution to fix this on Azure VMs. We can enable VSS copy using the registry:

https://azure.microsoft.com/en-in/documentation/articles/backup-azure-vms-introduction/#how-does-azure-back-up-virtual-machines

Azure Backup takes VSS Full back-up on the Windows VMs. To enable VSS Copy Backups the below registry key needs to be set on the VM.

[HKEY_LOCAL_MACHINE\SOFTWARE\MICROSOFT\BCDRAGENT] “USEVSSCOPYBACKUP”=”TRUE”

We created key and verified that Azure was taking copy_only backup and maintenance plan were able to run a maintenance plan successfully along with Azure VM backups.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Msg 3035, Level 16 – Cannot Perform a Differential Backup for Database “SQLAuthority”, Because a Current Database Backup Does not Exist

My every interaction with client teaches me something new. Recently someone contacted me through email and informed that they have issues with SQL Server Startup. When we checked ERRORLOG, we found below messages at the end of the ERRORLOG where Via Support library was mentioned.

The SQL Server failed to initialize VIA support library [QLVipl.dll]. This normally indicates the VIA support library does not exist or is corrupted. Please repair or disable the VIA network protocol. Error: 0x7e.

TDSSNIClient initialization failed with error 0x7e, status code 0x60.
TDSSNIClient initialization failed with error 0x7e, status code 0x1.

I asked them to check SQL Server configuration manager and look for the Protocols enabled for that instance of SQL Server. As per error message, it caused due to VIA protocol. Sure enough, it was enabled.

WORKAROUND/SOLUTION

We disabled VIA protocols using the SQL Server Configuration Manager. In the highlighted section, right click and choose disable.

On a side note, when I checked SQL Server 2016 configuration manager, I found that this is already removed from the product.

Here are a few blog posts which I have written on this subject you may find them helpful as well.

• SQL SERVER – Connecting to Azure Storage with SSMS
• SQL SERVER – How to Connect Using NT AUTHORITY \ SYSTEM Account?
• SQL SERVER – Connecting to Azure SQL DB

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX: The SQL Server Failed to Initialize VIA Support Library [QLVipl.dll]

Recently I came across an issue where I was not able to fix directly. My client reported below error in Skype, which is related to a service pack. 
Overall summary:
Final result: The patch installer has failed to update the shared features. To determine the reason for failure, review the log files.
Exit code (Decimal): -2068774911
Exit facility code: 1201
Exit error code: 1
Exit message: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.

I asked them to check below article

https://support.microsoft.com/en-us/help/2379466/fix-0x84b10001-error-code,-sql-server-2008-or-sql-server-2008-r2-setup-failure,-and-an-unhandled-exception-occur-when-you-run-sql-server-2008-or-sql-server-2008-r2-setup

But they were using SQL Server 2016 and having issues with latest patch. As per above “There are many possible scenarios that may initiate this issue. For example, a failure of a previous installation of SQL server may corrupt the registry, and this registry corruption may initiate this issue.”

I asked for Detail.txt and contains following details:

Microsoft.SqlServer.Configuration.MsiExtension.SetPatchInstallStateAction.GetInstalledPatchBySQLDiscovery
at Microsoft.SqlServer.Configuration.MsiExtension.SetPatchInstallStateAction.GetInstalledPatch
at Microsoft.SqlServer.Configuration.MsiExtension.SetPatchInstallStateAction.ExecuteAction
at Microsoft.SqlServer.Configuration.MsiExtension.ProductInstallProperty.GetInstalledPackages
at Microsoft.SqlServer.Configuration.MsiExtension.ProductInstallProperty.CollectProductData

WORKAROUND/SOLUTION

From above stack it looks like an issue with MSI/MSP missing issue. My client informed that they have already tried repair and it didn’t help. When I looked around on the internet, it sounded like they have some registry keys missing. I have suggested them to format the machine (after taking data backup) and install a fresh operating system. Later they did the same and restored the databases to the new instance.

Let me know if you have faced this error related to a service pack.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Service Pack Error – Index (Zero Based) Must be Greater Than or Equal to Zero and Less Than the Size of the Argument List

This blog is a response to a comment on our blog. Have you ever encountered below error related to media set?

Msg 3132, Level 16, State 1, Line 15
The media set has 2 media families but only 1 are provided. All members must be provided.
Msg 3013, Level 16, State 1, Line 15
RESTORE DATABASE is terminating abnormally.

Here are the steps to reproduce the error. First, I would take backup of database.

USE master
GO
IF (db_ID('SQLAuthority') is NOT NULL)
DROP DATABASE SQLAuthority
GO
CREATE DATABASE SQLAuthority
GO
BACKUP DATABASE SQLAuthority
TO DISK = 'SQLAuth_Full_Part01.bak',
   DISK = 'SQLAuth_Full_Part02.bak'
WITH FORMAT

And now, try to restore it.

RESTORE DATABASE [SQLAuthority_1] FROM
DISK = N'SQLAuth_Full_Part01.bak'
WITH
MOVE N'SQLAuthority' TO N'E:\DATA\SQLAuthority_1.mdf',
MOVE N'SQLAuthority_log' TO N'E:\DATA\SQLAuthority_1_log.ldf'
GO

As shown below, it fails with error.

I hope you understood the reason of error. If you look at backup command, I have taken a striped backup on 2 files, but during restore I have given only one.

This error message means that the original backup was done as a striped backup where the backup stream was split into two destination files. When attempting to restore, you need to specify all of the same files which were used to take the backup. So, the simple version is: If you back up to 2 files, you must specify 2 files to restore from.  If you back up to 5 files, you must specify 5 files to restore from, etc. You cannot restore any data from less than the full set of files used for the backup.

WORKAROUND/SOLUTION

We need to find out where is part of this backup. Below is the query which can be used to find the missing part of the backup.

DECLARE @DatabaseName NVARCHAR(max)
SET @DatabaseName = N'SQLAuthority'
USE msdb;
SELECT DISTINCT t1.NAME
	,t3.[media_set_id]
	,t6.family_sequence_number
	,t6.physical_device_name
	,(datediff(ss, t3.backup_start_date, t3.backup_finish_date)) / 60.0 AS duration
	,t3.backup_start_date
	,t3.backup_finish_date
	,t3.type AS [type]
	,CASE
		WHEN (t3.backup_size / 1024.0) < 1024
			THEN (t3.backup_size / 1024.0)
		WHEN (t3.backup_size / 1048576.0) < 1024
			THEN (t3.backup_size / 1048576.0)
		ELSE (t3.backup_size / 1048576.0 / 1024.0)
		END AS backup_size
	,CASE
		WHEN (t3.backup_size / 1024.0) < 1024
			THEN 'KB'
		WHEN (t3.backup_size / 1048576.0) < 1024
			THEN 'MB'
		ELSE 'GB'
		END AS backup_size_unit
	,t3.first_lsn
	,t3.last_lsn
	,CASE
		WHEN t3.differential_base_lsn IS NULL
			THEN 'Not Applicable'
		ELSE convert(VARCHAR(100), t3.differential_base_lsn)
		END AS [differential_base_lsn]
	,t6.device_type AS [device_type]
	,t3.recovery_model
FROM sys.databases t1
INNER JOIN backupset t3 ON (t3.database_name = t1.NAME)
LEFT JOIN backupmediaset t5 ON (t3.media_set_id = t5.media_set_id)
LEFT JOIN backupmediafamily t6 ON (t6.media_set_id = t5.media_set_id)
WHERE (t1.NAME = @DatabaseName)
ORDER BY t6.physical_device_name;

Media_set_id and family_sequence_number can tell if there is a split backup.

Do you use this feature?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX: Msg 3132, Level 16 – The Media Set has 2 Media Families but Only 1 are Provided. All Members Must be Provided

One of my friends who is a SharePoint expert told that he is getting overflow error while doing check-in on the document. He was confident that the error is related to SQL and that’s why he contacted me. I used profiler and found the table which had the problem and fixed that as well. This blog is an outcome of that work done. Let us see the error related to Arithmetic Overflow Error.

What is an overflow? Whenever we try to insert a value in a column which is more than the limit of data type, we will get an overflow error. In my case, it was an identified column, which was defined as INT and the below are the variations of error when it comes to IDENTITY columns. It all depends on the data type of the column. The number of values permitted would be dependent on data type.

Msg 8115, Level 16, State 1, Line 25
Arithmetic overflow error converting IDENTITY to data type tinyint.
Arithmetic overflow occurred.
Msg 8115, Level 16, State 1, Line 25
Arithmetic overflow error converting IDENTITY to data type smallint.
Arithmetic overflow occurred.
Msg 8115, Level 16, State 1, Line 25
Arithmetic overflow error converting IDENTITY to data type int.
Arithmetic overflow occurred.
Msg 8115, Level 16, State 1, Line 25
Arithmetic overflow error converting IDENTITY to data type bigint.
Arithmetic overflow occurred.

Here are the steps to reproduce. Keep in mind that tinyint has limit of 255 so if we try to insert 256 in tinyint column, it would fail.

USE [tempdb]
GO
IF OBJECT_ID('SmallTable') is NOT NULL
DROP TABLE [SmallTable]
GO
CREATE TABLE [dbo].[SmallTable] (
	[i] [tinyint] IDENTITY(1, 1) PRIMARY KEY NOT NULL
	,[j] [nchar](20) NULL
	) ON [PRIMARY]
GO
ALTER TABLE [dbo].[SmallTable] ADD CONSTRAINT [DF_SmallTable_j] DEFAULT(N'SQLAuthority')
FOR [j]
GO
SET NOCOUNT ON
GO
-- Inserting and Deleting the rows.
INSERT INTO SmallTable VALUES (DEFAULT)
DELETE FROM SmallTable
GO 255
-- Below inset would fail as we have already reached 255 (limit of tinyint)
INSERT INTO SmallTable VALUES (DEFAULT)
/*
Msg 8115, Level 16, State 1, Line 21
Arithmetic overflow error converting IDENTITY to data type tinyint.
Arithmetic overflow occurred.
*/
-- check the rows in table and since we are inserting and deleting, its having 0 rows.
SELECT * FROM SmallTable

Here is the screenshot of SSMS

WORKAROUND/SOLUTION

In our above example, we have zero rows so how to verify the identity value? Refer my earlier blog.

SQL SERVER – Find Current Identity of Table

DBCC CHECKIDENT('dbo.SmallTable')

Here is the output

Checking identity information: current identity value ‘255’, current column value ‘NULL’.

DBCC execution completed. If DBCC printed error messages, contact your system administrator.
Here is the command which I ran to fix the issue

DBCC CHECKIDENT('dbo.SmallTable',reseed,0)

You can read more about the command over here SQL SERVER – DBCC command to RESEED Table Identity Value – Reset Table Identity

I have selected value as TINYINT as its ready to demonstrate the error. On SharePoint issue which I worked, it was integer and due to amount of project, documents and check-ins, they hit the limit.

Have you seen more variations of above error? Please share via comments.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX : Msg 8115, Level 16, Arithmetic Overflow Error Converting IDENTITY to Data Type INT

Few days back I wrote a blog about error messages during differential backup restore. SQL SERVER – Msg 3136, Level 16 – This Differential Backup Cannot be Restored Because the Database has not Been Restored to the Correct Earlier State

This blog is a variation of the same, but instead of restoring complete backup, we were trying to restore a file from differential backup. Here is the error message

(Msg 3178, Level 16, State 1, Line 26
File SQLAuthority is not in the correct state to have this differential backup applied to it.
Msg 3119, Level 16, State 1, Line 26
Problems were identified while planning for the RESTORE statement. Previous messages provide details.
Msg 3013, Level 16, State 1, Line 26
RESTORE DATABASE is terminating abnormally.

Here are the steps to create the scenario

USE master
GO
CREATE DATABASE SQLAuthority
GO
BACKUP DATABASE SQLAuthority TO DISK = 'SQLAuth_Full_01.bak' WITH FORMAT
GO
BACKUP DATABASE SQLAuthority TO DISK = 'SQLAuth_Diff_01.bak' WITH DIFFERENTIAL, FORMAT
GO
BACKUP DATABASE SQLAuthority TO DISK = 'SQLAuth_Full_02_cp.bak' WITH COPY_ONLY, FORMAT
GO
BACKUP DATABASE SQLAuthority TO DISK = 'SQLAuth_Diff_02.bak' WITH DIFFERENTIAL, FORMAT
GO

The above image shows the series of backup I have taken. You can read more about COPY_ONLY backup in books online. The error would come if I restore Diff_02 on top of Full_02_cp. Below is the code to reproduce the error.

USE [master]
RESTORE DATABASE [SQLAuthority_1] FROM
DISK = N'SQLAuth_Full_02_cp.bak'
WITH
MOVE N'SQLAuthority' TO N'E:\DATA\SQLAuthority_1.mdf',
MOVE N'SQLAuthority_log' TO N'E:\DATA\SQLAuthority_1_log.ldf',
NOUNLOAD,  STATS = 5, NORECOVERY
GO

RESTORE DATABASE [SQLAuthority_1]  FILE = 'SQLAuthority'
FROM  DISK = N'SQLAuth_Diff_02.bak'
WITH RECOVERY,  NOUNLOAD,  STATS = 10
GO

SOLUTION/WORKAROUND

We need to restore differential backup on top of regular full backup, not copy_only full backup. Below command would work as we are restoring diff_02 after performing restore of Full_01.

DROP DATABASE [SQLAuthority_1]
GO
USE [master]
RESTORE DATABASE [SQLAuthority_1] FROM
DISK = N'SQLAuth_Full_01.bak'
WITH
MOVE N'SQLAuthority' TO N'E:\DATA\SQLAuthority_1.mdf',
MOVE N'SQLAuthority_log' TO N'E:\DATA\SQLAuthority_1_log.ldf',
NOUNLOAD,  STATS = 5, NORECOVERY
GO

RESTORE DATABASE [SQLAuthority_1]   FILE = 'SQLAuthority'
FROM  DISK = N'SQLAuth_Diff_02.bak'
WITH RECOVERY,  NOUNLOAD,  STATS = 10
GO

Have you ever encountered such issues during disaster recovery where chain is broken?

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX : Msg 3178, Level 16 – File SQLAuthority is Not in the Correct State to have This Differential Backup Applied to it

Here is another blog based on client’s interaction via Skype. They informed me that they are trying to execute the command ALTER DATABASE SET PARTNER while attempting to set up mirroring between two servers, we see the following mirroring error recorded in the secondary error log:

2017-03-20 14:44:32.49 spid52 Starting up database ‘PROD’.
2017-03-20 14:44:32.51 spid52 The database ‘PROD’ is marked RESTORING and is in a state that does not allow recovery to be run.
2017-03-20 14:46:13.53 Logon Database Mirroring login attempt by user ‘ASIA\SQL_SERVICE.’ failed with error: ‘Connection handshake failed. Could not send a handshake message because the connection was closed by peer. State 26.’. [CLIENT: 171.24.184.12]

At the same time when we checked on the primary server, we were seeing following messages

2017-03-20 14:45:07.05 spid30s Error: 1443, Severity: 16, State: 2.
2017-03-20 14:45:07.05 spid30s Database mirroring has been terminated for database ‘PROD’. This is an informational message only. No user action is required.
2017-03-20 14:45:50.01 spid47s Error: 1474, Severity: 16, State: 1.
2017-03-20 14:45:50.01 spid47s Database mirroring connection error 4 ‘An error occurred while receiving data: ‘10054(An existing connection was forcibly closed by the remote host.)’.’ for ‘TCP://171.24.184.13:5022’.

Later, we identified that this was caused due to authentication issues of service account from the domain controller.

WORKAROUND/SOLUTION

Before I could do much troubleshooting, their networking team informed that ports were blocked in Active directory so they opened them. I didn’t get much detail from them, but they said networking team easily figured it out.

If you know how to verify it, please share via comments.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Mirroring Error: Connection handshake failed. Could not send a handshake message because the connection was closed by peer. State 26

This was really one of the issue where I was totally confused because SQL Server was raising incorrect error message. Let me first tell you what I was doing. I was trying to move TEMPDB database files to a new location. I ran below alter commands and restarted SQL Server. Let us learn about OS Error 5.

ALTER DATABASE tempdb MODIFY FILE
   (NAME = 'tempdev', FILENAME = 'E:\TempDB')

Now, SQL was not getting started so I looked into ERRORLOG and found below the messages. I have trimmed all four 10s (system SPID) which is trying to start up tempdb database.

2017-03-09 09:16:07.84 spid6s Clearing tempdb database.
2017-03-09 09:16:07.84 spid6s Error: 5123, Severity: 16, State: 1.
2017-03-09 09:16:07.84 spid6s CREATE FILE encountered operating system error 5(Access is denied.) while attempting to open or create the physical file ‘E:\TempDB’.
2017-03-09 09:16:07.84 spid6s Error: 17204, Severity: 16, State: 1.
2017-03-09 09:16:07.84 spid6s FCB::Open failed: Could not open file E:\TempDB for file number 1. OS error: 5(Access is denied.).
2017-03-09 09:16:07.84 spid6s Error: 5120, Severity: 16, State: 101.
2017-03-09 09:16:07.84 spid6s Unable to open the physical file “E:\TempDB”. Operating system error 5: “5(Access is denied.)”.
2017-03-09 09:16:07.84 spid6s Error: 1802, Severity: 16, State: 4.
2017-03-09 09:16:07.84 spid6s CREATE DATABASE failed. Some file names listed could not be created. Check related errors.
2017-03-09 09:16:07.84 spid6s Could not create tempdb. You may not have enough disk space available. Free additional disk space by deleting other files on the tempdb drive and then restart SQL Server. Check for additional errors in the event log that may indicate why the tempdb files could not be initialized.
2017-03-09 09:16:07.84 spid6s SQL Server shutdown has been initiated
2017-03-09 09:16:07.84 spid6s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.

From above messages (Access is denied), it looked like permission issue, but I don’t think it was a permission issue as SQL Account was a local administrator on windows. Yeah, I know, I know… not recommended, but who cares when it’s a test VM.

While digging further in ERRORLOG, I realized that there is no reference to MDF and LDF file? Then I looked at my command again and damn…. I missed a file name there and left up to the directory.

WORKAORUND/SOLUTION

As found earlier, it was an incorrect tempdb configuration causing SQL not to start. To fix the configuration, we need to start SQL Service. Catch 22 situation.  Luckily, there are way to start SQL in such situations.  We started SQL Server /f and /m and Altered TempDB to have correct file path and name.

1. Start SQL with /f and /mSQLCMD (SQL SERVER – The Story of a Lesser Known Startup Parameter in SQL Server – Guest Post by Balmukund Lakhani)
2. Connect to SQL via SQLCMD.
3. Run following ALTER command

ALTER DATABASE tempdb MODIFY FILE
(NAME = 'tempdev', FILENAME = 'E:\TempDB\TempDB.mdf')

4. Stop SQL
5. Start SQL normally.

Here is how the command prompt would look like.

Have you ever faced a similar issue? Please comment and let me know.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Unable to Start SQL Service – OS Error 5 (Access is denied.) for TempDB

While trying to install SQL Server 2012 in clustered environment, client reported that it is hanging at “Please Wait” state forever. Eventually they have to kill the setup by task manager. Let us see an error related to RPC Server.

Since I am not an expert in setup, I asked them to have a look at setup log and see if they have any error in Detail.txt
(16) 2016-02-14 15:56:13 Slp: Sco: Attempting to check if container ‘WinNT://ASIA’ of user account exists
(16) 2016-02-14 15:56:15 ACE: ERROR: Encountered exception when validating service account ‘ASIA\SQL_SVC’: Microsoft.SqlServer.Configuration.Sco.ScoException: The RPC server is unavailable.
—> System.Runtime.InteropServices.COMException (0x800706BA): The RPC server is unavailable.
at System.DirectoryServices.DirectoryEntries.Find(String name, String schemaClassName)
at Microsoft.SqlServer.Configuration.Sco.User.LookupADEntry()
— End of inner exception stack trace —
at Microsoft.SqlServer.Configuration.Sco.User.LookupADEntry()
at Microsoft.SqlServer.Configuration.Sco.User.Exists()
at Microsoft.SqlServer.Configuration.Sco.UserSecurity.ValidateCredentials(ServiceContainer ctx, String userName, String password, String serviceName)
at Microsoft.SqlServer.Configuration.Agent.AccountsUtil.ValidateUserNameAndPassword(String serviceName, String userName, SqlSecureString pwd, Boolean userNameRequired, Boolean isCluster, ValidationState validationState, ServiceContainer context)

I had no idea about this error, so I searched on the internet. Few sites on the internet pointed that it could be an issue with DNS suffix. https://technet.microsoft.com/en-us/library/cc754143.aspx

Append primary and connection specific DNS suffixes specifies that resolution for unqualified names that are used on this computer are limited to the domain suffixes of the primary suffix and all connection-specific suffixes. Connection-specific suffixes are configured in DNS suffix for this connection. The primary DNS suffix is configured by clicking Properties on the Computer Name tab (available in System in Control Panel). The local setting is used only if the associated Group Policy is disabled or unspecified.

One of the link mentioned to verify that both machines are able to talk to each other using \\SQLNODE1 and \\SQLNODE2 which was working in this situation.

WORKAROUND/SOLUTION

For this server, we identified that the DNS Primary suffix was not correctly configured on the machine.

This was causing RPC error when resolving names through DNS Resolution. Once we added the DNS suffix with the domain name in the TCP property with the service account is, we could proceed to next screens and install SQL successfully.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Installation Wizard Hangs at Service Account Page – The RPC Server is Unavailable

When I was on-site for the performance tuning workshop, few DBAs suddenly got call as there was an unexpected downtime of SQL Server. As per them, after some maintenance activities were scheduled, the SQL Server resource failed to come online on both nodes. When they try to bring it online, it remains in online pending state for some time before failing eventually. Here are the things I tried. Let us see an error related to clustered SQL Resource not coming online.

• We tried starting it from services.msc and SQL Server service could start successfully.
• When we checked SQL Server ERRORLOG, and found no errors when bringing online in a cluster.
• As a last resort, I generated cluster log and found the below SQL SERVER – Steps to Generate Windows Cluster Log?

• As per above we can see that the cluster service was not to connect to SQL Server
• The error was Login failed for user ‘PRODUCTION\SQLNODE1$’
• This means the account; ‘PRODUCTION\NODE1$’ did not have login permission on SQL Server. This is actually the local machine account (NODE1 was the machine name).

WORKAROUND/SOLUTION

We started SQL Server from services console and gave the sysadmin server role of the LocalSystem (NT AUTHORITY\SYSTEM) account. Here was the command

ALTER SERVER ROLE [sysadmin] ADD MEMBER [NT AUTHORITY\SYSTEM]

After this, we stopped SQL Server from the services console and tried bringing it online from cluster administrator and it succeeded.

As a safely measure, we tested a failover and failback between the SQLNOD1 and SQLNODE2 and it worked perfectly.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Clustered SQL Resource Not Coming Online

Have you even been in a situation where system processes are taking CPU and not moving ahead? Recently, one of my clients showed me a system where checkpoint on the user database being blocked indefinitely by ghost cleanup.

Since I have never seen that behavior earlier, I decided to dig little deeper. I asked them to share SQL Server ERRORLOG files. I found below information in those files.

Error: 845, Severity: 17, State: 1.
A time-out occurred while waiting for buffer latch — type 3, bp 0000000152CE2980, page 10:4240576, stat 0x5c00008, database id: 6, allocation unit Id: 72057595261550592, task 0x00000000092125C8 : 0, waittime 300, flags 0x3a, owning task 0x000000000424A868. Not continuing to wait.

Whenever I see some strange error like above, I first try to make sure that there is no corruption in the database, so I asked them to run DBCC CHECKDB on DBID 10, as mentioned in the above error message. It came back with the following lines at the end of the output

CHECKDB found 0 allocation errors and 40 consistency errors in database ‘iFLUXDB’.
repair_rebuild is the minimum repair level for the errors found by DBCC CHECKDB (iFLUXDB).

WORKAROUND/SOLUTION

In my client’s case, it was caused due to corruption in the database. If we ran DBCC CHECKDB with a repair option provided. It was a sheer luck that repair_rebuild was suggested and we were safe to run it without causing data loss.

If you have same situation due to corruption, then evaluate the possibility of fixing corruption because you might get repair_allow_data_loss as minimum repair level. Restore from backup is always a preferred way as compare to repairing with data loss.

Note: If you have no experience with fixing corruption, I suggest you contact expert rather than attempting this yourself as there are chances to loss data.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Ghost Cleanup Getting Hung and Blocking Checkpoint Process. What’s Wrong?

One of my client was trying to convert database mirroring to the AlwaysOn availability group. They had a mirroring configure where the principal server was going to become a primary replica. When they attempted to join the AG, it was failing with an error. The error messages we were getting in the wizard are not helpful. I went ahead and looked into SQL Server ERRORLOG and found below the messages

The database Mirroring login attempt failed with error: ‘Connection handshake failed. There is no compatible encryption algorithm. State 22.’. [CLIENT: ]

We scripted the endpoint from both replicas using below method.

Primary

CREATE ENDPOINT [Mirroring]
      STATE=STARTED
      AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
      FOR DATA_MIRRORING (ROLE = ALL, AUTHENTICATION = WINDOWS NEGOTIATE
, ENCRYPTION = REQUIRED ALGORITHM AES)
GO

Secondary

CREATE ENDPOINT [Hadr_endpoint]
      STATE=STARTED
      AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
      FOR DATA_MIRRORING (ROLE = ALL, AUTHENTICATION = WINDOWS NEGOTIATE
, ENCRYPTION = REQUIRED ALGORITHM RC4)
GO

You can also use below to find an algorithm on both primary and secondary.

SELECT encryption_algorithm_desc FROM sys.database_mirroring_endpoints

Based on the error message, you should see different value on both ends. In my case I had one server with RC4 and one server with AES. They must match for AlwaysOn data movement to work.

SOLUTION/WORKAROUND

The solution in this situation is to make sure that we have same value passed in ALGORITHM parameter on both the sides. To fix the issue, we dropped and recreated endpoint with the same script to make sure it’s the same algorithm.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Database Mirroring Login Attempt Failed With Error: ‘Connection Handshake Failed. There is No Compatible Encryption Algorithm. State 22

While trying to help one of my clients, I was shown below error which was received using bulk insert command. The command was used to move data from Azure Blob Storage to SQL Azure Database. Let us see how to fix the error processing request.

Msg 40197, Level 20, State 1, Line 16
The service has encountered an error processing your request. Please try again. Error code 40549.

The file was big and error doesn’t come immediately. It takes several hours before we get error. I looked into the message I found 40549 which is another message number.

select * from sys.messages where message_id = 40549

Here is what it means.
Session is terminated because you have a long running transaction. Try shortening your transaction.

And now it makes perfect sense. Here was the T-SQL which they were using.

INSERT INTO HumanResource.EmployeeMaster
SELECT *
FROM OPENROWSET(BULK 'Employee.dat', DATA_SOURCE = 'blobsource',
       FORMATFILE='fmt/Employee.xml', FORMATFILE_DATA_SOURCE = 'blobsource') as data;

SOLUTION/WORKAROUND

As per the error message and behavior, we need to reduce the size of batch to make the transaction shorter. When I searched ways to do that in BCP I landed up https://technet.microsoft.com/en-us/library/ms188267(v=sql.105).aspx (Managing Batches for Bulk Import).  I found ROWS_PER_BATCH parameter which worked like a charm.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL Azure Database – Msg 40197, Level 20 – The Service has Encountered an Error Processing Your Request. Please Try Again. Error Code 40549