Sometimes it is important to read the documentation before assuming things based on the text. This blog is the result of one of my quick interaction with a client who had an issue with Maintenance Plan which they created for backup files.

Pinal,
We are having issues with a maintenance plan. We have created plans to take log backup every 15 mins and configured it in such a way that it deletes the backup. The maintenance plan is running fine, but backups are not getting deleted. Do you have any ideas?

Within a few minutes, I replied to my blog which I wrote on similar behavior. SQL SERVER – Maintenance Plan – Maintenance Cleanup Task not Deleting Files

Their response made me laugh.

Pinal,
We have gone through the blog, but we are not using clean task. We have selected maintenance plan with the option “backup set will expire” to delete the old backup files. Do you think it’s a bug?

I replied with a one liner.

“Backup set will expire”, means an expiration date to indicate when the backup set can be overwritten by another backup not that the file/data is removed.

Please use a cleanup task to achieve what you want.

SOLUTION/WORKAROUND

To remove old backup files, we need to use maintenance plan cleanup task (refer my earlier blog). We might need to create two tasks. One to clean up the backup files and other to cleanup maintenance plan reports.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Maintenance Plan – Backup Files Not Getting Deleted

If you are seeing this error in the ERRORLOG on your production server then you must read this blog. I have collected some information about the message and its causes. Let us learn about Autogrow of file.

First of all, the error message would come when the database file(s) is set to grow automatically and SQL is not able to complete the growth in time. Let’s assume that you start a transaction from application which causes the file to grow and growth kicks in. The time taken by the transaction would include real work done (any DML) and the time taken to grow the file. Now, if due to some reason the growth has taken more time and total transaction time is more than timeout period, the application would cancel the transaction. This would cause above error message and file won’t be grown. If same transaction comes again and same cycle repeats, you would see a continuous message in the ERRORLOG file.

2017-03-14 18:32:58.25 spid118 Autogrow of file ‘SQLAuth_Log’ in database ‘SQLAuth’ was cancelled by user or timed out after 10046 milliseconds. Use ALTER DATABASE to set a smaller FILEGROWTH value for this file or to explicitly set a new file size.

Based on my understanding, the error message can come due to two possible reasons:

1. The growth setting of Transaction log is set to a huge size. This can generally happen percent growth is used instead of fix growth.

The advice in this situation would be to use fixed size, growth rather than percentage growth.

2. Another possibility that the disk is performing slow. Even if the size of the growth is small, due to disk slowness, it might take more time to complete the growth and might time out.
3. If you still want to let the file grow automatically, then at least make sure that you provide perform volume maintenance task permission to the SQL Server service account. Instant file initialization would be activated when the service account has been granted SE_MANAGE_VOLUME (Perform Volume Maintenance Tasks). https://technet.microsoft.com/en-us/library/ms175935.aspx.This would help in “quick” growth of the data files NOT transaction log file because log file would be zeroed out.

Few of client, keep a track of database growth and grow the file manually during a scheduled time. I like this approach because you are making sure that database file doesn’t grow automatically.

If you have any more thought, please feel free to share via comments.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – My Thoughts – Autogrow of File ‘SQLAuth_Log’ in Database ‘SQLAuth’ was Cancelled by User or Timed Out After 30020 Milliseconds

One of my clients implemented AlwaysOn Multi-subnet listener. This is one of the message which I have heard from few other AlwaysOn availability group customers who are having multi-subnet deployment. This client reported that they are not able to bring listener online on one node, which was added newly. When they attempted to fail over AG resource in the cluster, we could see below messages reported in event logs. Let us see how we can fix errors related to no matching network.

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Event ID: 1045
Level: Warning
Source: IP Address Resource
Description: IP Address Resource NT AUTHORITY\SYSTEM No matching network interface found for resource ‘Cluster IP Address’ IP address ‘10.10.10.10’ (return code was ‘5035’). If your cluster nodes span different subnets, this may be normal.

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Event ID: 1069
Level: Error
Source: Resource Control Manager
Description: Cluster resource ‘Cluster IP Address’ of type ‘IP Address’ in clustered role ‘Cluster Group’ failed. Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet.

Above message contains return code 5035 which means A cluster network is not available for this operation.

EXPLAINATION

As we can see in the warning, this is expected since we are at a site that does not have a NIC with a 10.10.x.x IP address.  This is standard behavior in multisite clusters since we have to set up a network for each subnet in use at each site to allow the cluster to properly manage bringing IP addresses online and offline based on the networks available to the cluster node we are moving to.

SOLUTION/WORKAROUND

For failover to work, we must have two IP address, one for each subnet for Windows Cluster IP as well as AlwaysOn listener IP address.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Event ID 1045 – No Matching Network Interface Found for Resource ‘Cluster IP Address’ IP address ‘10.10.10.10’ (Return Code was ‘5035’)

Even though my consulting is mostly around performance tuning, I can see many customers who want to start using the new capabilities of SQL Server for their existing application as they plan to do an upgrade of their infrastructure. In one of engagement with a bank, they wanted to explore more about SQL Auditing capability. Since they were new I got engaged with them in implementing compliance, which they must follow. In the recent past, I have written below blogs on SQL Audit.

SQL SERVER – Error 33222, Level 16 – Audit ‘MyAudit’ Failed to Start

SQL SERVER – Script to Audit Login and Role Member Change

While I was on site, one of their DBA readers sent me an email stating that the time in the log is incorrect. I asked him to share the screenshot. I also reproduce the same behavior in my own server.

As we can see below, the time I clock is around 5:30 AM and in audit log it was shown as around midnight.

WORKAROUND/SOLUTION

I looked at the time difference reported and found that there is a difference on 5 hr. 3 min. which is the exact time difference between IST and UTC!! After some internet searching, it was clear that the SQL audit will record the time in UTC time into the logs rather than local time. So, we need to use the following query in our script

SELECT DATEADD(MINUTE, DATEDIFF(MINUTE, GETUTCDATE(), CURRENT_TIMESTAMP), event_time) AS event_time_afterconvert
	,getdate() 'Current_system_time'
	,*
FROM fn_get_audit_file('G:\DATA\*', DEFAULT, DEFAULT)

Once we understood the concept, we implemented SQL Audit and they were able to get compliance reports.

What tool do you use to audit SQL Server activities? Do you use 3^rd party solutions? Please reply by the comments and let me know.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – SQL Audit Date Time Does Not Match Machine Date Time – Solution

One of a unique advantage of freelancing is learning more and more. I get a chance to hear some of the real word problem and it gives me immense pleasure to provide a feasible solution to them. This blog is interaction on the airport when I was coming back from a recent trip from Canada. After seeing me on airport, an old client of mine met and said hello. While having a discussion, he told an interesting issue of SQL upgrade and he wanted an idea. He showed me below error message related to edition upgrade.

Here is the text of the error message. Did you know that you can click on that little copy icon to get details like below?

SQL Server Setup has encountered the following error:
Invoke or BeginInvoke cannot be called on a control until the window handle has been created..

After we hit OK on the dialog, SQL Server installation exits.

WORKAROUND/SOLUTION

When I searched for the error on the internet, It was easy to find here.

But we were running SQL 2012 upgrade so above was not applicable.

I have given him below idea to avoid using UI to run the installer. I shared below with him and he agreed to update me via email if it worked for him.

setup.exe /ACTION=EditionUpgrade /INSTANCENAME=MSSQLSERVER /IACCEPTSQLSERVERLICENSETERMS  /FEATURES=SQL,RS  /INDICATEPROGRESS  /Q

How many times do you install SQL Server or a patch via command line? In larger enterprise, it is very common and they love scripts and command line. Feel free to comment and let me know.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Edition Upgrade – Invoke or BeginInvoke Cannot be Called on a Control Until the Window Handle has Been Created

Once upon a time a customer wanted to apply a patch on SQL Server instance, but it failed at the end to start SQL Server services. As usual, I asked to check ERRORLOG and found below information. Let us see errors related to the DLL.

2017-05-01 11:04:40.00 spid5s Error: 17750, Severity: 16, State: 0.
2017-05-01 11:04:40.00 spid5s Could not load the DLL xpstar.dll, or one of the DLLs it references. Reason: 126(The specified module could not be found.).
2017-05-01 11:04:40.00 spid5s Uploading data collector package from disk: Install\SqlTraceCollect.dtsx
2017-05-01 11:04:40.00 spid5s Uploading data collector package from disk: Install\SqlTraceCollect.dtsx
2017-05-01 11:04:40.00 spid5s Error: 4861, Severity: 16, State: 1.
2017-05-01 11:04:40.00 spid5s Cannot bulk load because the file “Install\SqlTraceCollect.dtsx” could not be opened. Operating system error code 3(The system cannot find the path specified.).

The message in the ERRORLOG was clear that it was not able to find xpstart.dll or its dependency. When we check BINN path (E:\Program Files\Microsoft SQL Server\MSSQL11.MSSQLSERVER\MSSQL\Binn), we were able to see the DLL there and we were not sure what the problem could be?

We tried starting SQL from the command line using below options

NET START MSSQLSERVER /m /T3608 /T902

When we attempted to connect to SQL Server via SQLCMD, we were welcomed with the below error

—————————
SQLCMD.EXE – System Error
—————————
The program can’t start because ATL100.DLL is missing from your computer. Try reinstalling the program to fix this problem.
—————————
OK
—————————

Now we know that this could be one of the dependent DLL. We saw that ATL100.dll was missing under C:\windows\system32.

WORKAROUND/SOLUTION

As soon as we know the DLLs was missing, we don’t know how many such more DLLs might be missing. Based on my search on the internet it comes from Visual Studio Shell (vc_red.msi). I asked them to install VS Shell from SQL media and meantime, copy that missing file from a working machine. As soon as file was copied, we were able to connect SQL using SQLCMD and then we start SQL server normally.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Could Not Load the DLL xpstar.dll, or One of the DLLs it References

There have been many situations where rather than performance tuning, I am contacted for an installation issue. Due to my knowledge by looking at logs and pinpointing at the error message, I am able to fix them as well. In this blog post we will learn how to fix Error Installing Microsoft Visual Studio 2010 Shell.

Here is another blog which is related to patching issue which I was asked to fix recently. In the UI, the failure was caused while installing Visual Studio 2010 Shell. I looked at log pointed which was VC10Redist_Cpu64_1.log

Here is the content of the file.

******* RunEngine:
******* Product: C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi
******* Action:
******* CommandLine: **********
MSI (s) (A8:84) [13:58:01:399]: Note: 1: 2203 2: C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi 3: -2147287037
MSI (s) (A8:84) [13:58:01:399]: MainEngineThread is returning 3
MSI (s) (A8:64) [13:58:01:415]: User policy value ‘DisableRollback’ is 0
MSI (s) (A8:64) [13:58:01:415]: Machine policy value ‘DisableRollback’ is 0
MSI (s) (A8:64) [13:58:01:415]: Incrementing counter to disable shutdown. Counter after increment: 0
MSI (s) (A8:64) [13:58:01:415]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2
MSI (s) (A8:64) [13:58:01:415]: Decrementing counter to disable shutdown. If counter
MSI (c) (90:CC) [13:58:01:415]: MainEngineThread is returning 3

This was the only information in the log. If we have a look at the initial stage of the setup file, we see an error related to C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi

I checked the file and it was not there in the location mentioned above. We were not sure how it was missing. May be someone deleted the file due to space issue on C drive? For the path, it looks like the SQL setup caches the media locally for un-installation purpose?

WORKAROUND/SOLUTION

We download patch from KB3045318 again and looked for the file. It was found under VC10SP1\x64\vc_red.msi  folder. The fix the above issue was that we had to copy vc_red.msi directly into VC10SP1 so that the final path looks like — C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Update Cache\KB3045318\GDR\redist\VisualStudioShell\VC10SP1\vc_red.msi

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Error Installing Microsoft Visual Studio 2010 Shell

I write and play around a lot with SQL object level permissions and I love working with them. Having said that, I also explore the ways people want to run code to achieve certain objectives. In the DBA’s world when the number of servers we are talking is not a single digit, they are looking for ways to automate and script out. Recently I was at a retail company backed team in India and they said they have close to 1500+ databases which are running at their various outlets and point of sale counters and the DBA team working to manage these were still less than 10. I was pleasantly surprised by this level of details. To achieve certain tasks, they have to allow non-sysadmin users to run xp_cmdshell. Here is the issue which was they wanted to solve.

When we try to run a simple stored procedure that use xp cmdshell, it fails with below error:

Msg 229, Level 14, State 5, Procedure xp_cmdshell, Line 1 [Batch Start Line 0] The EXECUTE permission was denied on the object ‘xp_cmdshell’, database ‘mssqlsystemresource’, schema ‘sys’.

They followed error message and the granted permission to execute the stored procedure but got next error

Msg 15153, Level 16, State 1, Procedure xp_cmdshell, Line 1 [Batch Start Line 3] The xp_cmdshell proxy account information cannot be retrieved or is invalid. Verify that the ‘##xp_cmdshell_proxy_account##’ credential exists and contains valid information.

WORKAROUND/SOLUTION

I explained the reason and risks involved in using xp_cmdshell and they wanted to use proxy account. I provided below link which explains about it.

https://docs.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/xp-cmdshell-transact-sql

Here are the complete steps which can fix the issue for others.

1. Create proxy for xp_cmdshell. This is a mapping to provide account under whose credential xp_cmdshell will run. It is advisable that this should NOT be a part of a windows local administrator group else it would be like shooting in the foot.

EXEC sp_xp_cmdshell_proxy_account 'fareast\blakhani', 'Password for the user given here'

2. Grant execute to the user

GRANT EXECUTE ON xp_cmdshell TO foo1

In short, the proxy account needs to be a windows account. Then any non-sysadmin user (windows account or SQL account) needs public access to master and execute permission on xp cmdshell.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Fix: The EXECUTE Permission was Denied on the Object ‘xp_cmdshell’, Database ‘mssqlsystemresource’,schema ‘sys’.

Along with my performance consulting, I also assist many clients in fixing some quick issue. You can refer for various consulting services which I provide. This blog is an outcome of one such engagement where the client has deployed standalone SQL Server on two machines, created windows cluster between them and wanted to then use AlwaysOn availability group feature. This blog is about Primary Replica not being active.

Here is the text of the message (for search engines)

Joining database on secondary replica resulted in an error. (Microsoft.SqlServer.Management.HadrTasks)
——————————
ADDITIONAL INFORMATION:
Failed to join the database ‘SQLAuthority’ to the availability group ‘ProdAG’ on the availability replica ‘SRV2’. (Microsoft.SqlServer.Smo)
——————————
An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)
——————————
The connection to the primary replica is not active. The command cannot be processed. (Microsoft SQL Server, Error: 35250)
——————————

We get the same error even if we try to run JOIN command via T-SQL.

Msg 35250, Level 16, State 7, Line 1
The connection to the primary replica is not active. The command cannot be processed.

We checked ERRORLOG on both the replicas and found below the messages

Database Mirroring login attempt by user ‘DomainName\svc_sql.’ failed with error: ‘Connection handshake failed. The login ”DomainName\svc_sql’ does not have CONNECT permission on the endpoint. State 84.’. [CLIENT: 192.168.1.11]

SOLUTION/WORKAROUND

The error message is clear that the service account is not able to connect to SQL Server on database mirroring / AlwaysOn AG port. To fix the issue we need to run below command

GRANT CONNECT ON ENDPOINT::Hadr_endpoint TO [DomainName\svc_sql];

Once above was done, the issue was resolved and we were able to join the database to AG.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX: Msg 35250, Level 16, State 7 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

I must confess that consulting has taught me much more than what I have learned in my regular job. Freelancing comes with advantages and disadvantages. For me, I am looking at advantages and I am happy with my decision. Let us see how to fix an error related to Login Failed and Deleting Collection.

Whenever there is an upgrade of SQL version, it is expected that the server would go to the script upgrade mode and would finish it as a part of database recovery. There are many situations where I have seen server getting stuck to script upgrade mode. This is the blog which I have written long ago to troubleshoot such issues.

SQL SERVER – Login Failed For User – Reason Server is in Script Upgrade Mode

One of the client followed above article and shared information about their SQL script upgrade mode.

2017-05-16 01:29:18.66 spid7s Creating [dbo].[fn_sysutility_mi_get_batch_manifest] function
2017-05-16 01:29:18.78 spid7s Deleting collection set “Utility Information”…
2017-05-16 01:29:42.21 Logon Error: 18401, Severity: 14, State: 1.
2017-05-16 01:29:42.21 Logon Login failed for user ‘India\SQLSvc’. Reason: Server is in script upgrade mode. Only administrator can connect at this time. [CLIENT: ]

I have asked to share information about below tables as they were found in the script

use msdb
go
select COUNT(*) from sysssislog(nolock)
go
select COUNT(*) from syscollector_execution_log_internal(nolock)
go

There were having a huge number of records 400K+

WORKAROUND/SOLUTION

To allow connections to SQL, we need to start SQL services with trace flag 902. Here are the steps which helped my client.

• Keep the SQL Server Service running with the T902 startup parameter
• Backup MSDB database and remove the data of the above two tables.
• Remove the -T902 parameter.
• Restart the SQL Server service to allow the upgrade script to finish.

I was not able to spend much time to identify why those tables had a huge amount of records. It may be due to a lot of SSIS packages? May be due to excessive logging? Since the issue was resolved by manually cleaning the tables, they never wanted to spend more time and money to find the cause of the error.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Login Failed – Server is in Script Upgrade Mode – Deleting Collection Set “Utility Information”

A few months ago, I wrote a blog which tells us reasons about why we should use tools provided with the product rather than generic tools. Yeah, I am talking about services.msc vs. SQL Server Configuration manager.

Why to Use SQL Server Configuration Manager Over Services applet (services.msc)? – Interview Question of the Week #112

Once of the blog reader reported below error.

SQL Server Configuration Manager
—————————
Cannot connect to WMI provider. You do not have permission or the server is unreachable. Note that you can only manage SQL Server 2005 and later servers with SQL Server Configuration Manager.
Invalid class [0x80041010]

I contacted him via email provided and he also shared the image of the message.

While searching on the internet, there were various blogs talking about below solution.

1. Find the *.mof file for SQL Server 100 (SQL2008) or 110 (SQL2012) or 120 (SQL2014) or 130 (SQL2016)
2. Use command “mofcomp” to compile the mof file.

We tried that and it didn’t succeed.

An error occurred while processing item 10 defined on lines 73 – 79 in file C:\Program Files (x86)\Microsoft SQL Server\110\Shared\sqlmgmproviderxpsp2up.mof:
Error Number: 0x80070005, Facility: Win32
Description: Access is denied.
Compiler returned error 0x80070005

There are two important things in above failure. Error message – “Access is denied” and the line number. If you get error for 4 – 7 then most likely you have not run CMD using “Run As Administrator” but if you get error for some other line (like we got for 73-79) then the problem would be something else.

SOLUTION/WORKAROUND

Since it was an access denied error, I downloaded and run a process monitor and looked for access denied. We found below interesting key related to WMI.

svchost.exe 1023 RegOpenKey HKCR\CLSID\{73E709EA-5D93-4B2E-BBB0-99B7938DA9E4}\LocalServer32 ACCESS DENIED

Now, we know the key which is showing access denied.

We checked permission and found that Administrator has full permission on the above key. But access denied was for Network Service account so we added Network service on the above key and then ran the above command (MOFCOMP) again. Since permissions were fixed, we could run the above command successfully. After that SQL Server Configuration Manager just opened fine and we were able to manage the SQL related services successfully.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Configuration Manager – Cannot Connect to WMI Provider. You Do Not Have Permission or The Server is Unreachable

Whenever you are into a business of “fixing the issue” – you should always find in error message first. The whole approach of solving the issue changes if we go behind the wrong error message. Let us learn about the error Backup Failure.

Once of my client told that their backups are failing randomly. If you are a DBA, you would agree with me that there can be many different reasons of backup failure, mostly permission or space issue. So, I asked them to send me email with exact issue and the error message from a SQL Server ErrorLog file. So now, they reported that – Our backups are “randomly” failing to backup up a database to a file share will with below errors in ERRORLOG file.

2017-05-11 11:36:23.10 spid109 Error: 18204, Severity: 16, State: 1.
2017-05-11 11:36:23.10 spid109 BackupDiskFile::CreateMedia: Backup device ‘\\ozone-fs\Backups\DBName.bak’ failed to create. Operating system error 1240(The account is not authorized to log in from this station.).
2017-05-11 11:36:23.10 Backup Error: 3041, Severity: 16, State: 1.
2017-05-11 11:36:23.10 Backup BACKUP failed to complete the command BACKUP DATABASE DBName. Check the backup application log for detailed messages.

The first observation from error message is that it’s a share not a local drive. I was curious to know what kind of share it is? They informed that this is Scale-Out File Server (SOFS) configuration with CA (Continuously Available) shares. When I searched for the error message, it was talking about SMB and some registry level change, but had nothing to do with CA shares.

I had no idea about this feature so I did some research about this feature and found something interesting. Read the article here.

My thoughts:

Based on my research and learning, for the SQL backups, we should not use Scale-Out File Server (SOFS) configuration with CA (Continuously Available) shares, in favor of General Use File Server Cluster. Here is the reference.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER- Error: 18204, Severity: 16 – Backup Failure – Operating System Error 1240 (The Account is not Authorized to Log in From This Station)

There are many times when internet has answers to many questions. To get the right answer, you need to search for the right keyword and it comes from practice. Let us learn how we can fix the error Unable to Change “Shared Feature Directory” Under Feature Selection.

One of my blog readers pinged on Skype and told that they need quick assistance as they are unable to proceed with SQL Server installation. They were trying to install SQL Server 2014, but they were unable to change the shared feature installation directory in the setup.

As we can see above that both the textbox and browse button were disabled and it was stopping them to choose the desired path.

WORKAROUND/SOLUTION

I was able to find the cause for them. As per my search on the internet, if there are any shared components installed on a server in the default path, then we will not be able to change the default shared future path. It is not just a SQL version which we are installing. Even if we have any other instance shared directory installed in other drives. We will not be able to change the path during the next installation. So, what can be done? Well, if you really want to install the shared components in some other drives, then the currently installed drive, then we need to uninstall currently installed instances and have to install the new instance in different drives.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Unable to Change “Shared Feature Directory” Under Feature Selection

One of my client was running into the below error running xp_logininfo on some of their users.  It was interesting that when they run the script to capture user information it works for the same user in another domain. They asked me if there is anything specific they can look for to find out the difference between the two users in the different domains?

Msg 15404, Level 16, State 4, Server BRSQL3, Procedure xp_logininfo, Line 43
Could not obtain information about Windows NT group/user ‘INDIA\SQLGroup’, error code 0x2147.
Msg 15404, Level 16, State 4, Server VISQL3, Procedure xp_logininfo, Line 43
Could not obtain information about Windows NT group/user ‘EMEA\SQLGroup’, error code 0x2147.

I converted hex code 0x2147 to decimal (using the calculator as below)

And it was error number 8519 which means “A global group cannot have a cross-domain member.” As per the message, it seems like something to do with the type of group they were trying to add.

The error message helped them and they were able to find what was mentioned in the error message.

ANOTHER VARIATION

Below is another flavor of the error which you might see.

Msg 15404, Level 16, State 19, Procedure xp_logininfo, Line 62
Could not obtain information about Windows NT group/user ‘ASIA\SQL_svc’, error code 0x5.

The above error is generally reported in SQL Agent job failure. Notice that error code is 0x5 which is in hexadecimal. This is equivalent to decimal 5 = Access is denied.

TIP AND TRICK

You need not remember the message text. If you now any message ID for windows error, you can convert it to text using net helpmsg command.

The above can be fixed by changing the owner in job properties to ‘SA’ and it should fix 0x5 error.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – xp_logininfo – Msg 15404: Could not obtain information about Windows NT group/user, error code 0x2147

While I work on performance tuning most of the time, I do provide consultancy on issues which can be resolved quickly. These are the engagement where I get a lot of content for my blogs as well. Let us learn about the error related to redo log record.

One of my clients contacted me when they were not able to start their SQL Service. Without wasting any time, I got engaged with them and ask to share Errorlog to see the cause. Below is the error which I could see from the ERRORLOG.

2017-05-22 04:22:55.97 spid7s Starting up database ‘master’.
2017-05-22 04:22:55.98 spid7s Error: 3456, Severity: 21, State: 1.
2017-05-22 04:22:55.98 spid7s Could not redo log record (2579:456:5), for transaction ID (0:151685), on page (1:375), database ‘master’ (database ID 1). Page: LSN = (2579:424:5), type = 1. Log: OpCode = 4, context 2, PrevPageLSN: (2579:368:3). Restore from a backup of the database, or repair the database.

Since the master database was having corruption, there were not many choices left. The error message clearly says what we need to do. The message generally comes when there is a corruption in the transaction log file of the database. In our case, it was master database, which is one of the system database and due to that SQL was not getting started.

WORKAROUND/SOLUTION

To come out of the situation, below are the options.

• Restore from backup
• Rebuild System DBs
• Reinstall SQL Server

If we choose to restore from backup, then first we need to rebuild system databases from setup.exe and then perform disaster recovery procedures. Below is the command which can be used to rebuild system databases. We have put instance name as MSSQLServer because it was a default instance. For named instance, we need to use instance name. Make sure to take backup of existing MDF and LDF at a safe location before rebuilding.

setup /ACTION=REBUILDDATABASE /QUIET /INSTANCENAME=MSSQLSERVER /SQLSYSADMINACCOUNTS=”Domain\LoginName” /SAPWD=P@ssw0rd

It completed successfully and SQL was up and running. Now, this SQL server was as fresh as a new installation with no information about the earlier configuration of SQL Server. My client was having a hard luck as they were not having a backup of any system databases. All they had was MDF and LDF file of their user database. We tried to attach the user database but got the below error.

CREATE FILE encountered operating system error 5(Access is denied.) while attempting to open or create the physical file ‘S:\MSSQL2016\MSSQL14.MSSQLSERVER\MSSQL\Data\AX_PROD.mdf’. (Microsoft SQL Server, Error: 5123)

Then we went to the file properties and saw that this file was orphaned file as there was no owner. We provided SQL Startup account as the owner of this file. Once permission was set correctly, we were able to attach the database files and bring user database online.

Later, we stopped SQL Service and replaced database files for MSDB and model database and started SQL again. This time they were lucky and MSDB was clean so all information in MSDB came back. They were able to see the jobs etc. There were not many logins in the master database so they were able to recreate them and application was back online.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – FIX : Error: 3456, Severity: 21 – Could not redo log record (xx:xx:xx), for transaction ID (x:xx), on page (x:xx), database ‘master’

There are some weird situations which I see in SQL Server. This is also a one of the surprising situation. One of my clients reported via below screen shot. Let us see the issue of Master Database in Single User Mode.

Interesting enough, there was no issue with SQL functionality. It was acting just normally which means that we were able to make connections to SQL Server without any issues. They said that there are no users complains and users were able to get multiple connections to the Master as well.

We tried to switch it back to multi_user both in T-SQL and GUI, we get the same error:

Msg 5058, Level 16, State 5, Line 1
Option ‘MULTI_USER’ cannot be set in database ‘master’.

Since, it was interesting, I asked to check sp_helpdb for master and check the same in sys.database catalog views.

Both of the above have conflicting information. While this is an unexpected behavior, how do we come out of this situation?

WORKAROUND/SOLUTION

Here is the solution I offered them.

• Take the backup of master database using the backup database command.
• Restored the backup taken in the previous step as a user database. Let’s say master_one.
• The use ALTER command and get master_one in multiuser mode.
• Now take the backup of that user database (master_one) and then restored it in master using the below command. To restore the master database, we need to start SQL in single user mode.

Restore database master from disk= 'C:\Temp\Master_one.bak' with replace

Once they followed the above steps, the master database was back to multi user mode. My client was not sure how they ended up in such situation.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Master Database in Single User Mode – A Serious Issue

Along with performance tuning, I do provide consultancy services for deployments, including AlwaysOn availability group deployments. One of my clients who had a multi GB database wanted to configure AlwaysOn availability group and add a replica for disaster recovery purpose as well. They already had a secondary replica in the same data center for high availability purpose. Let us learn about AlwaysOn Join Error.

Since its not advisable to use UI for such big database to initialize the secondary, they were initializing secondary by will manual backup / restore of backup from primary followed by “join” option in the wizard. Whenever they tried this, below was the error.

Msg 1408, Level 16, State 211
The remote copy of database “SQLAUTH” is not recovered far enough to enable database mirroring or to join it to the availability group. You need to apply missing log records to the remote database by restoring the current log backups from the principal/primary database.

They contacted me because they were sure that there are no backups taken on primary replica, even then, somehow, they were seeing the error 1408 while trying to join the database to AG. They knew that the error is reported when there is a log backup taken from primary replica and there is a mismatch of LSN between the restored copy of secondary and current copy on the primary.

WORKAROUND/SOLUTION

While they knew that the error is due to the LSN mismatch, they missed one of the core concepts of backup taken from the AlwaysOn availability database.

It was easy to explain and demonstrate that they error would appear when a log backup is taken, after the full back-up from primary, even on a secondary replica. I used my below blog to find the history of the backup of the database.

When we ran the same query on the current secondary replica (remember, they already had one secondary) we found that log backups are taken regularly on that replica. As soon as we restored all the pending log backups to new secondary and tried join option again – it worked!

So whenever you see such errors, always use the script from my blog and find out the log backups which are taken. These log backups can happen even on secondary replica and would still maintain the chain with another log backup on other replicas.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – AlwaysOn Join Error – Msg 1408, Level 16 – The Remote Copy of Database “SQLAUTH” is Not Recovered Far Enough to Enable Database Mirroring or to Join it to the Availability Group

In SQL Server 2012 onwards, the cluster health check detection logic has been enhanced. Instead of the traditional pull mechanism of the cluster (IsAlive and LooksAlive), SQL Server (version 2012 onwards) uses a push mechanism to detect the health of the SQL instance. This is done by special stored procedure called sp_server_diagnostics. We should remember that the failover mechanism for the AlwaysOn FCI and the AlwaysOn Availability Groups is same.

While troubleshooting, it is very important to know which log to look at along with the basics of a feature. Recently, one of my clients was having trouble in bringing SQL Server AlwaysOn availability group resource online.

Whenever I am stuck with a cluster related issue, I always look at cluster log. You can refer my previous blog about how to generate cluster logs. SQL SERVER – Steps to Generate Windows Cluster Log?

Now let us change see the error logs.

Now let us inspect them heavily and you will find following error.

The User Does Not Have Permission to Perform this Action. (297)

I have removed date-time column from the output to provide the clarity. If we look at series of messages, you would notice that cluster has made a connection to SQL Server. After this it executes below statement

exec sp_server_diagnostics 10

As we can see in the next line that this execution failed with error.

The user does not have permission to perform this action

Due to above error diagnostic health check failed and SQL Server will not be able to come online in a cluster. Same error can also appear in the AlwaysOn availability group as well.

Now the real question is which user? And how to fix this issue?

WORKAROUND/SOLIUTION

The account which is used to connect to SQL Server from a cluster is a local system account. My client informed that due to hardening they have modified default permissions in SQL Server.

To fix this issue, we can add VIEW SERVER STATE permission to the SYSTEM account.

use [master]
GO
GRANT VIEW SERVER STATE TO [NT AUTHORITY\SYSTEM]
GO

Once done, the issue was resolved and SQL came online in the cluster as well.

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – sp_server_diagnostics – The User Does Not Have Permission to Perform this Action. (297)

One of my client was unable to install SQL Server 2014 on a Windows 2012 Cluster. They were using mount points with below the structure. Let us learn about installation error.

• M:\Data_MP
• M:\Log_MP
• M:\System_MP
• M:\Temp_MP
• M:\Backup_MP

Over here, M was 10GB local drive, Data_MP is a SAN disk mounted in the root drive. As we can see they have configured disks from the SAN as mount points. The validation is getting successful, but the SQL installation is failing with below error.

Error description: The resource ‘Cluster Disk 1’ could not be moved from cluster group ‘Available Storage’ to cluster group ‘SQL Server (MSQLSERVER)’. Error: There was a failure to call cluster code from a provider. Exception message: Generic failure . Status code: 183. Description: Cannot create a file when that file already exists

When I looked at the Detail.txt file for more details, the message is same.

WORKAROUND/SOLUTION

There are two workarounds which I found for them

1. Pre-create a Windows Cluster group/Role in Cluster and add the disks to that group before starting the setup. During the setup wizard, choose this group for SQL to be installed.
2. Mount the mount points in a folder instead of Root drive like below
   • M:\UserDB_Data\Data_MP
   • M:\UserDB_Log\Log_MP
   • M:\SystemDB\System_MP
   • M:\TempDB\Temp_MP
   • M:\Backup\Backup_MP

Over here, M was 10GB local drive, UserDB_Data is a folder inside M and Data_MP is SAN disk mounted inside folder.

Hope this would help someone in the world!

Reference: Pinal Dave (http://blog.SQLAuthority.com)

First appeared on SQL SERVER – Installation Error – Status Code: 183 Description: Cannot Create a File When that File Already Exists

In the month of January, I had offered a unique offering for training – “SQL Server Performance Tuning Practical Workshop“. I was initially planning to offer this for one month, however the response for this training was so much that I had to extend it for the entire first quarter. At this point of time, I am still delivering 2-3 classes every week as the training is extremely popular.

After delivering over 120+ training workshop (in just a short period of 5 months) for SQL Server Performance Tuning Practical Workshop, here are a few numbers at the glance.

Numbers at Glance

Let us see some of the numbers related to this workshop delivered. Please note that all the details are from January 1, 2017.

Workshop Delivered: 124
Distinct Organization: 117 (Some organizations wanted me to deliver this session, multiple times for different teams)
Attendees Attended: 787 (Average 6.34 per session)
Individual Personalized Training: 18 (This was 1:1 training)
SQL Server Version Breakdown: SQL Server 2008/R2 (14%), SQL Server 2012 (54%), SQL Server 2014 (22%), SQL Server 2016 (2%), Misc (8%)
Training Platform: GotoMeeting (98%), WebEX (2%)
Most Preferred Training Time: 8 AM to 12 PM Eastern Time

This is one of the best services available which gives sharp learning experience at the amazing cost. I have spent countless hours with many different organizations delivering this training and realized that every training is getting better as I go.

Good News

After braining storming with many different organizations, here is the good news.

I am going to continue offering SQL Server Performance Tuning Practical Workshop for the rest of the year 2017.

Every month, I am going to set aside 8 days when I will deliver this training online to different organizations. As the training takes so much of the time, I am not able to focus on other important tasks like building more courses or focusing on my the most sought out service of Comprehensive Database Performance Health Check.

Not So Good News

Well, every good news has another side. Here is the unpleasant news for many who look for shortcuts to their performance issues. I am going to discontinue Index Tuning and Strategy Guidance (99 minutes) from June 30. This service was very popular among the organization who wanted a shortcut to their performance problems. From July 1st, this service will be an integral part of Comprehensive Database Performance Health Check.

I will be no longer offering Index Tuning and Strategy Guidance as a stand alone service.

Last Chance – If you want, you can avail this indexing service at a discounted rate of USD 999 before June 30. (Just send me email at pinal@sqlauthority.com and we will get started)

So now we know that SQL Server Performance Tuning Practical Workshop is going to be there for the rest of the year, let us see some frequently asked questions.

How to sign up for SQL Server Performance Tuning Practical Workshop?

This workshop is more suitable for organizations or group of people. Do reach out to me with following details and in response to your email. I will provide you my earliest availability, fees and mode of payment.

1) Name of your organizations
2) Available time slot of 4 hours (date and country residing)
3) Number of attendees
4) Number of SQL Server Instances in your organizations

Is this training Online or Offline?

This training is Online training. Once we agree on time and payment details. I will send you GotoMeeting invite. You can share this meeting invite upto total of 8 members in your team. Everyone can learn together for one flat rate fee.

Is it like webinar where there will be attendees from other organizations?

No. This is personalized training for you and your organizations. It is not a generic training. You will have all the members in the training from your organizations only.

How long is the training?

3 to 4 Hours. The training is very well crafted from my experience of performance tuning consulting of lasting many years. I have learned that there are few powerful tricks which can improve your SQL Server performance many folds instantly. I share my secrets during this webinar.

I, personally, do not like to call it training, I think training is not a good word for what we do. I think it is more like Business Secrets Sharing. In this training, I explain and share all the tricks, scripts and secrets which I practice at my customer place and generate money for me.

How many people will be there in the training?

You should know the answer of this one. As this is going to be personalized training for your organization, you will know the number of attendees. I like to limit it to maximum 8 people, because from experience I have found that is the ideal number of the training.

I am an individual, can I join your existing group or scheduled webinar?

Yes and No.

First of Yes – You can for sure join the training as an individual. However, in that case the training will be exclusively for you. In that case, I will be happy to share all my performance tuning secrets with you by giving you 100% personalized attention. We will move at a much faster pace and cover a lot more.

Now about No – There is no generic class or scheduled session. My experience says, this training is more effective and powerful when we learn it 1:1.

What is the fee for this workshop?

Do reach out to me with following details and in response to your email. I will provide you my earliest availability, fees and mode of payment.

1) Name of your organizations
2) Available time slot of 4 hours (date and country residing)
3) Number of attendees
4) Number of SQL Server Instances in your organizations

Please note that fees for the training session is fixed rate and you can have maximum 8 people in one training. In other words – 8 individuals can learn for the price of 1.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL Server Performance Tuning Practical Workshop – Aftermath