Ask any DBA what would they love to monitor in their SQL Server and I promise you that you will get quite many different answers to this question. Though every DBA would love to see their server always up upon asked what exactly they would like to watch you will see different answers to the popular question. I have previously blogged about it over here: Solve 3 Biggest SQL Server Performance Challenges with Spotlight. Today, you can download Spotlight and Win Apple Watch Series 4.

You’ve got more instances and databases than one DBA should have to juggle. And with all the complexity and growth of your modern data center? Man, do you have your hands full!

But what if you could view the health of your entire database environment – all on a single screen, from your desk or your mobile device? Imagine the freedom! Imagine how easily you could improve database performance! Ready?

Get your free Spotlight trial to:

• Improve SQL Server performance proactively to avoid dropping the ball on your growing responsibilities.
• Gain deep visibility into all aspects of your database environment.
• Diagnose real-time and historical issues quickly.
• Achieve high availability.

Download Spotlight Free to Win Apple Watch Series 4. The winner will be announced on February 4.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on Win Apple Watch Series 4 and Solve SQL Server Performance Problems

Once there was an urgent request from my client and they asked my assistance. Without wasting any time, I sent them GoToMeeting invite and started working. In this blog we would learn how to fix FCB:: Open failed for TempDB database.

As soon as I joined the call with them, they explained that they were not able to start SQL Service. The first step in such situation is to look at SQL Server ERRORLOG. SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

Here are the messages at the end of the log file. I have added line number and done some more formatting for clarity.

1. Clearing a tempdb database.
2. Error: 5123, Severity: 16, State: 1.
3. CREATE FILE encountered operating system error 3(The system cannot find the path specified.) while attempting to open or create the physical file ‘T:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Data\tempdb.mdf’.
4. Error: 17204, Severity: 16, State: 1.
5. FCB:: Open failed: Could not open file T:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Data\tempdb.mdf for file number 1. OS error: 3(The system cannot find the path specified.).
6. Error: 5120, Severity: 16, State: 101.
7. Unable to open the physical file “T:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Data\tempdb.mdf”. Operating system error 3: “3(The system cannot find the path specified.)”.
8. Error: 1802, Severity: 16, State: 4.
9. CREATE DATABASE failed. Some file names listed could not be created. Check related errors.
10. Could not create tempdb. You may not have enough disk space available. Free additional disk space by deleting other files on the tempdb drive and then restart SQL Server. Check for additional errors in the event log that may indicate why the tempdb files could not be initialized.
11. SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.

Here are my earlier blogs with “similar” errors.

SQL SERVER – FIX: Msg 5123, Level 16 – CREATE FILE Encountered Operating System Error 5

SQL SERVER – FIX: CREATE FILE Encountered Operating System Error 5 (Access is Denied.)

In an earlier blog, it was just a permission issue which can be fixed, and SQL can be started. But, in the current situation, my client lost T drive and there is no way to get it back. So, we need to ALTER the TempDB configuration and point to an existing location. You can see the problem now, to ALTER the path, we need to start SQL Server, but SQL is not staring because the path is incorrect. Chicken and Egg situation!

SOLUTION/WORKAROUND

We need to start SQL Server with a minimal configuration using startup parameter f.  Here is what I generally do in such situations.

1. Start SQL Service via command prompt using NET START command. We also need to provide parameter “f” and “mSQLCMD” . This will make sure that other than SQLCMD, no application is able to connect to SQL Server. This is to avoid errors like below.

Login failed for user. Reason: Server is in single user mode. Only one administrator can connect at this time.

NET START MSSQLSERVER /f /mSQLCMD

Make sure SQLCMD is in all upper case and “f” and “m” are lowercase. You need to use the correct service name. MSSQLSERVER is for the default instance. For named instance, it would be MSSQL$INSTANCENAME

Once we start it, below is the message in SQL Server ERRORLOG.

Warning: The server instance was started using minimal configuration startup option (-f). Starting an instance of SQL Server with minimal configuration places the server in single-user mode automatically.  After the server has been started with minimal configuration, you should change the appropriate server option value or values, stop, and then restart the server.

Above is perfect!

2. Now, we would connect SQL using SQLCMD and run ALTER DATABASE for tempdb to fix the incorrect path. From the command prompt, we need to run below.

SQLCMD -S <ServerName> -E

Here are the ALTER commands which you need to modify based on your environment.

ALTER DATABASE tempdb Modify FILE (Name = 'tempdev',FileName = 'C:\tempDB\tempdb.mdf')

ALTER DATABASE tempdb Modify FILE (Name = 'templog',FileName = 'C:\tempDB\templog.ldf')

Here is the screenshot for my server.

3. After this, we need to stop SQL Services and start normally.

Have you been into such a situation and found some other solution to this error? Please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FCB::Open Failed – TEMPDB Files Fail to be Created with Error: “CREATE FILE Encountered Operating System Error 3”

Ask any DBA what is their dream infrastructure and I promise you that they will narrate immediately a story which will be centered around no downtime and amazing performance. However, when you ask them if they know what exactly will make them happy as well as run their business most efficient, why are they not going for it? The usual answer is budget. If we ask DBA to presume there is an infinite budget the next answer is Good Tool. In this blog post, we will discuss how we can replicate data from Oracle to Kafka and efficiently and quickly.

What is Kafka?

Kafka is used in the real-time streaming data architectures to provide real-time analytics. It is a fast, scalable, robust, fault-tolerant messaging system. If you are familiar with RabbitMQ or AMQP, Kafka can be used as their replacement.

Kafka is known in the recent time for its reliability and higher throughput. In the real world, I have seen very high adoption of Kafka where there are stream processing, high-speed filtering, and pattern matching.

The primary reason for its popularity is easy to use along with it is free and open source.

From Oracle to Kafka – SharePlex

If you are familiar with the Oracle Eco-system, it is very clear that you need a connector when you have to use Oracle to Kafka. I have personally used quite a few connectors where I have to move my data. However, I realized that is not that easy to configure something which can move my data from one source to another source when it is about high transactions streaming data. After long research, I finally have figured out a good replication tool which not efficient but also provides a high performance – SharePlex.

SharePlex for Oracle is a very stable, robust, high-performance, high-availability technology that offers a low-cost alternative to other Oracle replication tools. In addition to the replication, it also provides data compare, repair, in-flight data integrity, in addition to monitoring and alerting.

Why SharePlex?

As I mentioned earlier, I have used many different products including GoldenGate to replicate data from Oracle. I particularly prefer SharePlex over GoldenGate because of many reasons. SharePlex supports Oracle Standard Edition (SE) as well as Enterprise Edition (EE).

Data Accuracy and Integrity – When replicating data across Oracle to Kafka it is very critical that data accuracy is maintained along with peer-to-peer monitoring and conflict detection & resolution.

Improved Performance – SharePlex offloads process-intensive reports outside the main production database server. This helps to improve the overall network performance without increasing resource consumption on the primary database.

Better Price Point – Cost is always a concern and when SharePlex is compared with another similar tool, it is quite inexpensive and also provides lots of value adds.

Next Action Point

If you need to replicate your data from Oracle to Kafka, I strongly suggest you try out SharePlex as it provides zero downtime and zero data loss. Try out SharePlex Now.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SharePlex – Efficient and High Performance Replication from Oracle to Kafka

I have helped many customers to solve complex issues in their environment by Comprehensive Database Performance Health Check. Sometimes, the issue looks very complex but once the solution is found it seems very easy. In this blog, we would learn about a situation where the listener is missing in SSMS but working fine in failover cluster manager (cluadmin.msc).

While doing checks of their database, they showed me an interesting situation. Here it goes.

THE SITUATION

My client had 2 nodes Always On Availability Group on SQL Server 2017 and Windows Server 2016. They noticed that;

1. In failover cluster manager, we are able to see Network Name resource for Listener.
2. In SQL Server Management Studio (SSMS), we were not able to see anything under “Availability Group Listener”. It was empty!
3. Below query also doesn’t show any listener in SQL. (0 rows affected)

SELECT *
FROM sys.availability_group_listeners
GO

SELECT *
FROM sys.availability_group_listener_ip_addresses
GO

4. We were able to connect to the Listener and it was working fine, even after failover also.

SOLUTION/WORKAROUND

When I asked them the history of the listener creation, they informed me that this was created by Windows Admin team. SQLDBA team couldn’t create listener due to an issue which I have written in my previous blog

SQL SERVER – AlwaysOn Listener Error – The WSFC Cluster Could Not Bring the Network Name Resource With DNS Name ‘DNS name’ Online

When I checked availability group resource in cluster manager, I found that it was not having any dependency on the listener. As soon as dependency was added (no downtime needed) we were able to see the listener in #2 and #3 above.

In short, if you are creating the listener via cluster manager, a dependency must be added to the AG resource in Windows Failover Cluster Manager to make the AG dependent upon the listener. If you create it via SQL Server (using SSMS, T-SQL or PowerShell) you should not face this issue.

Have you seen such a situation in your production? Check it now and fix it!

Reference: Pinal Dave (https://blog.sqlauthority.com) 

First appeared on SQL SERVER – Always On Availability Group Listener Missing in SSMS but Working Fine in Failover Cluster Manager

While helping a client with Azure environment, I came across an interesting error. I this blog I would explain cause and fix of error “Authentication failed for account sqlauthprembackup and the provided key. Please provide a valid key and try again.”

In SQL Server Management Studio, there is an option to connect to storage to look at the backups.

When I tried with one of my Azure storage accounts, it failed with below error:

Authentication failed for account sqlauthprembackup and the provided key. Please provide a valid key and try again.
The value for one of the HTTP headers is not in the correct format. (Microsoft.SqlServer.StorageClient)
The remote server returned an error: (400) Bad Request. (System)

Here is the screenshot of the error

WORKAROUND/SOLUTION

It turned out that storage account which I created was of “Premium” storage tier. As per documentation: (Refer Limitation section)

Backup to premium storage is not supported.

To verify you can refer properties as shown below.

I wish Microsoft should have provided some meaningful error rather than a cryptic error.

• The value for one of the HTTP headers is not in the correct format.
• The remote server returned an error: (400) Bad Request.

When I used an account of Standard tier, it worked fine, and I was able to connect to storage.

On a side note, when I used SQL Server Management Studio 18.1 (Preview), I can see that the UI is changed completely. In this version, we need to login to Azure using UI not via earlier ways of access keys.

Reference: Pinal Dave (https://blog.sqlauthority.com) 

First appeared on SQL SERVER – Fix: 400 – Bad Request Authentication Failed for Account and the Provided Key. Please Provide a Valid Key and Try Again

While installing SQL Server Management Studio 18.0 (Preview) on my lab server, I got an error. In this blog, we would learn about error 0x800f081e which you also might see while installing other software also.

Here is the screenshot of the error 0x800f081e while installing SSMS

All we can see from the screenshot is that .Net Framework 4.7.2 installation had failure and code 0x800f081e. I clicked on the option provided in the screen to look at the log file. After scrolling and looking for error, I found below lines.

• [0518:1DF0][2019-01-22T00:52:14]e000: BootstrapperEngineDataModel.OnError: Burn engine encountered error. PackageId: DotNet47, ErrorType: ExePackage, ErrorCode: -2146498530, Data: , ErrorMessage: , UIHint: 0
• [0518:1DF0][2019-01-22T00:52:14]i000: MainViewModel.AddFailedCondition: Error: Microsoft .NET Framework 4.7.2:
• [0518:1DF0][2019-01-22T00:52:14]e000: MainViewModel.OnPackageError: setup failed for packageId: DotNet47, package: Microsoft .NET Framework 4.7.2, errorCode: -2146498530, errorMessage:
• [079C:2250][2019-01-22T00:52:14]e000: Error 0x800f081e: Process returned error: 0x800f081e
• [079C:2250][2019-01-22T00:52:14]e000: Error 0x800f081e: Failed to execute EXE package.
• [0518:1DF0][2019-01-22T00:52:14]e000: Error 0x800f081e: Failed to configure per-machine EXE package.
• [0518:1DF0][2019-01-22T00:52:14]i000: MainViewModel.OnPackageAction: Install Completed for package Microsoft .NET Framework 4.7.2
• [0518:1DF0][2019-01-22T00:52:14]i319: Applied execute package: DotNet47, result: 0x800f081e, restart: None
• [0518:1DF0][2019-01-22T00:52:14]e000: Error 0x800f081e: Failed to execute EXE package.

I have added a bullet on each line to have better readability in the blog. In the same folder where I had SSMS setup logs, there was another file for .NET install also, which also shows the same error.

I searched further and found below this: Error 0x800F081E is CBS_E_NOT_APPLICABLE which means that the update can’t be installed on the current system. This happens when a minimal install requirement is missing, or the installed files are on a higher version.

After doing some looking through the internet, I found that simply updating windows to the most recent version would automatically update .NET framework to the most recent version.

WORKAROUND/SOLUTION

For me, the error was fixed after I installed pending windows update (looks like some update of .NET 4.7.2 was pushed) and then did a reboot of the server. After this, the installation of SQL Server Management Studio (SSMS) went well.

Have you seen a similar error? What was the solution which worked for you?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Error 0x800f081e or ErrorCode: – 2146498530 While Installing SQL Server Management Studio (SSMS)

This was an interesting situation where I learned something new. In this blog we would learn about error – “A connection attempt failed because the connected party did not properly respond after a period of time.” while using “Azure Storage” option in SQL Server Management Studio.

As I wrote in my earlier blog, it is possible to connect to Storage Account using SQL Server Management Studio. Here is the link: SQL SERVER – Fix: 400 – Bad Request Authentication Failed for Account and the Provided Key. Please Provide a Valid Key and Try Again.

One of my blog readers sent an email with a different error.

Here is the text of the error message.

Unable to connect to the remote server (Microsoft.SqlServer.StorageClient)

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 52.239.220.32:443 (System)

In most of the situation, the above message comes due to the fact that the account which is opening SSMS is not able to reach to the storage account.

WORKAROUND/SOLUTION

There are few reasons which I could find.

1. If this is a Virtual Machine in Azure, then check if Internet “outbound” is blocked on the server. Actually, to reproduce the error I blocked it.
2. Check if there is any proxy server which is coming in between.
3. Check for any possible firewall which is blocking connection to the storage account.

I always prefer to use Test-NetConnection to test connectivity to specific IP and Port. You can also use PSPing to achieve the same.

Test-NetConnection -ComputerName 52.239.220.32 -Port 443

False in “TcpTestSucceeded” means that traffic couldn’t make it to the destination. So, you might need to engage network expert and figure out what’s going on.

When I checked with my blog reader, he informed that it was an on-premise server and they identified that certain accounts were able to use a proxy (and hence reach to storage) and few accounts were not using a proxy and hence their request was failing.

After opening the direct communication to Azure storage, the issue was resolved for them.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SSMS – Azure Storage Error – A Connection Attempt Failed Because the Connected Party did not Properly Respond After a Period of Time

Recently one of my old clients contacted me for an error execution via SQL Agent Job. They informed that it runs fine when they invoke the package from Visual Studio (SSDT). In this blog post, we will discuss Exception While Loading Script Task from XML.

Here is the error which was seen in SSISDB reports via management studio.

Script Task 1:Error: There was an exception while loading Script Task from XML: System.Exception: The Script Task “ST_24802bd6f41940e0befad660c2cba337” uses version 15.0 script that is not supported in this release of Integration Services. To run the package, use the Script Task to create a new VSTA script. In most cases, scripts are converted automatically to use a supported version, when you open a SQL Server Integration Services package in %SQL_PRODUCT_SHORT_NAME% Integration Services. at Microsoft.SqlServer.Dts.Tasks.ScriptTask.ScriptTask.LoadFromXML(XmlElement elemProj, IDTSInfoEvents events)

The error message is good enough in telling the cause. The script task version is 15.0 and SQL SSIS version is not compatible. Our target version was SQL Server 2016.

WORKAROUND/SOLUTION

When I checked various version, I found that the SSIS package was developed with Azure features packs on SSDT 2017. Since we are deploying it in SQL server 2016 and running via agent job, it is throwing an error.

To fix the problem, all we needed to do was change the TargetServerVersion values under project properties as shown below.

Have you encountered a similar error? Please comment and let me know if you found some other solution?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL Server Integration Services (SSIS) – There Was an Exception While Loading Script Task from XML

While doing Comprehensive Database Performance Health Check I always ask my client if there is any pain point which they have with the current state of the database/server. Once I got an interesting question which I am going to answer in this blog post – Why is my SQL Clustered Resource in Online Pending state for a long time before coming online.

Before I show you how I found the cause, here are few earlier blogs where the situation was different where SQL was not coming online at all.

• SQL SERVER – Unable to Bring SQL Cluster Online – Reason: The Account is Disabled
• SQL SERVER – Unable to Bring SQL Cluster Resource Online – Online Pending and then Failed
• SQL SERVER – Clustered SQL Resource Not Coming Online

When SQL is in Online Pending state, the SQL Service is not fully ready for connection or unable to make a connection. SQL SERVER – Steps to Generate Windows Cluster Log?

Here are a few important events:

1. Here is the offline event of Node1

Log Name: Microsoft-Windows-FailoverClustering/Operational
Source: Microsoft-Windows-FailoverClustering
Date: 1/28/2018 1:49:29 PM
Event ID: 1204
Task Category: Resource Control Manager
Level: Information
User: SYSTEM
Computer: NODE1.domain.com
Description: The Cluster service successfully brought the clustered service or application ‘SQL Server (MSSQLSERVER)’ offline.

2. Here is the online event on Node2

Log Name: Microsoft-Windows-FailoverClustering/Operational
Source: Microsoft-Windows-FailoverClustering
Date: 1/28/2018 1:57:11 PM
Event ID: 1201
Task Category: Resource Control Manager
Level: Information
User: SYSTEM
Computer: NODE2.domain.com
Description: The Cluster service successfully brought the clustered service or application ‘SQL Server (MSSQLSERVER)’ online.

If you observe closely, there is a gap of 8 minutes between above 2 events.

3. If we look at cluster logs, we found below messages.

• SQL Server <SQL Server (MSSQLSERVER)>: [sqsrvres] Service status checkpoint was changed from 0 to 1 (wait hint 20000). Pid is 2431
• SQL Server <SQL Server (MSSQLSERVER)>: [sqsrvres] Service status checkpoint was changed from 1 to 2 (wait hint 20000). Pid is 2431

.. number kept on increasing continuously. 2 to 3, 3 to 4 and so on. Finally, after around 40 attempts it came online. There was a gap of 2 seconds time in each line.

• SQL Server <SQL Server (MSSQLSERVER)>: [sqsrvres] Service is started. SQL Server pid is 2431
• SQL Server <SQL Server (MSSQLSERVER)>: [sqsrvres] Connect to SQL Server …
• SQL Server <SQL Server (MSSQLSERVER)>: [sqsrvres] The connection was established successfully
• SQL Server <SQL Server (MSSQLSERVER)>: [sqsrvres] Diagnostics is started
• SQL Server <SQL Server (MSSQLSERVER)>: [sqsrvres] Online worker helper is started
• SQL Server <SQL Server (MSSQLSERVER)>: [sqsrvres] SQL Server component ‘system’ health state has been changed from ” to ‘clean’
• SQL Server <SQL Server (MSSQLSERVER)>: [sqsrvres] SQL Server resource state is changed from ‘ClusterResourceOnlinePending’ to ‘ClusterResourceOnline’
• Resource SQL Server (MSSQLSERVER) has come online. RHS is about to report status change to RCM
• HandleMonitorReply: ONLINERESOURCE for ‘SQL Server (MSSQLSERVER)’, gen(0) result 0.
• TransitionToState(SQL Server (MSSQLSERVER)) OnlinePending–>Online.

WORKAROUND/SOLUTION

When we looked at ERRORLOG, I found recovery messages for 8 minutes. We also found out that there was a huge number of VLF which seems like the root cause of the issue.

We learned that after reducing the count of VLF, by taking log backups and shrinking the log file, we were able to resolve the issue and SQL failover was very quick.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – SQL Clustered Resource in Online Pending State for Long Time Before Coming Online

Sometimes there are some random unexpected errors and it’s always fun to find the cause of them. In this blog we would learn about an error 1051 (A stop control has been sent to a service that other running services are dependent on) which I received while stopping SQL Service in my lab server.

First, I must tell you that you might not see this error unless you do something unusual. In my case, I dropped some databases from SQL Server, which I thought were not needed, but they were having some purpose.

THE SITUATION

When I was trying to stop SQL Service, first I got below error:

Windows could not stop the SQL Server (MSSQLSERVER) service on Local Computer.
Error 1051: A stop control has been sent to a service that other running services are dependent on.

Here is the screenshot:

I saw that it didn’t stop so I did it again and this time a new message comes.

Followed by below message:

Windows could not stop the SQL Server (MSSQLSERVER) service on Local Computer.
Error 1061: The service cannot accept control messages at this time.

Have a look at below screenshot of the services which provided me some pointers.

SQL Service is having dependency with “PolyBase Data Movement” and “PolyBase Engine” which explains the cause of errors seen earlier. What next?

WORKAROUND/SOLUTION

The quick way to stop SQL Service at this point is to kill the dependent services.  Here was the path of the executable for the service.

“C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Binn\Polybase\mpdwsvc.exe” -dweng -Polybase

“C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Binn\Polybase\mpdwsvc.exe” -dms -Polybase

So, to come out of the situation, I killed below processes shown. They are the one which I found in the path above. Note both are the same exe so I killed both of them.

Once I killed the process, SQL Server was getting stopped and started successfully.

As I mentioned earlier, I messed up and dropped some databases from SQL Server without knowing about them. This was not my production server, so I was OK in uninstalling and reinstalling the feature. If this was a production server for you, look for backups of those databases and restore them.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Error 1051: A Stop Control Has Been Sent to a Service that Other Running Services are Dependent On

Few of my clients are still running older versions of SQL Server and they have their own reasons to not to move to the new SQL version. In this blog, we would learn about the fix of an error for which my client contacted me. My client was trying to apply the patch to a SQL Server 2008 R2 clustered instance which was running on Window Server 2012 and getting the error: Unable to retrieve the Cluster Service – Not clustered or the cluster service is up and online.

First, I asked to share Detail.txt to understand the exact error. Here is the information available. I have added a bullet point for clarity.

• Slp: Unable to retrieve the Cluster Service
• Slp: System.InvalidOperationException: The given key was not present in the dictionary. —> System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
• ThrowHelper.ThrowKeyNotFoundException()
• Collections.Generic.SortedList`2.get_Item(TKey key)
• SqlServer.Chainer.Infrastructure.ServiceContainer.GetService(Type serviceType)
• — End of inner exception stack trace —
• SqlServer.Chainer.Infrastructure.ServiceContainer.GetService(Type serviceType)
• SqlServer.Chainer.Infrastructure.ServiceContainer.GetService[T]()
• SqlServer.Chainer.Infrastructure.ServiceContainer.get_Cluster()
• SqlServer.Configuration.SetupExtension.RunDiscoveryAction.ExecuteAction(String actionId)

One of the causes of the above error is found earlier which I shared via below blog. SQL SERVER – Patch Rule Failure: Not Clustered or the Cluster Service is Up and Online

The blog above should be used when the cluster is in a bad shape. But in the current situation, my client had cluster working fine. It was just the SQL patch setup which was not able to talk to the cluster.

WORKAROUND/SOLUTION

I knew that there are articles from Microsoft which talks about but I was not getting any hit with the error which I saw in Detail.txt

Finally, we found that “Failover Cluster Automation Server” feature was not installed. This is default setting when install windows cluster service, but the feature is needed when installing the SQL Server 2008 R2 patch. Here is the PowerShell command to add the feature.

Add-WindowsFeature -Name RSAT-Clustering-AutomationServer

Finally, I found KB which talks about the solution.

Here is the snippet from the article.

To resolve this issue, do one of the following:

• Enable the Failover Cluster Automation Server feature on each node by using Server Manager. In Server Manager, expand Remote Server Administration Tools, expand Feature Administration Tools, expand Failover Clustering Tools, and then click to select Failover Cluster Automation Server.

Here is the PowerShell screenshot while installing it. I have used below command to view the feature.

Get-WindowsFeature RSAT-Cluster*

Hope this blog help in finding the solution of patch failure. Please comment and let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Patch Failure – Unable to Retrieve the Cluster Service

Once upon a time, I was trying to remove TempDB database additional files, but it was not working. In this blog I would explain the steps needed fix Msg 2555: Cannot move all contents of the file to other places to complete the emptyfile operation.

THE SITUATION

My client installed SQL Server using an automated method which kept tempdb database files on C Drive. As a good practice, he decided to add new files to T drive and remove files from C drive. While removing the files from C drive, they encountered below error.

Msg 2555, Level 16, State 1, Line 18
Cannot move all contents of file “tempdata1” to other places to complete the emptyfile operation.

The command which they were running was below.

USE [tempdb]
GO
DBCC SHRINKFILE (N'tempdb1', EMPTYFILE)
GO
ALTER DATABASE [tempdb] REMOVE FILE [tempdb1]
GO

When I double checked the messages window and I also found below.

DBCC SHRINKFILE: Page 3:39 could not be moved because it is a work table page.

Above is the main cause why SQL was not able to empty the file and remove it from TempDB database.

WORKAROUND/SOLUTION

Based on various other blogs on the internet, below T-SQL queries were tried and they DID NOT help.

DBCC DROPCLEANBUFFERS;
GO
DBCC FREESYSTEMCACHE ('ALL');
GO
DBCC FREEPROCCACHE;
GO
DBCC FREESESSIONCACHE;
GO

Note: Even restart of SQL Service was not helping us here and we were getting the same message for page 3:39

I remember writing a blog about the work table related error message for the tempdb database.

SQL SERVER – DBCC SHRINKFILE: Page 1:26423878 Could not be Moved Because it is a Work Table Page – TempDB Files

In short, starting SQL with a minimal configuration using “f” startup parameter allowed us to get rid of extra files. Have you faced such a situation? What solution worked for you?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Msg 2555: Cannot Move All Contents of File to Other Places to Complete the Emptyfile Operation

One of my existing clients has sent me an interesting question. In this blog, I would share one interesting situation where my client has created an availability group and it was visible in SSMS, but no role was visible in failover cluster manager.

Here is the picture which you explain what we were seeing.

Generally, when an availability group is created, an AG resource is created in a Windows cluster.

WORKAROUND/SOLUTION

The question which was asked by my client was perfectly valid. It is possible to run into the situation if we do below steps.

1. Enable feature of AlwaysOn availability group without creating windows cluster.
   
2. Create Availability Group and choose NONE as cluster type as there is no option called “NONE” as cluster type in below screen.
   
3. Now, enabled windows clustering feature and create windows cluster.

At this point, you can open cluster manager and would see no availability group role because availability group was created as Cluster Type None.

So, if you ever see such situation, check the Dashboard of Always On availability group and have a look at cluster type which is highlighted as below.

This feature is called as a Read Scale and you can read more about this at below link Use read-scale with Always On availability groups

Hope you found this information useful. Please comment and let me know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Availability Group Seen in SSMS but missing in Cluster Manager. What’s Wrong?

I am not sure what changes I made in one of my Virtual Machine, but I was running into an issue. In this blog, we would discuss how to fix error “The ‘SqlStudio Profile Package’ package did not load correctly” while launching Visual Studio.

I checked into the file mentioned but didn’t find anything interesting to tell me the solution.

WORKAROUND/SOLUTION

I search on the internet and found various solutions. In this blog, I would share the consolidated list of solutions.

1. Based on one of the StackOverFlow link, delete below file.

%UserProfile%\AppData\Local\Microsoft\VisualStudio\11.0\ComponentModelCache

2. Rename below folders

C:\Users\<UserName>\AppData\Roaming\Microsoft\VisualStudio\11.0

C:\Users\<UserName>\AppData\Local\Microsoft\VisualStudio\11.0

3. Download and install SSDT from below this link.
4. Go to ‘Add/Remove Programs’ in Control Panel. Select “Microsoft SQL Server 2012 Data-Tier App Framework”, right click and choose “Repair”. If you find more versions of it, do the repair for all of them.

For me, option # 1 worked. If you face the same problem, please comment and let me know which option works for you. If you find any other solution, please comment and share with others.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on Visual Studio – FIX: The ‘SqlStudio Profile Package’ Package Did Not Load Correctly

As they say – There are more ways than one to skin a cat. In this blog we will find another method to fix an error which you might see during SQL upgrades – Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.

Here are the messages in SQL Server ERRORLOG when the upgrade was unable to start SQL Service. I have added bullets to have the readability of messages in ERRORLOG.

• Taking SSISDB to single user mode
• Setting database option SINGLE_USER to ON for database ‘SSISDB’.
• ———————————
• Starting assembly upgrade
• ———————————
• Altering assembly [ISSERVER]
• Attempting to load the library ‘xpstar.dll’ into memory. This is an informational message only. No user action is required.
• Using ‘xpstar.dll’ version ‘2014.120.5556’ to execute extended stored procedure ‘xp_regread’. This is an informational message only; no user action is required.
• Error: 50000, Severity: 16, State: 127.
• Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.
• Error: 50000, Severity: 16, State: 127.
• Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.
• Creating function internal.is_valid_name
• Error: 6528, Severity: 16, State: 1.
• Assembly ‘ISSERVER’ was not found in the SQL catalog of database ‘SSISDB’.
• Error: 912, Severity: 21, State: 2.

I have written earlier blog having one of the solutions to fix the error. SQL SERVER – Assembly ‘ISSERVER’ Was Not Found in the SQL Catalog of Database ‘SSISDB’

In athe bove blog post, a situation was easy where SSIS catalog was not in use. However, for my current client asked not to drop catalogs as he was not sure who is using it.

WORKAROUND/SOLUTION

If we have another server with the same version and having SSISDB installed, then we can perform below steps to recreate the assembly. We need to make sure that the version and patch level of SQL Server is matching.

1. Start the problematic SQL with trace flag 902.
2. Connect to SQL instance with working SSISDB database
3. Expand SSISDB, Programmability, Assemblies
4. Highlight ISSERVER, Right-click and choose Script Assembly as CREATE to New Query Editor Window
   
5. Copy the CREATE ASSEMBLY statement and paste it into a query window on the problematic server
6. Execute the CREATE ASSEMBLY statement

Note that the text, which you would copy, is going to be very long as it’s the complete .NET code in binary format.

7. Once an assembly is created stop SQL and the start it without trace flag 902.

Is there any other way to create assembly ISSERVER? If I find it, I would share in another blog. If you find it, please comment and share with the SQL community.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Cannot Drop the Assembly ‘ISSERVER’, Because it Does not Exist or You do not Have Permission

One of my clients contacted me via my On Demand (50 Minutes) offering and they were seeing below error message in ERRORLOG – New parallel operation cannot be started due to too many parallel operations executing at this time.

Here is the complete message which was seen in the SQL Server ERRORLOG file.

New parallel operation cannot be started due to too many parallel operations executing at this time. Use the “max worker threads” configuration option to increase the number of allowable threads, or reduce the number of parallel operations running on the system.

I asked for complete ERRORLOG and Event Logs to look it from all possible angles.

• Error: 18210, Severity: 16, State: 1.
• BackupIoRequest::ReportIoError: write failure on backup device ‘{BD27F651-5DB3-4CB4-9615-9FDEC4D8EECE}331’. Operating system error 995(The I/O operation has been aborted because of either a thread exit or an application request.).

At the same time, I found below in event log

• SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1724. Thread=2300. Server. Instance=MSSQLSERVER. VD=Global\{FFC0C1C0-25D9-4A90-82B3-3ABBCFEA9476}247_SQLVDIMemoryName_0.
• BACKUP failed to complete the command BACKUP DATABASE ReportingService_Alerting. Check the backup application log for detailed messages.
• SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1724. Thread=9844. Server. Instance=MSSQLSERVER. VD=Global\{FFC0C1C0-25D9-4A90-82B3-3ABBCFEA9476}45_SQLVDIMemoryName_0.
• SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1724. Thread=8176. Server. Instance=MSSQLSERVER. VD=Global\{FFC0C1C0-25D9-4A90-82B3-3ABBCFEA9476}264_SQLVDIMemoryName_0.

From above it was clear that they are not taking native backups of SQL Server database using a maintenance plan of T-SQL. The backups are being taken using SQLVDI via 3^rd party software.

WORKAROUND/SOLUTION

Looking at the messages, it sounded like the error appears at the same time when a backup is running. Further investigation showed that they had more than 1000 databases and they were getting backed up via 3^rd party tool at the same time. I suggested them to

1. Talk to back up team and find a way to stagger the backups. This would not cause too many parallel threads for backups at the same time.
2. Increase the number of worker thread to avoid message in ERRORLOG.

Once they reduce many concurrent backups, error stopped.

Have you seen such messages earlier? What did you do to solve? Please share via comments with others in the SQL community.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – New Parallel Operation Cannot be Started Due to Too Many Parallel Operations Executing at this Time

One of the interesting features of PostgreSQL database is the ability to handle Unicode characters. In SQL Server, to store non-English characters, we need to use NVARCHAR or NCAHR data type. In PostgreSQL, the varchar data type itself will store both English and non-English characters.

Let us explore this with the following example. Create a table in PostgreSQL database as shown below

CREATE TABLE TESTING1(COL VARCHAR(100))

INSERT INTO TESTING1 (COL)
SELECT 'ЯНВАРЬ'

The above example adds a string of Russian characters (It means January in English)

Now let us SELECT the column value and see what it returns

SELECT COL FROM TESTING1

The result is

As you see, with just varchar data type, the PostgreSQL engine is able to store multi-language characters without any additional approaches

Also, read this post to know how to add multi-language characters in the SQL Server Engine SQL SERVER – Storing Data in Hindi, Chinese, Gujarati, and Tamil (or any other) Language in the Same Table

Let me know if any other Database Engine has this type of feature, I will be happy to learn from you and post on this blog with due credit to you.

Here are few of the blog post I wrote earlier about PostgreSQL.

• PostgreSQL – Learn Online in a Single Day – PostgreSQL Learning Path
• POSTGRESQL – How to Create a Function? – How to Declare Local Variable
• SQL – Difference between != and <> Operator used for NOT EQUAL TO Operation

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on PostgreSQL – Storing Unicode Characters is Easy

Having a scattered Microsoft SQL Server database platform is cumbersome to maintain, as you continue to accumulate waste capacity to your database servers. Therefore, you should consider consolidating your platform to make it more manageable. Let us learn about Why is SQL Server Consolidation Better Than Having a Scattered Environment?

It is good to understand the numerous benefits of proper SQL Server consolidation. It is not only a question about gaining cost savings but also a question of practicality. Several things will make your data platform administration smoother and more straightforward.

1. There is less administration work in SQL Server, Windows, and hardware.

It is obvious that when reducing the number of physical hosts for example from 100 to 10, there is a whole lot less administration work and patching in the SQL Servers and in Windows. This is especially true when you have a High Availability (HA) solution because the downtimes are minimal. When there are lesser host machines to look after, it is a lot easier to manage the hardware.

2. It becomes easier to monitor the whole infrastructure.

This is a no brainer. Having a lot less physical and virtual servers to monitor means a lot less work on following the server level resource usage and baselines.

3. Significant savings in Windows and SQL Server licensing and hardware costs.

I have come across a software solution that will ease the consolidation planning dramatically. The automation behind it reduces the workload a lot making the planning process also faster. The intelligent machine learning algorithms and benchmark data in the software make the planning a lot more accurate than manual methods can offer. This software is called SQL Governor.

The team behind SQL Governor shares that so far their customers have achieved on average 44% costs savings in SQL Server licenses. This can translate to savings worth even millions if you have a bigger data platform environment.

The team SQL Governor explains that when your data platform is intelligently consolidated, the overall usage of CPU and RAM is optimized. There are less sleeping CPU cycles in the system and the overall RAM usage is in better control. Thus, there is a need for less hardware.

Also, with SQL Governor you can gain a better understanding of the storage capacity needs. When planning the consolidation of the data platform, the workload forecasts can be followed on an individual database, instance, and server level. This helps to understand the capacity used as a whole and makes it easier to select suitable storage for each use case.

4. You can gain savings in electricity and rack space.

In medium-sized to bigger data centers, you can achieve major savings in both electricity consumption and rack space by consolidating your servers. This is especially important when it comes to environmental thinking and the carbon footprint of your data center.

I have written about the SQL Governor solution in my previous blog article, in addition to the benefits that capacity planning can offer to SQL Server performance. You can also look into the consolidation guide team SQL Governor have put together to help you in your consolidation planning.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on Why is SQL Server Consolidation Better Than Having a Scattered Environment?

A few days back, I published the blog on the same error message and one of the blog reader responded and ask how can I generate the script if I don’t have another SQL Server? In this blog, we would learn how to fix error Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.

Here is the earlier blog for your reference which talks about one solution to fix the error.

SQL SERVER – Cannot Drop the Assembly ‘ISSERVER’, Because it Does not Exist or You do not Have Permission

SQL ERRORLOG shows this:

Setting database option SINGLE_USER to ON for database ‘SSISDB’.
———————————-
Starting assembly upgrade
———————————-
Altering assembly [ISSERVER]
Error: 50000, Severity: 16, State: 127.
Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.
Error: 50000, Severity: 16, State: 127.
Cannot drop the assembly ‘ISSERVER’, because it does not exist or you do not have permission.
Creating function internal.is_valid_name
Error: 6528, Severity: 16, State: 1.
Assembly ‘ISSERVER’ was not found in the SQL catalog of database ‘SSISDB’.

I mentioned in an earlier blog to script the assembly from some other server and create it on a problematic server. One of my blog readers said –

I don’t have any other instance of SQL Server with the same version. Can I not use some other method?”

This was indeed a very valid ask. I spent some time on the internet, captured profiler while deploying the SSISDB database and found below in the profiler.

DECLARE @path NVARCHAR(255)
DECLARE @fullPath NVARCHAR(255)
EXEC master..xp_regread 'HKEY_LOCAL_MACHINE', @keypath, N'', @path OUTPUT
SET @fullPath = REPLACE( @path, '''', '''''' ) + N'Binn\Microsoft.SqlServer.IntegrationServices.Server.dll'
--PRINT(@fullPath);
-- Use OpenRowset to read the binary bits into @asm_bin
print ('DECLARE @asm_bin varbinary(max);
SELECT @asm_bin = BulkColumn FROM OPENROWSET (BULK N''' + @fullPath + ''',SINGLE_BLOB) AS dll
CREATE ASSEMBLY ISSERVER FROM @asm_bin WITH PERMISSION_SET = UNSAFE');

After running this, I got below statement as output

DECLARE @asm_bin varbinary(max);  
SELECT @asm_bin = BulkColumn FROM OPENROWSET (BULK N'd:\Program Files\Microsoft SQL Server\120\DTS\Binn\Microsoft.SqlServer.IntegrationServices.Server.dll',SINGLE_BLOB) AS dll
CREATE ASSEMBLY ISSERVER FROM  @asm_bin  WITH PERMISSION_SET = UNSAFE

The output might vary based on your SQL Server version.

Remember that you might be running into this when SQL is not getting started so you need to start SQL with trace flag 902 to bypass upgrade scripts, run CREATE ASSEMBLY and then restart SQL without trace flag. If you want to know about steps to enable trace flag, please read my earlier blog. SQL SERVER – What is Trace Flag – An Introduction

Please comment and let me know if this blog has helped you in solving upgrade issue.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Upgrade Error – Cannot Drop the Assembly ‘ISSERVER’, Because it Does not Exist or You Do Not Have Permission

This was indeed one of an interesting error which I have never seen earlier and it is about DAC Connection.

Here is the complete error message which my client reported.

Msg 3637, Level 16, State 3, Line 1
A parallel operation cannot be started from a DAC connection.
Msg 3013, Level 16, State 1, Line 1
BACKUP DATABASE is terminating abnormally.

By looking at the error message, I asked him how he is connecting to SQL Server and where exactly the error is appearing.

They informed that they have scheduled a backup job from Windows. Since it is an SQL Express edition, they created a batch file to take backup of the database. The batch file was running as a scheduled task via windows task scheduler. The error appears when a backup is initiated.

I asked them to show the batch file which is taking backup and it didn’t take much time to find what is going wrong while taking backup.

I was able to reproduce the error in my local lab environment as well.

WORKAROUND/SOLUTION

To reproduce the error, I connected to SQL Server via SQL Server Management Studio and gave the server name as “Admin:ServerName”. This caused the connection to go as Dedicated Admin Connection or DAC. This is not a normal connection to SQL Server via 1433 port. There are some limitations of DAC connection and unable to take a backup is one of them.

In my client’s situation, the .bat file had below.

SQLCMD -S <ServerName> -E -A -I”<path to backup script>

When I looked at the documentation, it says  [-A dedicated admin connection] and that explained everything. Here are my test results, with -A and without -A and you can see the difference.

In short, I asked the customer to change the script to remove “A” parameter to avoid DAC connection and since then the backups are running like a charm.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: 3637 – A Parallel Operation Cannot be Started From a DAC Connection