This was one of an interesting question which I heard from one of my clients who deployed Always On availability groups. I was not able to find much documentation and clarity so did some research and found an answer. This can also be an interview question as well.

Question:

I have configured Always On availability group using the Wizard. While configuration, I have selected a shared folder in below screen.

Due to some unavoidable reasons, I must remove that share.  If I remove that share, will availability group continues to function? Is it needed like the way we have shared location in log-shipping?

Answer:

Yes, we can remove that share and it won’t have any impact on the existing availability group. The purpose of the share is to do an initial sync via backup and restore method done by Wizard interface when we use Initialize now option. One backup and restore is complete and databases are synchronized, we can remove the files or even the share itself. Log Shipping needs a share to propagate transaction log backups in a shared location which is not the case with availability.

Have you faced a similar situation before? Let me know your thoughts about this blog post via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Can I Delete Always On Availability Groups Initial Sync Folder?

Earlier I wrote a blog on same error message which you can read from below link. SQL SERVER – FIX: Msg 35250, Level 16, State 7 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

One of my blog readers emailed me that he is getting same error 35250 (The Connection to the Primary Replica is Not Active. The Command Cannot be Processed.)

But he did not see “Database Mirroring login attempt by user” in the SQL Server ERRORLOG. He told me that my blog didn’t help me.

I reached him back via email and provided few steps to check. First, I asked to run below query to get port and service account details.

SELECT tep.NAME 'Endpoint Name'
	  ,sp.NAME 'Owner'
	  ,tep.state_desc 'State'
	  ,tep.port 'Port'
FROM sys.tcp_endpoints tep
INNER JOIN sys.server_principals sp ON tep.principal_id = sp.principal_id
WHERE tep.type = 4

Here was the output and this tells me that they are using the default port for Always On availability groups.

hadr_endpoint, database_mirroring, started, 5022

As a next step, I asked him to do two more things.

1. Telnet <Primary replica> 5022 from secondary replica via command line (cmd.exe).
2. Telnet <Secondary replica> 5022 from primary replica via command line (cmd.exe).

Telnet from the primary to the secondary worked, but it failed from the secondary to the primary. He was smart enough and tried to ping the IP address, but that did not work as well.

WORKAROUND/SOLUTION

Based on our test we concluded that this was an issue with network component. My client engaged his networking experts and found the cause of ping failure and fixed it.

Have you seen such error and found some more solution?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: Msg 35250 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

While preparing for a demo for an upcoming session, I came across an error which I didn’t see earlier. Here is the error in the UI and it is related to Backup Log Operation.

I am also trying to add a file to the database using below command.

USE [master]
GO
ALTER DATABASE [SQLAuth] ADD FILE ( NAME = N'SQLAuth_LogNew', FILENAME = N'F:\DATA\SQLAuth_LogNew.ndf' , SIZE = 8192KB , FILEGROWTH = 65536KB ) TO FILEGROUP [PRIMARY]
GO

And it also failed with following error.

If I change the file name, it works. As mentioned in error message I performed a log backup, and I was able to add the file again.

ROOT CAUSE

While working yesterday, I already added the file and dropped it. As per error message, if I need to reuse the same logical name of the database file, I must take a log backup. Here is the simple reproduction of the error.

CREATE DATABASE [DropFileTest]
GO
USE [master]
GO
-- take a full backup to make it real full else it would act as simple.
BACKUP DATABASE [DropFileTest] TO  DISK = N'DropFileTest.bak'
GO
-- add a new LDF file
USE [master]
GO
ALTER DATABASE [DropFileTest] ADD LOG FILE ( NAME = N'NewLDF',
FILENAME = N'F:\LOG\NewLDF.ldf' , SIZE = 8192KB , FILEGROWTH = 65536KB )
GO
-- reomve the file.
USE [DropFileTest]
GO
ALTER DATABASE [DropFileTest]  REMOVE FILE [NewLDF]
GO
-- Adding same file again and it should fail.
USE [master]
GO
ALTER DATABASE [DropFileTest] ADD LOG FILE ( NAME = N'NewLDF',
FILENAME = N'F:\LOG\NewLDF.ldf' , SIZE = 8192KB , FILEGROWTH = 65536KB )
GO

If you run above, you can notice the same error.

WORKAROUND/FIX

As mentioned in the error message, take a transaction log backup of the database. In the above demo script, we can do below before adding a file and it should work.

BACKUP LOG [DropFileTest] TO  DISK = N'DropFileTest.trn'
GO
-- Adding same file again and it should work this time.
USE [master]
GO
ALTER DATABASE [DropFileTest] ADD LOG FILE ( NAME = N'NewLDF', FILENAME = N'F:\LOG\NewLDF.ldf' )
GO

Have you seen such interesting error?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Msg 1833 – File Cannot be Reused Until After the Next BACKUP LOG Operation

Today the progress made in technology is astounding and beyond our imaginations, but still there are clients who can’t move to higher, latest and greatest version of SQL so easily. They are still stuck with SQL 2005 and SQL 2008 SQL servers. In this blog post we will learn how to fix RequireKerberos Error During SQL 2008 Installation on Windows 2008 R2 Cluster.

One of my client was trying to install SQL Server 2008 on Windows 2008 R2 cluster. While the setup was almost at the end, here was the error popped up.

The following error has occurred:
There was an error setting private property ‘RequireKerberos’ to value ‘1’ for resource ‘SQL Network Name (DELHISQLPRD)’. Error: Value does not fall within the expected range.

He was not sure what was wrong with his cluster. He contacted me and since I have worked with many other clients and this was seen earlier. I told him that this is a known issue (List of known issues when you install SQL Server on Windows 7 or on Windows Server 2008 R2) and Microsoft has fixed it in SP1 for SQL Server 2008

WORKAROUND/SOLUTION

As mentioned in knowledge base article by Microsoft, the issue is fixed in SP1 of SQL 2008. We need to use “Slipstream media” to install SQL Server 2008. Slipstream media mean include patch (like service pack) into the original installation. We need to follow below article to create slipstream media.

https://support.microsoft.com/en-us/help/955392/how-to-update-or-slipstream-an-installation-of-sql-server-2008

We made sure that we uninstalled SQL using “Remove Node” option before using slipstream media to install SQL cluster again. This time it worked like a charm.

Have you seen similar setup issues which you solved? Please comment and share with other readers.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: RequireKerberos Error During SQL 2008 Installation on Windows 2008 R2 Cluster

Sometimes some behaviors in SQL Server are unexpected and that’s the right time when an expert need to be engaged. In this blog post we will learn about how automatic startup sometimes does not work in SQL Server Services.

One of my client told that they had a business down situation for few hours. Based on their findings they realized that there was a restart of SQL Server machine due to power failure. They are OK with that, but their expectation was that SQL Service should have automatically started after machine reboot which didn’t happen. I asked to share the screenshot of SQL Server Configuration Manager.

By looking at the screenshot it’s clear that they have already configured “Start Mode” of SQL Service for Automatic. Which means their expectation was correct.

Next, we looked into Event Logs to see any interesting information. We found Event ID 5719 via source NETLOGON.

Log Name: System
Source: NETLOGON
Event ID: 5719
Description:
The MSSQLSERVER service was unable to log on as FORTUNE\SQLSvc with the currently configured password due to the following error: There are currently no logon servers available to service the logon request. To ensure that the service is configured properly, use the Services snap-in in Microsoft Management Console (MMC)

It was not very difficult to search below knowledge base article from Microsoft which explained the cause of such message. https://support.microsoft.com/en-in/help/139410/err-msg-there-are-currently-no-logon-servers-available

ROOT CAUSE

Based on the error messages we can conclude that in this scenario the SQL Service didn’t start automatically because SQL Server was unable to communicate to domain controller (DC). When I checked with client about above and asked about network connectivity to DC, they replied – “Unfortunately, domain controller was also rebooted due to power failure”.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Automatic Startup Not Working for SQL Server Service

It is interesting to see that in the past few days I have seen a similar pattern of an issue hit by three clients. All of them contacted me for a short engagement for the same issue. As you might know, every single day I keep two slots available for On Demand (50 minutes) so here is one of their emails. Let us learn about adding node in the AlwaysOn availability group.

“Pinal -Need your urgent help On Demand! We have added Node in AlwaysOn AG and things have broken. Basically, this cluster was set up a many months ago. Now, we were attempting to a new node at our DR site to the cluster. After adding the node all of the Disk Drives on the original two servers are showing offline and will not come online.” 

When I joined a session with them, I confirmed that they have added a new node to windows cluster which broke the disks. They were using Always On availability group, there was no need of shared storage. They used Add Node in Windows Failover Cluster Manager and keep moving next, next, finish. As soon as they did that, they found that their local drives became clustered. In our case, we had standalone instances for SQL and due to that the disk should be local disk and should not show up in the cluster administrator.

WORKAROUND / SOLUTION

1. We need to remove the disk resource from Failover Cluster Manager interface. Since there is no shared storage in our case, we do not want disks to be part of a cluster.
2. Then we need to bring these drives online in Disk Management (after making sure, step # 1 is completed)

ROOT CAUSE / CAUTION

While adding a new node to Windows Cluster, make sure that we uncheck “Add all eligible Storage to the Cluster”. I don’t know why Microsoft has this checked by default when there is no shared storage.

Hope this blog would help you in saving downtime which would be caused due to one checkbox. Just a single checkbox can help adding node issue in availability group.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Caution: Adding Node in AlwaysOn Availability Group

While preparing for a demo for a client I was hit with an interesting error in AlwaysOn Availability Group backup and restore scenario. In my lab setup, I had four nodes SQLAUTH1 to SQLAUTH4 and I have always on high availability turned on SQL Server 2016. I played and did several operations with database and found that I have bloated the transaction log file to 100 GB. And now I need to shrink the transaction log on SQLAUTH1. As soon as I did that, I was welcomed with below message about database cannot be shrunk until all secondaries have moved past the point.

The log for database ‘SQLAuth’ cannot be shrunk until all secondaries have moved past the point where the log was added.

It’s worth mentioning that I did add files in the database so above message is valid. I even took transaction log backup, but no luck. File usage was still shown as 99%.

WORKAROUND/SOLUTION

I looked around at AlwaysOn Dashboard and found that one of the secondary was not getting synchronized. Since it was a lab machine I decided to delete the AG and the thing came back to normal.

But in production, if you can’t afford to delete and reconfigure Always On availability group, then you need to find the cause why secondary is not getting synchronized with primary. In theory, the transaction log files on primary would keep on growing due to secondary not synchronizing with primary.  If you find that the data movement is suspended, then you may want to resume it and find the possible cause in the SQL Server ERRORLOG file

SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

One of the possible case I can think of is a change of the service account.

SQL SERVER – FIX: Msg 35250, Level 16, State 7 – The Connection to the Primary Replica is Not Active. The Command Cannot be Processed

STILL STUCK?

If you are not able to find the cause, feel free to contact me and use my consulting services. As you might know that I have been an independent consultant for a while and one of the services I provide is “On Demand (50 minutes)” service. This service is very helpful for organizations who are in need immediate help with their performance tuning issue. Though, I have set working ours for my regular clients, every single day, I keep two hours available for this offering. This way, I can make sure that anyone who urgently needs my help, can avail the same. Click here to read more about it.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – FIX: The Log for Database Cannot be Shrunk Until All Secondaries Have Moved Past the Point Where the Log was Added

Many companies do hardening of servers to make sure they are protected from any known vulnerabilities. It is important to test all the application in test server before doing hardening. Let us learn about how to fix Eventlog Service not starting.

Recently, one of my existing clients contacted me and informed that they are not able to start SQL Server Agent Service after running a hardening script. Without wasting much time, I joined the session with them and started looking at the logs.

I asked to start SQL Server Agent from configuration manager. It failed with an unhelpful error message

The request failed or the service did not respond in a timely fashion. Consult the event log or other applicable error logs for details.

When I checked the LOG folder, there was no SQLAgent.out generated. I thought of checking the event viewer and found below the message.

But is that related to SQL Server Agent startup issue? I ran the SQLAgent executable in console mode using below command

SQLAGENT.EXE" -i SQL2016 -c -v

This failed with an error:

The EventLog service has not been started
2017-08-03 09:24:45 – ? [098] SQLServerAgent terminated (normally)

When I checked “Windows Event Log” Service, it was stopped, and I was unable to start it. I was getting access denied error when trying to start.

Windows could not start the Windows Event Log service on Local Computer. Error 5: Access is denied.

I used Process Monitor tool and found that we had “Access Denied” on C:\Windows\System32\winevt\Logs\System.evtx file. This is the System Event Log. When we checked the properties, we found “Read-only” was checked which was not the case with other machines.

As soon as we removed the checkbox, we were able to start SQL Server Agent service.

Have you ever encountered such issues due to hardening? Please share via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – SQL Agent Not Starting. The EventLog Service has Not Been Started

One of my clients reported an interesting issue and I was needed to wear detective cap to solve it. The mystery, for which they hired me was as below for Database Files Initial Size Changing Automatically.

Pinal,
We see an interesting issue, and we believe there is something not right with our environment. As per best practices from one of your blog, we have configured the initial file size of the database files to big enough for next 1 year data growth.

We are noticing that the file size is not getting set and it changes to smaller size every night.  Would you be able to suggest us something?

MY INVESTIGATION

I asked them if there is any maintenance plan which runs every night and does the shrink of the database, like below?

And they said that they have no maintenance plan doing such activity.

Here are the various data points I looked at.

1. SELECT * FROM database_files

2. SELECT * FROM databases

3. sp_helpdb SAPDB

When I looked at sys.databases, I could see is_auto_shrink_on was set to 1.

You can run below query in you production database and make sure auto shrink is set to 0.

SELECT is_auto_shrink_on, * FROM sys.databases

ROOT CAUSE

As you can see above, we concluded that the behavior which they saw was because Auto Shrink property was set to ON for many databases. Since this was one of an active database, auto shrink was kicked as compared to other databases. This article discusses the use of both Auto options.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Database Files Initial Size Changing Automatically

The other day, I received an email asking – What does elastic scalability mean?

Honestly, a question such as this is really complicated as there is no one clear answer to it. Everyone has their own definition of elastic scalability. However, everyone will agree with my following statement about elastic scalability.

Elastic Scalability is the capability of the database to provide amazing scale-out performance and scale-in cost savings on-demand.

Every organization wants to get the most out of any growth opportunity presented to them. A successful business often predicts when they need more resources for their database. However, there are moments in the life of organizations when they grow so fast, they need to scale-out pretty fast. This is the time when one needs elastic scalability, more than any other day. You do not want your application to crash or start performing poorly when there are more business opportunity.

Remember for running a good business, you need your application to be responsive and available. To keep your database always available to scale, you need an architecture which can easily scale. With that said, if you look at all the database application around, you will agree that database relatively very complicated to scale.

On the other hand, these bursts of growth can sometimes not last long, so you need to be able to also cut your resources down so you aren’t spending too much money when you don’t have the volume.

Scaling a traditional relational database usually requires lots of planning and it is a very nervous road for even experienced DBA. Whereas, NoSQL often requires development work for the application.

Good News

Well, here is good news for everyone. Now you do not have to worry about application re-write when you have to scale-out your database. NuoDB preserves ACID guarantees and a standards-based SQL interface while providing simple and elastic scaling. In simple words you can add and remove database capacity as you need them, which eventually lower the total cost of ownership and your application team will have to not worry about database programming or architecture.

There are three unique advantages –

1. Lower total cost by matching resources with demand
2. Improve your product by focusing on application code
3. Grow your business without technical limitation

Call to Action

There are two simple call to action-

First – Read about how Elastic Scalability can be achieved by NuoDB

Second – Download and Try out NuoDB

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on NuoDB – Achieving Performance Through Scale-Out – Elastic Scalability

If you have ever contacted me via email, you would know that I am very active in replying to emails. Many of the emails are for suggestions and I don’t get much time to help everyone, but I do reply to them letting them know the alternatives. If you are following my blog, you would know that I do provide “On Demand” services to help critical issues. This blog is an outcome of one of such short engagement. This is how it started and it is about RPC Server.

Received an email
Need your urgent help “On Demand”!

We are trying to install SP2 on the passive node of a SQL Server 2016 cluster. This patch has already worked on one server, but now we’re getting RPC too busy errors.

We are under strict timelines to finish this activity. Can you please help us quickly?

Without spending much time, I asked them to join GoToMeeting and got started. When they showed me the screen, the SQL Server setup was on the error “Failed to retrieve data for this request”.

I asked to cancel the setup and share the setup logs to see exact error.

Microsoft.SqlServer.Management.Sdk.Sfc.EnumeratorException: Failed to retrieve data for this request. —> Microsoft.SqlServer.Configuration.Sco.SqlRegistryException: The RPC server is too busy to complete this operation.

WORKAROUND/SOLUTION

As we can see above, SQL setup is failing to do an activity on remote node. I immediately recalled a similar blog which I wrote earlier having same remote node symptoms, but the error message was different.

SQL SERVER – Microsoft.SqlServer.Management.Sdk. Sfc.EnumeratorException: Failed to Retrieve Data for This Request

We checked “Remote Registry Service” on remote node and sure enough, it was in “stopped” state. As soon as we started, we were able to move forward and finish the activity in less than the scheduled time.

If you are having any quick issue to resolve, you can also avail the same kind of services. Click here to read more about it.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Cluster Patching: The RPC Server is Too Busy to Complete This Operation

This blog explains what needs to be done when there are tools which can monitor and report issue about SPNs. Once a client informed that SCOM (System Center Operations Manager) is connected to the databases on SQL Server and raising below warning regarding the SQL Server (Service Principal Name) SPNs:

SQL Server cannot authenticate using Kerberos because the Service Principal Name (SPN) is missing, misplaced, or duplicated.

Service Account: SQLAUTHORITY\ProdSQLSrv

Missing SPNs: MSSQLSvc/SAPSQLSERVER.SQLAUTHORITY.NET:SAP, MSSQLSvc/ SAPSQLSERVER.SQLAUTHORITY.NET:1433

Here is the screenshot for the alert.

Whenever I see errors/warning related to SPN, I always try to use Kerberos Configuration Manager tool.

WORKAROUND/SOLUTION

It is important to note that this is an alert not a failure because SPN issue will not cause every connection to fail. It means that any application, which is trying to use Kerberos authentication is going to fail with errors:

1. Login failed for user ‘NT AUTHORITY\ANONYMOUS LOGON’
2. Cannot generate SSPI context

I downloaded the tool and installed in on the server. As soon we are launching the tool, we can just hit connect. The best piece about this tool is that it can help in finding missing SPN and provide script to run or fix it directly if you have permission. Basically, it can

• Gather information on OS and Microsoft SQL Server instances installed on a server.
• Report on all SPN and delegation configurations on the server.
• Identify potential problems in SPNs and delegations.
• Fix potential SPN problems.

If you are not a fan of a installing tool on the server then you can use SETSPN.exe to set the correct SPNs. Based on above error, here is the command.

setspn -A MSSQLSvc/ SAPSQLSERVER.SQLAUTHORITY.NET:1433 SQLAUTHORITY\ProdSQLSrv

Note that we need to run above line from an elevated command prompt with an account with domain admin permissions. We ran the command and after a minute we got the message all was fine. And as expected, the alerts don’t come back anymore.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SCOM – Alert: SQL Server Cannot Authenticate Using Kerberos Because the Service Principal Name (SPN) is Missing, Misplaced, or Duplicated

After installing the latest version of SSMS (17.2), I was not able to connect to SQL Server Integration Services SSIS on my local machine.

Here is the text of the error message.

Connecting to the Integration Services service on the computer “LOCALHOST” failed with the following error: “The specified service does not exist as an installed service.”.

This error can occur when you try to connect to a SQL Server 2005 Integration Services service from the current version of the SQL Server tools. Instead, add folders to the service configuration file to let the local Integration Services service manage packages on the SQL Server 2005 instance.

This error can occur when you try to connect to a SQL Server 2005 Integration Services service from the current version of the SQL Server tools. Instead, add folders to the service configuration file to let the local Integration Services service manage packages on the SQL Server 2005 instance.

I checked services.msc to make sure it was running.

Above image confirmed that error message “The specified service does not exist as an installed service” is totally misleading.

WORKAROUND/SOLUTION

After trying various steps given on various sites, I was looking at the documentation and found https://docs.microsoft.com/en-us/sql/integration-services/service/integration-services-service-ssis-service#manage-the-service This says:

To connect directly to an instance of the legacy Integration Services, Service, you have to use the version of SQL Server Management Studio (SSMS) aligned with the version of SQL Server on which the Integration Services Service is running. For example, to connect to the legacy Integration Services, Service running on an instance of SQL Server 2016, you have to use the version of SSMS released for SQL Server 2016

Which means that, I need to download and install older version of SQL Server Management Studio. (16.5.3).

As of today, the download link is http://go.microsoft.com/fwlink/?LinkID=840946 which might change in the future so follow below steps.

1. Go to https://docs.microsoft.com/en-us/sql/ssms/sql-server-management-studio-changelog-ssms
2. Search for “SSMS 16.5.3” and you should reach to the below image.
   
3. Click on the hyperlink to download

Hope this would help someone’s time as I wasted close to 30 min.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL Server Management Studio (SSMS) – Unable to Connect to SSIS – The Specified Service Does Not Exist as an Installed Service

Have you ever seen an issue related to MS DTC, SQL Server and JDBC XA distributed transactions? A client contacted me for help and he wanted my opinion within one hour. They signed up for my On Demand consulting services and we started looking at the server and application.

They showed me the error in their application which was as below

Caused by: javax.transaction.xa.XAException: com.microsoft.sqlserver.jdbc.SQLServerException: Failed to enlist. Error: “Import of Microsoft Distributed Transaction Coordinator (MS DTC) transaction failed: 0x8004d00e(XACT_E_NOTRANSACTION).”
at com.microsoft.sqlserver.jdbc.SQLServerXAResource.DTC_XA_Interface(SQLServerXAResource.java:647)
at com.microsoft.sqlserver.jdbc.SQLServerXAResource.start(SQLServerXAResource.java:679)
at com.ibm.ws.rsadapter.spi.WSRdbXaResourceImpl.start(WSRdbXaResourceImpl.java:1525)
at com.ibm.ejs.j2c.XATransactionWrapper.start(XATransactionWrapper.java:1475)
at com.ibm.ws.Transaction.JTA.JTAResourceBase.start(JTAResourceBase.java:157)
at com.ibm.tx.jta.impl.RegisteredResources.startRes(RegisteredResources.java:1044)

I asked them to show SQL ERRORLOG to see if there are some DTC related messages.

2017-08-20 09:47:16.76 spid78 Attempting to load library ‘SQLJDBC_XA.dll’ into memory. This is an informational message only. No user action is required.
2017-08-20 09:47:16.77 spid78 Using ‘SQLJDBC_XA.dll’ version ‘0004.00.2206’ to execute extended stored procedure ‘xp_sqljdbc_xa_start’. This is an informational message only; no user action is required.
2017-08-20 09:47:21.45 spid77 Attempting to initialize Microsoft Distributed Transaction Coordinator (MS DTC). This is an informational message only. No user action is required.
2017-08-20 09:47:23.95 spid77 Recovery of any in-doubt distributed transactions involving Microsoft Distributed Transaction Coordinator (MS DTC) has completed. This is an informational message only. No user action is required.
2017-08-20 09:47:23.96 spid77 Error: 8509, Severity: 16, State: 1.
2017-08-20 09:47:23.96 spid77 Import of Microsoft Distributed Transaction Coordinator (MS DTC) transaction failed: 0x8004d00e(XACT_E_NOTRANSACTION).

Every time we tried XA transaction from application, it caused error 8509 in ERRORLOG.

Here are a few more details about the environment which you may want to match before implementing:

1. Clustered Instance of SQL Server.
2. Separate group for MSDTC resource in cluster.

DTC Settings were OK. Below is the checklist

• Go to “Administrative Tools > Component Services” (or Start > Run > DcomCnfg > Enter)
• On the left navigation tree, go to “Component Services > Computers > My Computer> Distributed Transaction Coordinator > Clustered DTCs “
• Right click on the DTC service for this SQL Server group and select “Properties”.
• Go to the security tab and check “Network DTC Access”, “Allow Inbound”, and “Allow Outbound”.

Since it was a clustered DTC, the next thing which came to my mind was DTC and SQL mapping in the cluster. I looked up at various resources and found the solution.

WORKAROUND/SOLUTION

Here are the three commands I used to fix the mapping.

1. View the mapping

msdtc -tmmappingview *

2. Clear the mapping.

msdtc.exe -tmMappingClear -name DTC_INST02_Mapping

3. Create correct mapping

Msdtc -tmMappingSet -name DTC_INST02_Mapping -service "MSSQL$INST02" -ClusterResourceName "MSDTC-INST02"

Do you need a DTC If you are not using SQL JDBC XA? You do not need to set TM mappings. SQL 2008 Server knows which clustered DTC instance, it needs to use. Since TM mappings are stored in the cluster registry which is shared across all nodes, we can set them at from any of the cluster nodes.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Error: 8509 – Import of Microsoft Distributed Transaction Coordinator (MS DTC) transaction failed: 0x8004d00e(XACT_E_NOTRANSACTION)

I was engaged with a client for an AlwaysOn project and they had some follow-up questions. I took some time to find the answers and encountered an interesting error. I am sharing them here so that others can get benefited. They informed me that they are not able to see and modify listener properties. Let us learn about this error related to opening resource.

Initially, I shared script to get the properties of the listener via T-SQL. As you can see below, we can use catalog views.

SELECT grp.name AS [AG Name],
lis.dns_name AS [Listener DNS Name],
lis.port AS [Listener Port]
FROM sys.availability_group_listeners lis
INNER JOIN sys.availability_groups grp
ON lis.group_id = grp.group_id
ORDER BY grp.name, lis.dns_name

Here is the output.

My client came back and told that networking team has asked to change RegisterAllProvidersIP setting. We are not able to use PowerShell and getting error “An error occurred opening resource”. We are not sure what wrong with the listener in the cluster.

Get-ClusterResource AGListener | Get-ClusterParameter
PS C:\> Get-ClusterResource AGListener | Get-ClusterParameter
Get-ClusterResource : An error occurred opening resource 'AGListener'.
At line:1 char:1
+ Get-ClusterResource AGListener | Get-ClusterParameter
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 + CategoryInfo : ObjectNotFound: (:) [Get-ClusterResource], ClusterCmdletException
 + FullyQualifiedErrorId : ClusterObjectNotFound,Microsoft.FailoverClusters.PowerShell.GetResourceCommand

If we look at cluster the value “AGListener” we are using seems correct, but still PowerShell thinks its incorrect. Here is the screenshot from cluster manager.

I did some more searching and found that when we create Listener in through SSMS its naming convention like AGNAME_ListenerName. This is the reason that when we run the command Get-ClusterResource for the listener, we can’t see the properties. Here are the properties of the listener resource. (Right Click)

WORKAROUND/SOLUTION

Based on above explanation, we need to use the “name” as shown in properties and the command was working as expected.

Get-ClusterResource BO1AG_AGListener | Get-ClusterParameter

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Get Listener Properties Using PowerShell – An Error Occurred Opening Resource

Cloud Computing is not new anymore; everyone is talking about the cloud. Has everyone already touched the sky?  

Apparently not. There are many organizations around that hesitate to make the move, or cannot because of compliance issue or other limitations.

How about adopting a Hybrid cloud approach? This means keeping some of the data on premise, while expanding to the cloud with other data. The cloud data can be new data, or non operational data or any other way of data distribution that fits the business’s needs. The hybrid solution reduces the risk and keeps the crucial data close to the business while gradually establishing cloud traction.

Sounds like the perfect solution. But how do you start?

A company named PartitionDB identified the need of data distribution. It provides a simple solution to partition a database and manage it as a single database, while the partitions reside anywhere, on premise or in the cloud. This is a natural hybrid cloud solution that lets you work exactly as before. This is definitely something you want to try…

The below example is a great introduction for anyone that has never worked with the cloud before. It shows how to distribute a database in which the operational data reside on premise and the non operational data is in the cloud.

A Gate

PartitionDB engine is called a Gate. It is responsible to manage and control the distributed databases.  

The Gate can be created as a separate database, or to be integrated into the on premise database. The following diagrams depict the two different options:

Separate Gate	The gate is in the operational database

The article follows the second case of Gate combined into the operational database.

Before you start

In order to run a hybrid service (as well as this example) set up first both environments; on premise and cloud. Please see instructions here.

Ready to go

This demo is using BayMart database. Refer to PartitionDB Hybrid Example for a full description.

Step 1) Create a Gate

A gate should be created as part of BayMart database. The following commands create the Gate:

Step 2) Create a database in the cloud

We currently have a BayMart database on premise. Now we create a database in the cloud:

We use BayMartCloud as the cloud database name.

The ecosystem is ready, and we can continue to work as before. Yet, now we can store data remotely, in the cloud.

Time to work

Let’s start with an example of creating a table in the cloud.

As we can see, the table is created, but we now want to have it stored in the cloud and not on premise. To move the table to the cloud, copy and run the command that is printed in the Messages area:

exec PdbtargetSplitTable @GateName=’BayMart’, @SchemaName=’dbo’, @TableName=’CustomerLogins’,
@DatabaseName=’BayMartCloud’;”

The table is now located in the cloud database ‘BayMartCloud’:

Note that database distribution can be applied to any object: tables, stored procedures, views etc.

It’s time to really take advantage of the hybrid environment that we have just set up. You keep working as before, and there is no need to change your code. You can run queries that cross over the two databases:

This script runs an inner join between two tables, Customers that is located on the premise and Customer Logins that is in the cloud. The code is the same as if the two tables were located in the same database.

In the next article I will talk about cross hybrid solutions to distribute a database between few machines and even between different clouds.

Meanwhile, checkout – PartitionDB.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – How to Reach Out to Cloud – Cloud Computing with PartitionDB

While doing a preparation of a demo, I encountered below error while creating the listener of the AlwaysOn availability group. I was trying this from Management Studio.

Here is the text of the error message.

The configuration changes to the availability group listener were completed, but the TCP provider of the instance of SQL Server failed to listen on the specified port [AGListener:1433]. This TCP port is already in use. Reconfigure the availability group listener, specifying an available TCP port. For information about altering an availability group listener, see the “ALTER AVAILABILITY GROUP (Transact-SQL)” topic in SQL Server Books Online. (Microsoft SQL Server, Error: 19486)

If you investigate SQL Server ERRORLOG, you should see below.

2017-09-11 11:29:34.60 spid74 Error: 19476, Severity: 16, State: 4.
2017-09-11 11:29:34.60 spid74 The attempt to create the network name and IP address for the listener failed. If this is a WSFC availability group, the WSFC service may not be running or may be inaccessible in its current state, or the values provided for the network name and IP address may be incorrect. Check the state of the WSFC cluster and validate the network name and IP address with the network administrator. Otherwise, contact your primary support provider.
2017-09-11 11:30:01.19 Server The Service Broker endpoint is in disabled or stopped state.
2017-09-11 11:30:01.19 Server Error: 26023, Severity: 16, State: 1.
2017-09-11 11:30:01.19 Server Server TCP provider failed to listen on [ 10.0.1.50 1433]. Tcp port is already in use.
2017-09-11 11:30:01.19 Server Error: 26075, Severity: 16, State: 1.
2017-09-11 11:30:01.19 Server Failed to start a listener for virtual network name ‘AGListener’. Error: 10013.
2017-09-11 11:30:01.19 Server Stopped listening on virtual network name ‘AGListener’. No user action is required.
2017-09-11 11:30:01.19 Server Error: 10800, Severity: 16, State: 1.
2017-09-11 11:30:01.19 Server The listener for the WSFC resource ‘dc977169-2387-499e-8047-3b197e7ada61’ failed to start, and returned error code 10013, ‘An attempt was made to access a socket in a way forbidden by its access permissions. ‘. For more information about this error code, see “System Error Codes” in the Windows Development Documentation.
2017-09-11 11:30:01.19 Server Error: 19452, Severity: 16, State: 1.
2017-09-11 11:30:01.19 Server The availability group listener (network name) with Windows Server Failover Clustering resource ID ‘dc977169-2387-499e-8047-3b197e7ada61’, DNS name ‘AGListener’, port 1433 failed to start with a permanent error: 10013. Verify port numbers, DNS names and other related network configuration, then retry the operation.

SOLUTION/WORKAROUND

Above confirmed that there is some other process listener on 1433 port. Then I use one of my old blog to find out which process is using that port.

SQL SERVER – Unable to Start SQL Service – Server TCP provider failed to listen on [‘any’ 1433]. Tcp port is already in use.

I found that it was another system process, so I should change my listener port. Since listener was already created all I needed was to change the port. Here is the T-SQL I have used to change the port to 2433.

USE [master]
GO
ALTER AVAILABILITY GROUP [BO1AG]
MODIFY LISTENER N'AGListener' (PORT=2433);
GO

This time, command was completed without any error. I confirmed from the Errorlog that there was no error related to the listener.

2017-09-11 11:34:35.460 Server Started listening on virtual network name ‘AGListener’. No user action is required.

Have you encountered any such error? Feel free to comment and share it with others.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – AlwaysOn Availability Group Listener – This TCP Port is Already in Use

In my lab setup, I already have a 2 node windows 2012 R2 cluster. I already had one SQL server 2012 instance is working fine without issues. However, when I was trying to install a new additional SQL 2012 instance, the installation reached till the last phase and getting failed with errors related to a cluster resource.

Here is the text of the error message.

The following error has occurred:
The cluster resource ‘SQL Server’ could not be brought online due to an error bringing the dependency resource ‘SQL Network Name(SAPSQL) ‘ online. Refer to the Cluster Events in the Failover Cluster Manager for more information.
Click ‘Retry’ to retry the failed action, or click ‘Cancel’ to cancel this action and continue setup.

When we look at the event log, we saw below message (event ID 1194)

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: 20/06/2017 19:55:45
Event ID: 1194
Task Category: Network Name Resource
Level: Error
Keywords:
User: SYSTEM
Computer: NODENAME1.internal.sqlauthority.lab
Description:
Cluster network name resource ‘SQL Network Name (SAPSQL)’ failed to create its associated computer object in domain ‘internal.sqlauthority.com’ during: Resource online.

WORKAORUND/SOLUTION

To solve this problem, we logged into the domain controller machine and created the Computer Account: SAPSQL (called as VCO – Virtual Computer Object). Gave the cluster name WINCLUSTER$ full control on the computer name. If we carefully read error message, we have the solution already listed there. Then clicked on the retry option in the setup. The setup continued and completed successfully.

Here are the detailed steps (generally done on a domain controller by domain admin):

1. Start > Run > dsa.msc. This will bring up the Active Directory Users and Computers UI.
2. Under the View menu, choose Advanced Features.
3. If the SQL Virtual Server name is already created, then search for it else go to the appropriate OU and create the new computer object [VCO] under it.
4. Right click on the new object created and click Properties.
5. On the Security tab, click Add. Click Object Types and make sure that Computers is selected, then click Ok.
6. Type the name of the CNO and click Ok. Select the CNO and under Permissions click Allow for Full Control permissions.
7. Disable the VCO by right clicking.

This is also known as pre-staging of the VCO.

Hope this would help someone to save time and resolve issue without waiting for someone else assistance. Do let me know if you ever encountered the same.

I have used following two articles to find a solution for the error related to cluster resource.

1. Failover Cluster Step-by-Step Guide: Configuring Accounts in Active Directory
2. Event ID 1194 — Active Directory Permissions for Cluster Accounts

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – The Cluster Resource ‘SQL Server’ Could Not be Brought Online Due to an Error Bringing the Dependency Resource

Those who are my regular clients would know that I am very active in replying to emails. My average time of response is around 24 minutes. Many of the emails are for suggestions and I don’t get much time to help everyone, but I do reply to them letting them know the alternatives. If you are following my blog, you would know that I do provide “On Demand” services to help critical issues. This blog is an outcome of one of such short engagement about login failed.

One of my client was worried about login failed messages which they were seeing in the SQL Server ERRORLOG file.

2017-09-11 04:53:19.880 Logon Error: 18456, Severity: 14, State: 38.
2017-09-11 04:53:19.880 Logon Login failed for user ‘GLOBAL\PORTAL01$’. Reason: Failed to open the explicitly specified database. SharePoint_Config’ [CLIENT: ]

As per them, there is no complaint from anyone about any issue, but those messages are not looking good.

WORKAROUND/SOLUTION

First, we needed to figure out the account which is trying to access. If you look at the account which is shown in the error message is ending with “$” which means a machine account. In our case PORTAL01 was a front server in the SharePoint farm. This comes when there is some service, running under Local System account, it is trying to connect. After digging further, I found that this was SharePoint server which was trying to connect.

State 38 of Login failed, is logged when the account is having insufficient access to the database (SharePoint_Config). To fix it, we connected to SQL Server using SSMS and navigated to the Security > Logins > Right click on the account, and went to properties. We clicked on “User Mapping” tab and there we saw that the login was not mapped with the database SharePoint_Config. We noticed that it was only mapped to the master database. Now, this explains the cause of the error messages in ERROLROG.

Once we mapped that login to that mentioned database, we stopped receiving login failed error messages.

If you are having any quick issue to resolve, you can also avail the same kind of services. Click here to read more about it.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Login Failed – Error: 18456, Severity: 14, State: 38 – Reason: Failed to Open the Explicitly Specified Database

Backing up databases is a balancing act: you want to make sure you’re getting a safe solution that’s effective, but you don’t want to break the bank doing it. Today we’re looking at software for MS SQL Server that hits all the right notes, SqlBackupAndFtp. It’s an automated, scriptable backup program that lets you backup to local drive, FTP or cloud storage. And, with a free tier that lets you backup upto two databases at no charge, it’s designed to let you grow into the solution you need.

SqlBackupAndFtp is designed to work with all versions of Microsoft SQL Server, especially SQL Server Express 2005, 2008 & 2014 which lack any backup mechanism at all. It provides an easy-to-use interface for scheduling backups, checking on the results, and restoring if necessary. Below, I will walk through a few of the reasons why I’m recommending it as a simple backup solution for SQL Server.

Interface

The time is long past when you had to read software manuals in order to start using some application. Nowadays, it’s a must for software to have a clear and intuitive interface and I think SqlBackupAndFtp fully meets this requirement. The windows is divided into three sections: on the left you see all your backup jobs with their statuses, in the middle you can tune the selected job, and on the right section you see all backup history related to the job.

Backup storages

All backups has a tendency to grow over time. Businesses grow, the amount of data you’re storing grows, and the amount of time you’re required to store it can grow as well, all of which can lead to expanding storage needs. That’s why having multiple options for destinations is important. Often times, you’ll start with a simple backup host such as an FTP, and find that you eventually have to upgrade as time goes on. With SqlBackupAndFtp you can start with a simple FTP solution out of the box, then upgrade to any major cloud provider. It has support for Amazon’s S3 and Azure’s Blob Storage, meaning you can use however much space you need.

On top of that, you can compress and encrypt the backups, allowing you to save space and provide an additional layer of security. When you’re paying for each GB you use with a cloud provider and for the bandwidth to transmit them, zipping your backups before you send them can save you hundreds. And if you’re storing sensitive data, you’ll appreciate AES encryption and password-protected zip files that let know your data can only be read by you.

Automation

Database backups are about making sure your most valuable content is secure. That means you need to be able to create your own schedule, and you need to be sure that the software you’re using follows through with it. With SqlBackupAndFtp, you build your backup schedule. Because it integrates closely with the server, you can include multiple different types of backups in the same schedule, full backups, differential, or the transaction logs.

Restore

The automation doesn’t end with the scheduling. Trusting in your backups means you need to be able to restore as fast as possible to prevent costly downtime. SqlBackupAndFtp offers one-click restoration, you just need to identify the backup you’re restoring from and it will be restored on your server. On top of that, the application will correctly choose and restore in the right order all transaction log and differential backups necessary for restoring your database to the selected point in time.

Backup confirmation emails

After you’ve set up your backups, SqlBackupAndFtp sends automatic emails letting you know when they’ve run, giving you some extra peace of mind. The program also gives you the option of setting a separate email address for failure notifications, allowing you to notify your team of any irregularities before they become a real issue. The emails can be sent through SqlBackupAndFtp service, meaning that instead of filling up all your SMTP information you need just to type your email address only.

Web Log

Another interesting feature in SqlBackupAndFtp is the web log. If you check “See the backup history on the web” in job settings, you can see the job history on sqlbackupandftp.com/weblog:

If you have multiple computers running SqlBackupAndFtp, it is very convenient to see the status of all of your computers in one place

A Complete Solution for Lower Price

It’s interesting that while SqlBackupAndFtp is about $90 per license, it can successfully compete with other enterprise database backup software that runs almost $1000 for a single license. You can meet your essential needs and secure your data for a fraction of the price, and still get the full feature set of more expensive competitors.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Effective Way to Take Backup with SQL Backup and FTP 11