All RDBMS products are good, but I love SQL Server most because its fun to troubleshoot the issue. Meaningful error messages can make life easier and helps us in fixing the issue faster. Another reason I love SQL Server is that is my source of earning and I can run my family life due to SQL Server. In this blog, I am going to share my experience to fix an issue where my client was unable to failover SQL Server to another node.

If you search the internet, you would find many probable causes of failover issue. As I mentioned earlier, the exact cause can be determined with an error message and logical reasoning. To hunt right error message, I started looking at various logs.

SQL Server ERRORLOG:

You can refer to below blog which talks about ERRORLOG.

SQL SERVER – Where is ERRORLOG? Various Ways to Find ERRORLOG Location

<DateTime> spid20s Service Broker manager has shut down.
<DateTime> spid11s SQL Server is terminating in response to a ‘stop’ request from Service Control Manager. This is an informational message only. No user action is required.
<DateTime> spid11s SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.

From above we can see that it was not a SQL Server crash or a failure. SQL Server received a STOP request from Service Control manager due to which it went down. Which means SQL Server had started successfully, but due to some issues, it was asked to shut down. Now, remember this is a cluster. There are 2 stages involved to get the SQL Server successfully started in a cluster.

• Get SQL Server service locally started.
• Get SQL Server resource online in failover cluster manager.

If any of the 2 stages fail, SQL Server will go to failed state or it will fallback to the other node. In the current situation, it looked like it failed in the 2^nd stage. Because it was the 2^nd stage, it makes all the sense to look at the cluster log to check why it failed.

Cluster.log

This file needs to be generated manually. Refer my earlier blog to know the steps. SQL SERVER – Steps to Generate Windows Cluster Log?

Let me simplify with only the main errors of interest. Generally, you need to focus on lines having “ERR” in them.

1. [Microsoft][SQL Server Native Client 10.0]Registry information is corrupt or missing. Make sure the provider installed and registered correctly.
2. [Microsoft][SQL Server Native Client 10.0]Client unable to establish a connection
3. [Microsoft][SQL Server Native Client 10.0]A network-related or instance-specific error has occurred while establishing a connection to SQL Server. Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections. For more information see SQL Server Books Online.

The errors are talking about registry information missing or corrupted. Luckily, the client had another SQL Server instance present in the same cluster. They were OK to test a quick failover. As I was suspecting in my mind, that SQL instance too failed to come online, and the errors were the same. So, this was an issue related ONLY to the NODE 2. Then when I was comparing the settings between both the nodes, I noticed that it with the 64bit Client Protocols were missing on Node2.

SOLUTION/WORKAROUND

Based on my search on the internet, these protocols are represented by the registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSSQLServer

As expected the above key was not present on Node2 and was present in Node1. Now I knew that why we did not have any issues on Node1. Without making the solution too complicated, we extracted the above-mentioned registry key from Node1 and imported it on Node2. After that, we were able to successfully failover the SQL Server instance. I must add that registry modification needs to be done with caution because there is no undo possible without backup. That’s why I always back up the registry before taking any such actions.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Failover SQL Server Instance – Error: Registry Information is Corrupt or Missing

There are many causes of SQL server patching failure. Looking at right log and finding the right cause is the key to solve it in time. In this blog, we would learn about an upgrade error which comes during “Upgrading subscription settings and system objects in database.”

One of my blog reader reached out to me via email. I offered them my incredibly famous On Demand assistance and I got started working in 10 minutes via GoToMeeting.

When I checked ERRORLOG, I found below.

2018-04-19 14:16:38.98 spid9s Upgrading subscription settings and system objects in database [BAPDB].
2018-04-19 14:16:39.12 spid9s Invalid object name ‘MSreplication_subscriptions’.
2018-04-19 14:16:39.12 spid9s Error executing sp_vupgrade_replication.
2018-04-19 14:16:39.12 spid9s Saving upgrade script status to ‘SOFTWARE\Microsoft\MSSQLServer\Replication\Setup’.
2018-04-19 14:16:39.12 spid9s Saved upgrade script status successfully.
2018-04-19 14:16:39.12 spid9s Database ‘master’ is upgrading script ‘upgrade_ucp_cmdw_discovery.sql’ from level 201330692 to level 201331592.
2018-04-19 14:16:39.40 spid9s Database ‘master’ is upgrading script ‘msdb110_upgrade.sql’ from level 201330692 to level 201331592.

WORKAROUND/SOLUTION

I Informed them they I already have a blog with exact same error SQL SERVER – Upgrade Error – ALTER DATABASE Statement Not Allowed Within Multi-statement Transaction

But they informed that they don’t have any replication as of now and they don’t want to create any system table using the method which I explained in my blog. I thought there could be a chance that some replication metadata was still left over, and script upgrade process thinks replication still persists in this database and was expecting this table. Based on their decision, I started SQL with trace flag 902.

NET START MSSQLSQLSERVER /T902

Refer: SQL SERVER – 2005 – Start Stop Restart SQL Server from Command Prompt

We used the system stored procedure (sp_removedbreplication) to remove replication related config from this database (this is there in ERRORLOG file)

USE BAPDB
GO
sp_removedbreplication 
GO

Once this was done then, I stopped SQL and started it normally (without trace flag 902) and it was able to start successfully.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Upgrade Error – Upgrading Subscription Settings and System Objects in Database [DBName].

As a part of my AlwaysOn related consultancy, one of my clients was having challenges to install SQL Server 2016 in a clustered environment. In this blog, we would learn about the cause of Slow SQL Server 2016 Installation in Cluster.

When I started SQL Server Setup, it got hung on this screen.

In the Detail.txt we saw the below info:

(01) 2018-03-30 09:25:34 Slp: Running Action: RunRemoteDiscoveryAction
(08) 2018-03-30 09:25:34 Slp: Discovered update on path C:\SQL2016\SQLServer2016SP1 \PCUSOURCE; Update: Microsoft SQL Server 2016 with SP1, Type: PCU, KB: 3182545, Baseline: 13.0.1601, Version: 13.1.4001
(01) 2018-03-30 09:25:34 Slp: Running discovery on remote machine: NODE2
(01) 2018-03-30 09:25:34 Slp: Running discovery on local machine
(08) 2018-03-30 09:25:34 Slp: Using service ID ‘3da21691-e39d-4da6-8a4b-b43877bcb1b7’ to search product updates.
(10) 2018-03-30 09:25:34 Slp: Searching updates on server: ‘3da21691-e39d-4da6-8a4b-b43877bcb1b7’

Based on above snip of the log, we can see that the action which setup was running is — Running Action: RunRemoteDiscoveryAction

What came to my mind is what does it take for setup to connect to Node2. This is where it trying to perform RemoteDiscoveryAction. Based on my previous experiences fixing such things, I could think of

• Remote registry service in a stopped state
• Remote Registry connectivity is disabled
• Admin$ shares are disabled.

SOLUTION/WORKAROUND

In this case, we saw that the Admin$ shares were disabled. It can be easily tested by typing the below command in CMD prompt or use any file explorer window.

C:\>\\NODE1\c$

As soon as we hit the enter key, we got the message.

Here are the steps to get Admin$ share back (Reference)

• Open a registry editor, start > Run > Regedit.exe.
• Navigate to: HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters
• In the right pane, locate and double-click AutoShareServer.
• Change the value from 0 to 1.
• Close the registry editor and restart the “Server” service for the change to take effect.

After allowing Admin$ share access, SQL setup did not have any further challenges and completed successfully. This action needs to be done all the nodes participating in the cluster. A reboot of the node is also required. Maybe on the latest operating systems (like Windows 2016), a reboot may not be required.

If above steps solve your installation issue, please let me know via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Slow SQL Server 2016 Installation in Cluster: RunRemoteDiscoveryAction

In my lab environment, I was testing a script to change service account and it worked fine. I had Always On configured on this SQL Server and soon I realized that after changing the SQL Server service account, the secondary replica went into a disconnected state.

We could see the below errors in the Primary Replica SQL Errorlog

<DateTime> Logon  Database Mirroring login attempt by user ‘MyDC\SQLAccount’ failed with error: ‘Connection handshake failed. The login ‘MyDC\SQLAccount’ does not have CONNECT permission on the endpoint. State 84.’.  [CLIENT: 192.168.x.x]

If we change the service account back to the old one, everything goes back to normal. There are no issues. So, it seemed like that there were some issues with the new account we were using and apparently AlwaysOn AG did not like that account. The error also says that the account does not have “connect” permission to the endpoints. We ran the below query to check who all have got permissions on the AlwaysOn Endpoint — Hadr_endpoint

SELECT e.name AS mirror_endpoint_name
	,s.name AS login_name
	,p.permission_name
	,p.state_desc AS permission_state
	,e.state_desc endpoint_state
FROM sys.server_permissions p
INNER JOIN sys.endpoints e ON p.major_id = e.endpoint_id
INNER JOIN sys.server_principals s ON p.grantee_principal_id = s.principal_id
WHERE p.class_desc = 'ENDPOINT'
	AND e.type_desc = 'DATABASE_MIRRORING'

The output looked like below:

We had changed the SQL service account to — ‘MyDC\SQLAccount’. The old one was ‘MyDC\SQLSvcAccount’

Looks like we were missing the CONNECT GRANT on the EndPoint permission here. We performed the following steps that resolved the issue.

WORKAROUND/SOLUTION

1. Created login of the newly added service account on both replicas.

USE [master]
GO
CREATE LOGIN [MyDC\SQLAccount] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
GO 

2. Granted connect permission the endpoints on both replicas.

GRANT CONNECT ON ENDPOINT::hadr_endpoint TO [MyDC\SQLAccount]
GO

3. Stopped the endpoints on both the replicas.

ALTER ENDPOINT hadr_endpoint STATE=STOPPED

4. Started endpoints on both the replicas.

ALTER ENDPOINT hadr_endpoint STATE=STARTED

After making the above changes replicas were back in the connected state. We tested the failovers and it worked great. Have you faced a similar issue with AlwaysOn AG? Please share your experience via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com) 

First appeared on SQL SERVER – Always On Replica Disconnected After Changing SQL Server Service Account

I was trying to install SQL BI features for one of the assignments I had got. In the setup wizard when I reached the Features Page I got a surprise of my life. There were many features missing! I could not see features available while installing SQL Server listed here:

• SQL Analysis Service
• SQL Reporting Services
• SQL Integration Services
• Data Quality Services
• Polybase Query Service for External Data

SOLUTION/WORKAROUND

After a brief period fiddling with this issue, I was able to figure that the corresponding MSI files for these missing features were not even present in the setup media. These are usually present in the below path,

<Media>\x64\Setup

This could have happened when I was extracting the ISO image. I quickly extracted the ISO once again and found that this time I was able to see the missing MSI files and I was also able to see the features listed in the setup wizard,

This was a really interesting behavior where setup doesn’t give any error. It just removes the component from the list! I hope you find this blog post interesting as I personally did not ever encountered such an error and for me, it was a strange error.

Please leave your opinion in the comments section.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – All the Features not Listed While Installing SQL Server

My lab machine is my playground. Other than my real-world client’s problem, many blogs have come from my lab machine because I keep on breaking something or another on my server. Once I found that I was unable to start SQL Server Agent service for one of the SQL Server instances. In this blog, we would learn how to fix error Failed to initialize SQL Agent log (reason: Access is denied)

As the first step of troubleshooting, I went to the SQL Server Errorlog folder to check for errors. The SQL Server Errorlog default path is,

<Drive>:\Program Files\Microsoft SQL Server\MSSQL1x.<InstanceID>\MSSQL\Log\SQLAGENT.OUT

Surprisingly when I went to the above location, I could not find the latest SQL Agent log. It only showed the older ones. We tried to start the SQL Agent service from service control manager and got the below error.

I still didn’t see the SQL Agent log. I tried to investigate the Event Viewer logs and found the below info in application event log.

Information SQLSERVERAGENT 102 Service Control SQLServerAgent service successfully stopped.
Error SQLSERVERAGENT 324 Alert Engine Failed to initialize SQL Agent log (reason: Access is denied).

Now, even the above error is not helping us much. But it revealed few vital clues for us.

• We have a permission issue.
• And it is related to SQL Agent Log.

I went ahead and checked the required permission on the SQL Agent log folder and they all looked fine. From my experience, for any “Access Denied” message the next tool which I prefer is Process Monitor (Procmon). I will not deal with procmon on this blog. So, let’s try to start SQL Agent in CMD Prompt and use some startup parameters.

I tried starting SQL Agent using the below command,

<Drive>:\Program Files\Microsoft SQL Server\MSSQL1x.<InstanceID>\MSSQL\Binn\SQLAGENT.exe -i MSSQLSERVER -c -v

• -i = Instance name
• -c = console mode
• -v = verbose mode

I got the below errors when we started SQL Agent using the above command.

This error looks similar to the error we saw in event viewer. But read again!! I’m sure you might have missing reading the first line.

Unable to rename C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Log\ to C:\Program Files\Microsoft SQL Server\MSSQL13.1 (reason: Access is denied)

If you closely notice, SQL Agent service trying to rename the above folder! After spending a few moments decrypting this, I was finally able to get the logic. Hear me out and let me know if my logic was correct.

The current SQL Agent log file is called as “SQLAGENT.OUT” Every time we restart SQL Agent service, the SQLAGENT.OUT is renamed to SQLAGENT.1 and a new SQLAGENT.OUT file is created and eventually becomes the current log file.

In the above scenario, SQL Agent thinks the current log file name is- “MSSQLSERVER\MSSQL\Log”

C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Log

This is because we have DOT separator as an extension of a file. So, SQL Agent is trying to rename MSSQLSERVER\MSSQL\Log to .1 as per the renaming logic and thus failing. In an ideal scenario, it should look like,

Rename – C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Log\SQLAGENT.OUT to
C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Log\SQLAGENT.1

WORKAROUND/SOLUTION

It looks like the SQL Agent Log file name is missing from the string and we had to edit that to fix the issue. From my experience, I know that this information is stored in,

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL13.MSSQLSERVER\SQLServerAgent\ErrorLogFile

When we went to the above-mentioned registry location we could that the SQL Agent log file name was missing.

After we added the log file name we were able to start SQL Agent successfully.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – SQL Server Agent Not Starting – Failed to Initialize SQL Agent Log (Reason: Access is Denied).

Alongside specialists, an IT department needs people who are broadly skilled and adaptable, who can pivot quickly to new projects, slot easily into new teams, and be productive quickly. In this blog post, we will learn about database coding standards and productivity.

SQL Prompt is the industry-leading SQL IntelliSense and code-formatting tool and is an investment in developer versatility and productivity. It will both increase personal coding productivity and improve team-based development practices, leading to higher quality, more consistent and more reliable code.

Improving the design and quality of the code will speed up the development process and help minimize the occurrence of mistake that can result in the loss of code or even data.

It is sometimes best to think of SQL Prompt as if it were a Treasure Island. You tend to come across the good things by setting out to discover them. There is plenty of buried treasure for the busy full-stack programmer, but just because you’ve found one chest full of good things doesn’t mean that you’ve got everything that lies around about.

The map gives a hint of buried gold. What is in the Refactoring Redoubt, the Sands of Suggestion? Is it worth exploring the Caves of Code analysis, the Formatting Forest or the Terraces of Tab Management?

OK, you can put away your shovels and maps, because we’re going to tell you where everything is hidden, what is likely to be found, and why it is worth having.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on Driving up Database Coding Standards and Productivity with SQL Prompt

A few days ago, one of my clients contacted me to fix an error which they were getting while using maintenance plan – Integration Services evaluation period has expired. Here is the complete error message which was taken from job history.

Executed as user: NT Service\\SQLAgent$MSSQLSERVER. Microsoft (R) SQL Server Execute Package Utility Version 12.0.5000.0 for 64-bit Copyright (C) Microsoft Corporation. All rights reserved. Started: 1:00:00 AM Error: 2018-06-07 01:00:01.17 Code: 0xC0000033 Source: Backup Description: Integration Services evaluation period has expired. End Error Error: 2018-06-07 01:00:01.25 Code: 0xC0000033 Source: Backup Description: Integration Services evaluation period has expired. End Error Could not execute package because of error 0xC0000033. Started: 1:00:00 AM Finished: 1:00:01 AM Elapsed: 0.922 seconds. The command line parameters are invalid. The step failed.

In past, I have helped my clients in scenarios where they were not able to start SQL Service because the SQL Server installed using evaluation edition media which got expired. This is the first time, I saw that SQL Server was running and packages were not. I looked into the System Event Logs and I found following interesting messages from past 10 days.

Executed as user: NT Service\SQLAgent$MSSQLSERVER1. Microsoft (R) SQL Server Execute Package Utility  Version 12.0.5000.0 for 64-bit  Copyright (C) Microsoft Corporation. All rights reserved.    Started:  1:00:01 AM  Warning: 2018-04-16 01:00:02.24     Code: 0x80000036     Source: Backup Description: There are 10 days left in the evaluation. When it expires, packages will not be able to be executed.  End Warning 

Which means, SQL packages were giving warning, and no one noticed it. Finally, SSIS got expired and unable to run any package.

WORKAROUND/SOLUTION

When they contacted me my first question to them was that if they have any backup taken recently? They informed me that since the package is failing, they have no backups.

As soon as I joined GoToMeeting, I first took databases backup using T-SQL command. It is interesting to note that their SQL was running because they didn’t restart after the expiry date. I warned them not to stop SQL Service till we fix the issue as we would have longer business downtime.

The steps are straightforward, which I have already explained in my earlier blog.

SQL SERVER – Evaluation Period Has Expired – How to Activate SQL Server?

SQL SERVER – Edition Upgrade from Evaluation to Enterprise Core – AlwaysOn Warning!

As soon as we upgraded the edition of SQL Server, SSIS also got upgraded and things were back to normal.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Run Maintenance Plan. Error: Integration Services Evaluation Period has Expired

In recent past, I was assisting a client in installing SQL Server clustered instance in a Windows Cluster. There were many errors encountered and I learned a lot from this experience. In this blog, we would learn about Microsoft Cluster Service (MSCS) cluster verification errors which might appear during installation or AddNode.

We got the below error and as we can see, it was a setup “rule” failure rather than an installation error.

Setup wizard failed with the rule — Microsoft Cluster Service (MSCS) cluster verification errors. Once we have this error, we would not be able to proceed next unless we fix it or follow KB to skip this rule.

From the Detail.txt we found the below error:

<DateTime>Slp: Initializing rule : Microsoft Cluster Service (MSCS) cluster verification errors
<DateTime>Slp: Rule applied features : ALL
<DateTime>Slp: Rule is will be executed : True
<DateTime>Slp: Init rule target object: Microsoft.SqlServer.Configuration.Cluster.Rules.ClusterServiceFacet
<DateTime>Slp: Validation Report not found on Node1.
<DateTime>Slp: Validation Report not found on Node2.DomainName.com.
<DateTime>Slp: Rule ‘Cluster_VerifyForErrors’ detection result: Is Cluster Online Results = True; Is Cluster Verfication complete = False; Verfication Has Warnings = False; Verification Has Errors = False; on Machine Node1
<DateTime>Slp: Evaluating rule : Cluster_VerifyForErrors
<DateTime>Slp: Rule running on machine: Node1
<DateTime>Slp: Rule evaluation done : Failed
<DateTime>Slp: Rule evaluation message: The cluster either has not been verified or there are errors or failures in the verification report. Refer to KB953748 or SQL Server Books Online for more information.

<DateTime>Slp: Send result to channel : RulesEngineNotificationChannel
<DateTime>Slp: Initializing rule : Microsoft Cluster Service (MSCS) cluster verification warnings
<DateTime>Slp: Rule applied features : ALL
<DateTime>Slp: Rule is will be executed : True
<DateTime>Slp: Init rule target object: Microsoft.SqlServer.Configuration.Cluster.Rules.ClusterServiceFacet
<DateTime>Slp: Validation Report not found on Node1.
<DateTime>Slp: Validation Report not found on Node2.DomainName.com.
<DateTime>Slp: Rule ‘Cluster_VerifyForWarnings’ detection result: Is Cluster Online Results = True; Is Cluster Verfication complete = False; Verfication Has Warnings = False; Verification Has Errors = False; on Machine Node1
<DateTime>Slp: Evaluating rule : Cluster_VerifyForWarnings
<DateTime>Slp: Rule running on machine: Node1
<DateTime>Slp: Rule evaluation done : Warning
<DateTime>Slp: Rule evaluation message: The MSCS cluster has been validated but there are warnings in the MSCS cluster validation report, or some tests were skipped while running the validatation. To continue, run validation from the Windows Cluster Administration tool to ensure that the MSCS cluster validation has been run and that the MSCS cluster validation report does not contain errors.

We had discussed a few issues in Part 1 and Part 2 of this blog and you can find it here

• SQL SERVER – Install Error: Microsoft Cluster Service (MSCS) Cluster Verification Errors – Part 1
• SQL SERVER – Install Error: Microsoft Cluster Service (MSCS) Cluster Verification Errors – Part 2

Today we will discuss another scenario which produces the same error, but the solution is different.

Unable to find the Cluster Validation report:

If the respective file is not found in the given location, we would see it clearly mentioned in Detail.txt file as shown below:

<DateTime>Slp: Validation Report not found on Node1.DomainName.com
<DateTime>Slp: Validation Report not found on Node2.DomainName.com

If you use one of my favorite tool and a procmon trace and you will notice entries like below. The user might have deleted/moved the given file. The report is usually located under – C:\Windows\Cluster\Reports

<DateTime>
setup1xx.exe
7212 CreateFile
\\Node1.Domain.com\admin$\Cluster\Reports\Validation Data For Node Set 9431D731EF6810180D4000D550DDB48DD960F349.xml
NAME NOT FOUND

SOLUTION/WORKAROUND

Open the cluster validation report to check the results. If there are no errors found, just rename the file to the name found in the procmon (Validation Data For Node Set 9431D731EF6810180D4000D550DDB48DD960F349.xml). else re-run Windows Cluster validation to generate a new report and this new report will be used by the setup the next time when it runs.

I have written three blogs in the same series as I found multiple reasons for such behavior. Please comment and let me know if you found some more causes and ways to fix it.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Install Error: Microsoft Cluster Service (MSCS) Cluster Verification Errors – Part 3

One of my existing clients contacted me for assistance in fixing an error. Sometimes, if the error is easy and it doesn’t take more than 10-15 minute, I give them the benefit of being my existing customer and I give free consulting (not always though). They informed me that they need moving SQL logins (username and password) from one server to another. To achieve that they were using sp_help_revlogin. Here are the steps:

• Create sp_help_revlogin and sp_hexadecimal on the source server using KB
• Execute the procedure in text mode in SSMS.
• Copy the resulting SQL output (SQL Script)
• Paste output from the previous step to the destination server
• Execute the script.

Here is the error they were getting while moving logins.

Msg 15021, Level 16, State 2, Line 1
Invalid value given for parameter PASSWORD. Specify a valid parameter value.

When I looked at the password value supplied, it is a hashed value rather than text. That’s how we can see the password in sql_logins so that should be OK.

WORKAROUND/SOLUTION

While checking further, I found that they were using this method to move logins from SQL Server 2012 to SQL Server 2008 R2.  When I check on my machine the password_hash was starting with 0x01 in SQL Server 2008 R2 and in SQL Server 2012 it was starting with 0x02. Looks like that defines the SHA1 and SHA2.  I also found that error is already documented in the KB of sp_help_revlogin and we need to use a blank password in the script and set the password for the account later.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Transfer Logins Error: Msg 15021: Invalid Value Given for Parameter Password

There are so many components in SQL Server and sometimes there is an error in installing components which I have not used at all. One such component is SSRS with SharePoint. To use this, we need to install Reporting Service Add-In for SharePoint. In this blog, we would see how to fix error System.ArgumentException which was encountered during the installation of Reporting Service Add-In for SharePoint.

No matter which way I tried to install the RS Add-in I kept seeing the same errors in Detail.txt

(01) 2018-06-18 03:08:31 RS: Installing web service managed performance counters.
(01) 2018-06-18 03:08:32 RS: Installing Windows service managed performance counters.
(01) 2018-06-18 03:08:34 RS: Continue after failure in installing or reinstalling managed performance counters. – System.ArgumentException: Cannot create Performance Category with counter name Report Requests/sec because the name is a duplicate.
At system.Diagnostics.PerformanceCounterCategory.CheckValidCounterLayout(CounterCreationDataCollection counterData)
at System.Diagnostics.PerformanceCounterCategory.Create(String categoryName, String categoryHelp, PerformanceCounterCategoryType categoryType, CounterCreationDataCollection counterData)
at Microsoft.ReportingServices.Common.RSPerfCounterInstallUtil.InstallProgressiveReportPerfCounters()
at Microsoft.ReportingServices.Common.RSPerfCounterUtil.SetManagedPerfCounters(Boolean isSharePointMode)

Then I shifted my focus to the RsSharePoint log in the setup log files. I saw these errors, or you might call it “Interesting Info”.

CA MSG: Running RSCustomAction
CA MSG : Launching C:\Users\sa_admin\AppData\Local\Temp\rsCustomAction.exe with command line parameter /i
CA MSG: rsCustomAction.exe failed to configure, an Error code is: 1
CustomAction RSSP_CAInstall_64 returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)

When trying to understand what rsCustomAction.exe is, I got this article.

Basically, we have a manual way to install and configure RsSharepoint Addin.

Files-only installation

To install the files but skip the custom action phase of installation, run the rssharepoint.msi from the command line with the SKIPCA option. Open a command prompt with administrator permissions. Run the following command:

Msiexec.exe /i rsSharePoint.msi SKIPCA=1 

The installation user interface will open and run as normal and the rsCustomAction.exe file is installed. However, the .exe will not run at the end of the installation and rsCustomAction.exe will remain on the computer when the installation is completed.

Run the custom actions executable

Below are the steps.

1. Open a command prompt with administrator permissions
2. Navigate to the folder that contains the file exe

CD %temp%.

The file should be in \AppData\Local\Temp

3. Type the following command. This configuration step will take several minutes to finish. The W3SVC service will be restarted during this process. Several Status messages will be displayed as the program copies files, registers components, and runs the SharePoint Product Configuration Wizard.

rsCustomAction.exe /i

Setup Log Files

When Setup runs, it logs information to a log file in the %temp% folder for the user who installed the Reporting Services Add-in.

For example c:\Users\\AppData\Local\Temp

The file name is RS_SP_<number>.log, for example RS_SP_0.log.

Each error in the log starts with the string “SSRSCustomActionError“.

In my case when running — rsCustomAction.exe /i   we got the below errors.

The applicationcontent command is invalid or a failure has been encountered.
The server farm will not work with missing installs. Add “-cmd installcheck -noinstallcheck” to the command-line to ignore this warning.
The following is missing on <ComputerName>:
Security Update for Microsoft SharePoint Enterprise Server 2013 (KB2737989) 64-Bit Edition
Security Update for Microsoft SharePoint Enterprise Server 2013 (KB2880473) 64-Bit Edition
Security Update for Microsoft SharePoint Enterprise Server 2013 (KB2880473) 64-Bit Edition

Above is really an interesting information which looks like the cause of the failure.

WORKAROUND/SOLUTION

And, indeed it was. After installing all the missing updates, we again ran the below command. This time Reporting Service Add-In for SharePoint got installed successfully.

rsCustomAction.exe /i  

Hope, this would help someone in finding the right cause of the error.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to Install Reporting Service Add-In for SharePoint – System.ArgumentException

While I was writing my earlier blog, I came across another interesting issue with sp_help_revlogin script. In this blog we would learn about the cause of Msg 15419, “Supplied parameter sid should be binary(16)” while transferring the login using sp_help_revlogin and see how to fix it.

Here is my earlier blog about an error raised while using sp_help_revlogin. SQL SERVER – Transfer Logins Error: Msg 15021: Invalid Value Given for Parameter Password

I used sp_help_revlogin to generate create login script on SQL Server A. When I executed on SQL Server B, I received an error

Msg 15419, Level 16, State 1, Line 12
Supplied parameter sid should be binary(16).

Here is the failing query which I had from the first server.

CREATE LOGIN [superman] WITH PASSWORD = 0x0200E5F46BE65CBBE7E043 
HASHED, SID = 0x01, 
DEFAULT_DATABASE = [master], 
CHECK_POLICY = ON, 
CHECK_EXPIRATION = OFF;

SOLUTION/WORKROUND

Its clear from the error message that CREATE LOGIN command didn’t like the SID value, but I didn’t supply it manually. It came from an output of sp_help_revlogin.

Later I realized that I rename sa account to Superman and that’s why I am getting 0x01 which is the fixed SID for sa account. If I needed the same account name, then I should rename the sa on destination. Or I can also create new login superman without giving SID value in the command and make it sysadmin.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Transfer Logins Error: Msg 15419: Supplied Parameter sid Should be Binary(16) – sp_help_revlogin

When I work with customers, there are situations when I get chance to learn something from them. I was engaged with an AlwaysOn availability group engagement and got some interesting information from a customer which I am sharing here. In this blog, we would learn about how to solve event id 1135 – Cluster node ‘NodeName’ was removed from the active failover cluster membership.

Here are two “Critical” errors which you might see in System Event logs:

Event ID: 1135   

Message: Cluster node ‘N2’ was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

Event ID: 1177

Message: The Cluster service is shutting down because quorum was lost. This could be due to the loss of network connectivity between some or all nodes in the cluster, or a failover of the witness disk.  Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

Based on my knowledge about clustering, event Id 1135 indicates that the heartbeat communication failed between some nodes. It could be mostly the network connection or communication is failed among the cluster nodes. Next, the event 1177 indicates that fail-over occurred since the network connectivity between some or all nodes in the cluster, or a failover of the witness disk.

SOLUTION/WORKAROUND

Of course, your networking team needs to be engaged first to understand the root cause of network issue. If it is happening on random basis and network team has no clue about it then here are few things which DBA can also do.

$cluster = Get-Cluster
$cluster.SameSubnetDelay=2000
$cluster.SameSubnetThreshold=10
$cluster.CrossSubnetThreshold=10
$cluster.CrossSubnetDelay=4000

Along with cluster setting, one of my clients also told me to disable TCP offloading and few more properties. As per him, they might cause network delays and intermittent failures. You can run the following commands in the CMD (run as administrator) on all nodes.

Netsh int tcp set global chimney=disabled
Netsh int tcp set global rss=disabled
Netsh int tcp set global netdma=disabled
Netsh int tcp set global autotuninglevel=disabled
netsh interface teredo set state disabled
netsh int ipv4 set global taskoffload=disabled

Also, update the NIC drivers, firmware, and teaming software (if there is) on all cluster nodes.

Above steps have solved the issue for them on several servers and they gave me permission to blog. If above steps solve the issue, please comment and let them know.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Event ID: 1135 – Cluster node ‘NodeName’ was Removed From the Active Failover Cluster Membership

There are so many wait types available in SQL Server and sometimes I feel that it’s an ocean of wait types. I have already written about most common wait types earlier. In this blog, we would learn about VDI_CLIENT_WORKER / VDI_CLIENT_OTHER wait type. SQL SERVER – Introduction to Wait Stats and Wait Types – Wait Type – Day 1 of 28

Recently one of my clients showed me many background threads having wait type VDI_CLIENT_WORKER. As per them, they must restart the instance to get rid of them but after few hours, it appears again. When I was looking at the CPU and IO for this SPID, it was not increasing. All I observed is that wait type was increasing for VDI_CLIENT_WORKER and then it changes to VDI_CLIENT_OTHER.

Looking at those SPIDs we observed their start time in sysprocesses and then I looked further in ERRORLOG for the same timings. Interestingly, I could see below.

Automatic seeding of availability database ‘PROD_HRMS_DB’ in availability group ‘PRODUCTION_AG’ failed with a transient error. The operation will be retried.

SOLUTION/WORKAROUND

As soon as I saw seeding relate message, I immediately checked their AlwaysOn availability group. I found that one of the availability databases was not present on the secondary replica. So, as soon as they start SQL Service on secondary, seeding kicks off automatically.

To fix this problem, we disabled automatic seeding for this replica using the following command.

ALTER AVAILABILITY GROUP [MyAGNameGoesHere] 
    MODIFY REPLICA ON 'Name_of_secondary_replica'   
    WITH (SEEDING_MODE = MANUAL)
GO

And then we performed manual backup and restore. Here are few more blogs about Automatic seeding I wrote earlier.

• SQL SERVER – AlwaysOn Automatic Seeding – Database Stuck in Restoring State
• SQL SERVER – AlwaysOn Automatic Seeding Failure – Failure_code 15 and Failure_message: VDI Client Failed

Have you seen any such interesting issues with Always On Availability Group seeding feature?

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Many BACKGROUND Threads with Command VDI_CLIENT_WORKER

Have you ever configured database mail using Gmail account? There are many articles on the internet describing the steps. In this blog, we would learn about fixing an error “The SMTP server requires a secure connection or the client was not authenticated” which you might see when database mail is used. Note that this can also appear for non-Gmail setups also. So, here we go.

THE PROBLEM

I was working on a project with my client and they had a requirement to send alerts via database mail. They wanted to use Google’s email service i.e. Gmail for this. As soon as they configured and sent a test email, we received below error while sending a test email.

Log Database Mail (Database Mail Log)
Process ID: 2654
Mail Item ID: 23
Last Modified By: sa
Message: The mail could not be sent to the recipients because of the mail server failure. (Sending Mail using Account 4 (DateTime). Exception Message: Cannot send mails to mail server. (The SMTP server requires a secure connection or the client was not authenticated. The server response was: 5.5.1 Authentication Required. Learn more at).)

And of course, mail went to unsent/failed state.

SOLUTION/WORKAROUND

It took a while to find a solution as there were many solutions available on the internet. In our case, it was a checkbox. If you run into the same error, make sure that you check below checkbox under account setting section of database mail.

After this, we sent another test email and it was successfully delivered to the recipient.

While searching on the internet, I was also able to find another error which is fixed by the same setting. Here is the error:

Exception Message: Cannot send mails to mail server. (The SMTP server requires a secure connection or the client was not authenticated. The server response was: 5.7.0 Must issue a STARTTLS command first).’

Above error is also due to SSL checkbox.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Database Mail Error: The SMTP Server Requires a Secure Connection or the Client was not Authenticated. The Server Response Sas: 5.5.1

This is one of the most common errors which is seen my most DBA when they are using a domain account as the service account. In this blog, we would learn about the cause and solution of error The service did not start due to a logon failure while using gMSA.

Log Name: System
Source: Service Control Manager
Event ID: 7000
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: node1.sqlauthority.com
Description:
The SQL Server (MSSQLSERVER) service failed to start due to the following error:
The service did not start due to a logon failure.

Some of you might run into the same error where the password is really expired. I have blogged about that earlier.

SQL SERVER – Event ID 7000 – The service did not start due to a logon failure

SQL SERVER – FIX – ERROR – Service Logon Failure (ObjectExplorer)

Interestingly, this time the situation was little different. My client was using group managed service account (gMSA) for SQL Server service account. In such account, the password is auto-managed by the domain controller. Here are some documentation which talks about how to configure it

SQL Server 2014; Click here and check “Group Managed Service Accounts”.

SQL Server 2016; Click here and see the section under “Managed Service Accounts, Group Managed Service Accounts, and Virtual Accounts.”

While using gMSA, you don’t provide a password in configuration manager so earlier blogs won’t help.

WORKAROUND/SOLUTION

When we setup gMSA, you need to allow machines to retrieve the latest password from the domain controller. This setting is called “PrincipalsAllowedToRetrieveManagedPassword”. My client had two nodes AlwaysOn availability group. Here are the commands we have to execute in PowerShell on the domain controller.

Import-Module ActiveDirectory
Set-ADServiceAccount -Identity sqlsvcacct -PrincipalsAllowedToRetrieveManagedPassword Node1$
Set-ADServiceAccount -Identity sqlsvcacct -PrincipalsAllowedToRetrieveManagedPassword Node2$

After running above command, we were able to start SQL Service on both the nodes.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – gMSA: The Service Did Not Start Due to a Logon Failure

In the previous blog posts, we discussed the evolution of the DBA along with the challenges, changes and their upcoming trends. Data is the most critical part of any application. The changing nature of the data always brings new challenges to DBAs. New age DBAs now have to do many different tasks than before. Along with running the system at the most optimal settings, DBAs now have to get their hands dirty with development, DevOps, NoSQL technology as well as help organizations to migrate to the cloud.

In this blog post, we will discuss three new trends and challenges in the Database Administration world.

Greater Responsibility

In the earlier times when we used to consult with the DBAs, they often used to manage 3 to 9 database servers.

“68% of DBAs manage between 11-500 database instances. DBAs are managing more data and database platforms than ever before.”

Well, changing times have made the DBA’s life more stressful as now they have to take care of multiple different platforms. It is common to see DBAs continually learning new technology and features to accomplished the tasks which they have been assigned.

The evolution of the NoSQL database has further made the life of the traditional DBA very complicated as now not only they have to learn a new language and features but also adopt a new radical concept.

Always Peak Performance

Earlier time was a simpler time when DBAs just had to watch selected essential counters for the database and few key metrics for the operating systems. Based on the limited information, DBAs were providing the necessary suggestions to keep the database up. However, the time has indeed changed and now DBAs not only have to keep the database up and running, but they are also responsible for its optimal performance.

“72% DBAs say managing database performance is their most important responsibility.”

Compared to earlier times the role and responsibility of the DBAs are also changed. As we discussed earlier, now that DBAs are responsible for multiple database platforms and each platform has its methods to collect data. Once the data is collected, DBAs have to process it in such meaningful way that it makes sense for that platform and workload.

This recent changes in the job responsibility have guided many DBAs to use third party tool for monitoring the database. Multiple third-party vendors have come up with many different tools for automated routine and mechanical tasks. Many DBAs found that new tools have made their life once again simpler and now they can spend more time learning new technology.

Cloud Migration

When the cloud was first introduced, I did not find it interesting. I interpreted cloud as hosting on the server of the branded organizations. Well, it was right to a certain extent, and hence, I was never too keen to move to the cloud. However, the entire ball game got changed when Amazon and Microsoft started to compete for the cloud business.

Research says – “34% of DBAs cite increased flexibility as a motivator for moving to the cloud. The majority of DBAs surveyed are already supporting cloud databases or plan to soon.”

A few years ago, DBAs were scared of the cloud as they thought it would take their job away, but now the ecosystem is very different. On cloud platform provisioning, data storage and security are very different from what DBAs used to do on-premises.

Many of the organizations have read the writing on the walls and are moving at rapid pace to adopt cloud. This quick adoption process by organizations has created some amusing problems for DBAs.

In the earlier times, DBAs were always asking for more resources regarding IO, Memory, and CPU. However, in the cloud, their role is not about building a bigger rig. DBAs are now forced to think like a business analysis. There are often moments when DBAs have to cut the resources to strike a balance between available resources, limited budget, and business needs.

New Trends and Challenges – Call to Action

Data is essential to any organizations, and it is continuously changing, just like that life of the DBA. New trends will initially bring out the fear of unknown and plethora of presumed challenges. Once the dust settles some of the problems will become real, and some of the obstacles will disappear.

The only way to foolproof of our career as a DBA is to stay informed about the upcoming changes and challenges. As an evolving DBA, we must adapt and overcome the challenges of new technology.

To gain insight into the evolving challenges for DBAs, Quest commissioned Unisphere Research to survey DBAs and those responsible for the management of the corporate data management infrastructure. I strongly recommend you download the whitepaper DBAs Face New Challenges: Trend in Database Administration.

Additionally, here is another interesting whitepaper – A Zombies Survival Guide in a Changing Database World. Do let me know what you think of it after you read it.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on New Trends and Challenges in Database Administration

In my earlier blog, we saw one possible cause database mail error due to SSL settings. Today we will learn about another Database Mail Error.

SQL SERVER – Database Mail Error: The SMTP Server Requires a Secure Connection or the Client was not Authenticated. The Server Response Was: 5.5.1

In this blog we would learn about another cause and the solution of the database mail error:  The SMTP server requires a secure connection or the client was not authenticated. The server response was: 5.5.1. This specifically applied to Gmail accounts/SMTP Server.

After reading my blog, one of my blog readers contacted me and informed that even after changing the settings as per blog, the emails are still giving the same error.

He informed that he has created a new Gmail account and using that to send emails. Here was the exact message in the database mail log.

Date 6/30/2018 1:25:47 AM
Log Database Mail (Database Mail Log)
Log ID 8
Process ID 8388
Mail Item ID 4
Last Modified 6/30/2018 1:25:47 AM
Last Modified By sa_sqlauthority
Message
The mail could not be sent to the recipients because of the mail server failure. (Sending Mail using Account 1 (2018-06-30T01:25:47). Exception Message: Cannot send mails to mail server. (The SMTP server requires a secure connection or the client was not authenticated. The server response was: 5.5.1 Authentication Required. Learn more at).)

I double checked and found that SSL checkbox was checked. I asked him to check his mailbox in Gmail to check if there is an email. Interestingly, we saw below email

So, Google was thinking that it’s a malicious attack and hence blocked their email via google server.

WORKAROUND/SOLUTION

The email also contained below link to fix the problem.

https://myaccount.google.com/lesssecureapps

Once we login to google account, we should see below.

All you need is turn that on and try sending email again. It worked for my client.

Do you know any other problems like this? Please share via comments and help others.

Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Database Mail Error: The SMTP Server Requires a Secure Connection or the Client Was Not Authenticated. The Server Response Was: 5.5.1 – Using Gmail

I get many emails every day and I do respond to them as soon as I can.  Sometimes they are about the business and sometimes they are from my blog readers who have followed my other blog and need help with something else. Most of the time, I do have a solution for them. In this blog, we would learn about how to open SQL Server Configuration Manager when there is no shortcut in start menu.

One of my blog readers sent below email.

Hi Pinal,
Sorry for the direct email rather. You will not know me, but I have been taking help from you via your blogs for many years. I am sure there are many in the world who are getting benefited from the knowledge which you share via daily blogs. I would like to thank you on behalf of everyone who has been successful due to your blogs.

I wanted one assistance from you as I didn’t see any blog about the problem which I am facing as of today. I have been using SQL Server Configuration Manager to manage SQL Server related services. Today, due to some custom patching from my company, I saw that I don’t have SQL Server Configuration Manager on any of the server. I am not sure where it is gone.

Do you know how to get it back?

Thanks.

WORKAROUND/SOLUTION

I did some search and found that it is a snap-in in MMC. There are multiple ways to open it. Here are the steps

1. Start > Run > MMC
2. It would open below. Choose File > Add/Remove Snap-in.
   
3. Open the snap-in window opens, we should choose “SQL Server Configuration Manager” from the list and hit “Add”
   
4. It should be in the “Selected snap-ins” as seen in the following image.
   

That should be it. Hit OK and you should get SQL Server Configuration Manager.

If you want a shortcut then below is the command line which would open SQL Server Configuration Manager for SQL Server 2017. You can right click and create a new shortcut with below command.

C:\Windows\SysWOW64\mmc.exe /32 C:\Windows\SysWOW64\SQLServerManager14.msc

The last piece is the version of SQL Server installed on the machine. Version 14 represents, SQL Server 2017. Here is the version list as of today.

Have you encountered this same issue earlier? Is there any other solution? Please comment and share with others. Reference: Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Unable to See SQL Server Configuration Manager. How to Open it?

I have been seeing many customers moving their SQL Server to Cloud. Few of them have chosen Microsoft Azure platform to host their production server and few others have selected other cloud platforms like AWS. When they host SQL in Azure, they can choose various backup methods. In this blog, we would learn about fixing an interesting error “Account Verification Failed” while using Microsoft SQL Server Backup to Microsoft Azure Tool.

This is a free tool from Microsoft which can be downloaded from the link here Microsoft SQL Server Backup to Microsoft Azure Tool

One of my clients has downloaded and installed it on the SQL Server Azure Virtual Machine. Here is how they were trying to configure. As soon as they launch, we get below screen.

Click on “Add” and provide a backup path and file extension.

Hit next and provide storage account details.

As soon as we click on “Verify account” we get an error – Account verification failed. This is not a very informative error as we have triple checked the account name, keys, and container multiple times.

WORKAROUND/SOLUTION

We used another account and it was working fine. Then we compared each property of both storage accounts and found that this was caused due to below setting – “Secure transfer required”. In good case it was disabled and in bad case it was enabled.

Once we made it “Disabled” we were able to proceed.

Have you encountered such error where there is no meaningful solution in the message? I hope Microsoft fixes it soon.

Another cosmetic issue: If you look at the tool installer and usage it says “Microsoft SQL Server Backup to Windows Azure Tool” whereas download link says “Microsoft SQL Server Backup to Microsoft Azure Tool”. This is something like what I blogged here SQL SERVER – SSMS 17.7 Observation – Two Miscellaneous Category in Database Property Options

Hope it would reach Microsoft team via this blog and they provide some meaningful error message.

Pinal Dave (https://blog.sqlauthority.com)

First appeared on SQL SERVER – Backup to Microsoft Azure Tool – ERROR: Account Verification Failed