SQL SERVER – Always On Replica Disconnected After Changing SQL Server Service Account

In my lab environment, I was testing a script to change service account and it worked fine. I had Always On configured on this SQL Server and soon I realized that after changing the SQL Server service account, the secondary replica went into a disconnected state.

We could see the below errors in the Primary Replica SQL Errorlog

<DateTime> Logon  Database Mirroring login attempt by user ‘MyDC\SQLAccount’ failed with error: ‘Connection handshake failed. The login ‘MyDC\SQLAccount’ does not have CONNECT permission on the endpoint. State 84.’.  [CLIENT: 192.168.x.x]

If we change the service account back to the old one, everything goes back to normal. There are no issues. So, it seemed like that there were some issues with the new account we were using and apparently AlwaysOn AG did not like that account. The error also says that the account does not have “connect” permission to the endpoints. We ran the below query to check who all have got permissions on the AlwaysOn Endpoint — Hadr_endpoint

SELECT e.name AS mirror_endpoint_name
	,s.name AS login_name
	,p.permission_name
	,p.state_desc AS permission_state
	,e.state_desc endpoint_state
FROM sys.server_permissions p
INNER JOIN sys.endpoints e ON p.major_id = e.endpoint_id
INNER JOIN sys.server_principals s ON p.grantee_principal_id = s.principal_id
WHERE p.class_desc = 'ENDPOINT'
	AND e.type_desc = 'DATABASE_MIRRORING'

The output looked like below:

We had changed the SQL service account to — ‘MyDC\SQLAccount’. The old one was ‘MyDC\SQLSvcAccount’

Looks like we were missing the CONNECT GRANT on the EndPoint permission here. We performed the following steps that resolved the issue.

WORKAROUND/SOLUTION

1. Created login of the newly added service account on both replicas.

USE [master]
GO
CREATE LOGIN [MyDC\SQLAccount] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
GO 

2. Granted connect permission the endpoints on both replicas.

GRANT CONNECT ON ENDPOINT::hadr_endpoint TO [MyDC\SQLAccount]
GO

3. Stopped the endpoints on both the replicas.

ALTER ENDPOINT hadr_endpoint STATE=STOPPED

4. Started endpoints on both the replicas.

ALTER ENDPOINT hadr_endpoint STATE=STARTED

After making the above changes replicas were back in the connected state. We tested the failovers and it worked great. Have you faced a similar issue with AlwaysOn AG? Please share your experience via comments.

Reference: Pinal Dave (https://blog.sqlauthority.com) 

First appeared on SQL SERVER – Always On Replica Disconnected After Changing SQL Server Service Account