Database availability groups are reasonably smart and robust systems, but they do need monitoring, or else you might find one day they are not providing the HA that you need them to. One of the useful PowerShell cmdlets for keeping an eye on things is Test-ReplicationHealth. It can be used to help troubleshoot database availability groups by running it locally or against a remote server.
[PS] C:\>Test-ReplicationHealth Server Check Result Error ------ ----- ------ ----- E15MB2 ClusterService Passed E15MB2 ReplayService Passed E15MB2 ActiveManager Passed E15MB2 TasksRpcListener Passed E15MB2 TcpListener Passed E15MB2 ServerLocatorService Passed E15MB2 DagMembersUp Passed E15MB2 ClusterNetwork Passed E15MB2 QuorumGroup Passed E15MB2 DatabaseRedundancy Passed E15MB2 DatabaseAvailability Passed E15MB2 DBCopySuspended Passed E15MB2 DBCopyFailed Passed E15MB2 DBInitializing Passed E15MB2 DBDisconnected Passed E15MB2 DBLogCopyKeepingUp Passed E15MB2 DBLogReplayKeepingUp Passed
To test a remote server, use the -Identity parameter.
[PS] C:\>Test-ReplicationHealth -Identity E15MB2 Server Check Result Error ------ ----- ------ ----- E15MB2 ClusterService Passed E15MB2 ReplayService Passed E15MB2 ActiveManager Passed E15MB2 TasksRpcListener Passed E15MB2 TcpListener Passed E15MB2 ServerLocatorService Passed E15MB2 DagMembersUp Passed E15MB2 ClusterNetwork Passed E15MB2 QuorumGroup Passed E15MB2 DatabaseRedundancy Passed E15MB2 DatabaseAvailability Passed
The cmdlet also accepts pipeline input, however if you were to simply pipe Get-MailboxServer into it and you have Mailbox servers in the organization that are not DAG members then you risk seeing errors in your results that just get in the way. Instead you can pipe only the members of a database availability group into Test-ReplicationHealth using the following method:
[PS] C:\>Get-DatabaseAvailabilityGroup | select -ExpandProperty:Servers | Test-ReplicationHealth
There’s a lot of output to look at, so it’s often easier to filter the results to only those that have not passed.
[PS] C:\>Get-DatabaseAvailabilityGroup | Select -ExpandProperty:Servers | Test-ReplicationHealth | Where {$_.Result.Valu
e -ne "Passed"}
Server Check Result Error
------ ----- ------ -----
EX2016SRV2 DatabaseAvailability *FAILED* Failures:...
EX2016SRV1 DatabaseAvailability *FAILED* Failures:...
The error details are usually truncated, so outputting the results as a list will let you see more information.
[PS] C:\>Get-DatabaseAvailabilityGroup | Select -ExpandProperty:Servers | Test-ReplicationHealth | Where {$_.Result.Val
e -ne "Passed"} | Format-List
By the way, if you’re curious about what each of the tests do, there’s a “CheckDescription” provided for each of the checks that Test-ReplicationHealth performs. Some of them are still a bit vague, but there’s some useful info there that can help point you in the right direction to investigate further.
[PS] C:\>Test-ReplicationHealth | Format-List Check*
Check : ClusterService
CheckDescription : Checks if the cluster service is healthy.
Check : ReplayService
CheckDescription : Checks if the Microsoft Exchange Replication service is running.
Check : ActiveManager
CheckDescription : Checks that Active Manager is running and has a valid role.
Check : TasksRpcListener
CheckDescription : Checks that the Tasks RPC Listener is running and is responding to remote requests.
Check : TcpListener
CheckDescription : Checks that the TCP Listener is running and is responding to requests.
Check : ServerLocatorService
CheckDescription : Checks that the Server Locator Service is running and is responding to requests.
Check : DagMembersUp
CheckDescription : Verifies that the members of a database availability group are up and running.
Check : MonitoringService
CheckDescription : Checks that the Monitoring WCF Service is running and is responding to requests. This endpoint is
hosted inside the DAG Management service.
Check : ClusterNetwork
CheckDescription : Checks that the networks are healthy.
Check : QuorumGroup
CheckDescription : Checks that the quorum and witness for the database availability group is healthy.
Check : FileShareQuorum
CheckDescription : Verifies that the path used for the file share witness can be reached.
Check : DatabaseRedundancy
CheckDescription : Verifies that databases have sufficient redundancy. If this check fails, it means that some
databases are at risk of losing data.
Check : DatabaseAvailability
CheckDescription : Verifies that databases have sufficient availability. If this check fails, it means that some
databases are at risk of losing service.
[adrotate banner=”48″]
HI Paul,
I hope you’ll find time to response for my request.
I’ve a reccurent error under exchange 2013 : “OUTLOOK.PROTOCOL UNHEALTHY”.
I made many research with no success to find an issue with.
I hope you’ll have ones
Brgds
Hi Paul, Thank you for the time you take to write these articles. Not only are they clear, but also help with getting a better understanding of the PowerShell cmdlets.
Nien.
Hi Paul,
thanks once again for excellent work. Just like the test-exchangehealth script, I would like to have this output to an email that my team could get say twice a day, i.e. at the typical working start time of the day and before we go home for the day.
Any assistance to help generate it will be greatly appreciated.
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Sending any PowerShell output as an email is pretty simple really.
https://www.practical365.com/powershell-how-to-send-email/
Or use my Test-ExchangeServerHealth.ps1 script.
How about the “Database Availability Group Member Health” column like:
Cluster Service
Replay Service
Active Manager
Tasks RPC Listener
TCP Listener
Server Locator Service
DAG Members Up
Cluster Network Quorum Group
File Share Quorum
Database Redundancy
Database Availability
DB Copy Suspended
DB Copy Filed
DB Initializing
DB Disconnected
DB Log Copy Keeping Up
DB Log Replay Keeping Up
the result is n/a, even though when I check manually passed
Hi Paul,
Would you happen to have the firewall ports required for this to work? We have an exchange site behind a firewall and not getting any results when I run this test.
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
I don’t. Try a wireshark trace?
Pingback: Turning to a PAL when troubleshooting Exchange… |
Pingback: Get your brand featured on snapback shouts
Hi Paul, what are the next steps if you get a failure from Test-ReplicationHealth? We are running Exchange 2010 and despite seeing no ill effects all 12 members of our three DAG’s are now failing with:
The health test of the Microsoft Exchange Server Locator Service on server ‘DAG-MEMBER’ failed.
Error: Server Locator Service call had a communication error.
Searching the for this error it does’t seem to appear in any MS documentation so we’re at a bit of a loss of how to proceed before contacting MS.
Many thanks, Ryan.
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
According to TechNet that…
“Verifies the Active Manager client/server processes on DAG members and on the Client Access Server that perform lookups in Active Directory and Active Manager to determine where a user’s mailbox database is active.”
I don’t see any info out there on what to do about it. Are there any other event log entries that might be relevant? Particularly events that may be logged when the server restarts?
Thanks for the advice Paul, I tried rebooting a DAG member but couldn’t see anything useful so investigated further…
This issue only came to light because we run Test-ReplicationHealth for each node daily and email an alert on failure. The Server Locator Service error first appeared last month and then disappeared before returning last week. It would always affect all servers or none.
We are in the process of updating from SP2 to SP3 and it seems the Test-ReplicationHealth results vary depending on where you run the cmdlet. If run on SP3 we don’t see ClusterNetwork, QuorumGroup and FileShareQuorum checks but gain the ServerLocatorService failure.
The server running the daily health checks would connect to a different Exchange server each day thus we got differing results. So far we’ve only upgraded our CAS and HT servers so hopefully once we complete the mailbox servers the ServiceLocatorService will report healthy.
Thanks again for initial advice though Paul 🙂
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
I suppose that would make sense that running from a down level server might throw some odd results for higher SP level servers. Though I’ve never run into that issue myself, because I generally run monitoring scripts from the highest version/SP level in the org.