For most organizations the main benefit of deploying an Exchange Server 2010 Client Access Server array is to minimize downtime.
So when it comes time to update the CAS array members with patches, update rollups or service packs, the update process needs to be managed in a way that prevents all of the CAS array members from being offline at the same time.
Typically this means installing the updates to CAS array members one at a time, allowing each one to complete the update and (if necessary) reboot before updating the next member.
This tutorial demonstrates how to update a Client Access Server array without causing the entire array to go offline at once. For this tutorial Update Rollup 4 for Exchange Server 2010 RTM is being installed.
Preparing the NLB Cluster for Updates
The first step is to remove the server that is about to be updated from the Network Load Balancing (NLB) cluster.
There are two ways to take a CAS array member our of the NLB cluster:
- Issue a Stop command to the server
- Issue a Drainstop command to the server
The difference between the two is that Stop will immediately stop the server regardless of who is currently connected to it, while Drainstop will put the server in a state where it will not accept new connections but will continue serving existing connections until they disconnect.
For urgent updates a Stop command may be necessary, but for planned maintenance a Drainstop has the least potential impact on active client connections to the CAS array.
To issue a Drainstop launch Network Load Balancing Manager, right-click on the desired server, choose Control Host and then Drainstop.
When the server has no more active connections it will be in a stopped state.
Right click the server and choose Properties. Set the default state of the server to Stopped. This will prevent it from automatically starting and accepting client connections after any reboots that the updates require, to allow you time to verify the updates were successful first before rejoining the NLB cluster.
Stop Conflicting Services
The Client Access Server role is often installed on the same server as the Hub Transport server role, even when deployed as a CAS array.
Hub Transport servers often run additional applications such as antivirus and anti-spam software that hooks into the Exchange Server services. These can cause conflicts with Exchange Server updates, for example if a third party application tries to automatically restart a service that it depends on that has been stopped by the update process.
Forefront is one example of this, so for servers running Forefront Protection for Exchange those services can be stopped using FSUtility.
C:\> fsutility /disable
Disabling Monitoring
If the CAS array members are monitored using SCOM or another system this should also be disabled, or placed into maintenance mode before the update is performed. This prevents unnecessary alarms in the monitoring system due to stopped services or server restarts, and also prevents the monitoring agent from trying to perform any automatic remediation such as restarting services.
Backing Up the Server
Some organizations will require an ad-hoc backup be run of at least one CAS array member before updates are applied. Others will be happy to rely on the latest scheduled backup instead. And some will even be satisfied that multiple CAS array members exist and so if a bad update puts one of them out of action there is no outage to end users, and the server can simply be manually reinstalled.
Updating the Server
Install the update following the procedure for that update type.
Update rollups come in the form of a .MSP file (Windows Installer Patch) that is applied to the server. Simply double-click the file or launch it from a command line window.
Service packs are a complete reissue of the Exchange Server setup files and are installed by running setup in upgrade mode, which can be run in either graphical or command line mode.
C:\> setup /m:upgrade
Both update rollups and service packs can take some time to install, so plan a large window of time for these updates.
Verifying the Update
After the update has completed, and if necessary the server rebooted, you should check the server’s health before placing it back into production in the CAS array.
Event Logs – look for error or warning events that have started since the update was applied.
Setup Logs – service packs write a complete setup log file to C:ExchangeSetupLogs
Services – check the Exchange services are running (or at least those that you expect to be running, some such as IMAP and POP will be stopped if you have not explicitly enabled them)
[PS] C:\>get-service *exchange* Status Name DisplayName ------ ---- ----------- Running MSExchangeAB Microsoft Exchange Address Book Running MSExchangeADTop... Microsoft Exchange Active Directory... Running MSExchangeAntis... Microsoft Exchange Anti-spam Update Running MSExchangeEdgeSync Microsoft Exchange EdgeSync Running MSExchangeFBA Microsoft Exchange Forms-Based Auth... Running MSExchangeFDS Microsoft Exchange File Distribution Stopped MSExchangeImap4 Microsoft Exchange IMAP4 Running MSExchangeMailb... Microsoft Exchange Mailbox Replication Stopped MSExchangeMonit... Microsoft Exchange Monitoring Stopped MSExchangePop3 Microsoft Exchange POP3 Running MSExchangeProte... Microsoft Exchange Protected Servic... Running MSExchangeRPC Microsoft Exchange RPC Client Access Running MSExchangeServi... Microsoft Exchange Service Host Running MSExchangeTrans... Microsoft Exchange Transport Running MSExchangeTrans... Microsoft Exchange Transport Log Se... Stopped msftesql-Exchange Microsoft Search (Exchange) Running vmickvpexchange Hyper-V Data Exchange Service
Returning the Server to Production
If the update was successful and the server healthy then it can be placed back into production.
Re-enable services such as Forefront Protection for Exchange.
C:\> fsutility /enable
Start the server in the NLB cluster.
Set the NLB initial host state to Started.
And re-enable monitoring agents and alarms for the server.
After the first CAS array member has been successfully updated you can move on to the next one.
Thanks a ton for your article, excellent information.
I’m running Exchange Server 2010 SP3 & I have two separate servers one facing internet (CA/HUB) installed & The other ( HUB,CA,Mailbox,) . both version 14.3(buids 123.4)
I am trying to install Exch 2016 & installation fails it does not recognize first server as service pack installed. it’s end up saying ***All Exchange 2010 servers in the organization must have Exchange 2010 Service Pack 3 or later installed.**
which both servers I have already updated . appreciates your expert advise on this . thx
would i need to consider mail queues if patching a HUB/CAS server in an array? or should they just behave?
Pingback: February 2016 Updates for Exchange 2016/2013/2010/2007
I’m running Exchange Server 2010 SP3. How do I move the DAG Witness to another server?
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Use the Set-DatabaseAvailabilityGroup cmdlet. Refer to TechNet for the parameters to use when setting the FSW.
Hello Paul
Good article on HUB / CAS update
We are planning to change the 2 X 1 GB NICs with 10 GB NICs on HUB / CAS servers which are in Windows NLB unicast. We do the changes one server at a time. Here do we need to remove the server from NLB before starting the activity and add it back once we complete the activity or simply change the NIC is enough? The IPs will same
Thank you
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
I don’t know. I haven’t touched NLB in a long time and have never had to replace the NIC on an NLB cluster member. Personally I would probably err on the side of caution and remove the member from the NLB cluster, then add it back in after the NIC has been changed. But you should look into it further.
Pingback: Exchange Server 2010 SP3 Update Rollup 11 Released
What about if you have a multi role CAS/HT and NLB is not used. The load balancers are hardware firewalls, do we need to remove it CAS/HT from NLB, or we can just proceed to update one at a time without removing form LB?
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Do the equivalent drain/stop in your load balancer.
Pingback: New Updates for Exchange Server 2013, 2010 and 2007
Hi Paul,
Excellent article as usual.
We are still on Exch 2010 SP1 and are ready to upgrade. Do we need to apply each SP in order ; SP2, then SP3? Or can we install SP3 only (assuming it contains all the updates for SP2)? And would we install the Rollups as well?
We have 2 edge, 2 cashub using NLB and 2 mbx DAG using windows clustering at one site and then the same at another site. DAGs don’t share across site.
Thanks for any guidance,
Wes
Forgot to ask. If we need to install each SP, then do we need to install each on all servers across all sites before moving to the next SP install?
Thanks again,
Wes
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Service Packs and Update Rollups are cumulative. You don’t need to install each SP in turn, you can go straight to the latest SP. Similarly with URs, you can go straight to the latest UR for that SP.
So in your case you would be deploying SP3, followed by UR7.
You don’t need to upgrade every server in the org with SP3 before rolling out UR7. You could do SP3 + UR7 on each server as part of the same maintenance window.
Some reading for you:
https://www.practical365.com/installing-exchange-server-2010-service-pack-3/
Hi Paul,
Interesting, I though NLB is no longer working with CAS 2010 anymore. We are running with F5 hardware LB now and constantly have issue with that. Can we use windows NLB (Windows 2008 R2 SP1) with Exchange 2010 SP2 now? Please let me know if there is any issue.
Thanks as always,
Harry
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
The NLB story for Exchange 2010 hasn’t changed.
Your F5 appliance is a better solution than NLB. I suggest fixing your F5 config or contacting F5 support to help you out with it.
What sort of problem that you have with F5 ? Please share it here.
My 2x HT/CAS Exchange Server 2007 SP3 still having a problem with F5 hardware load balancer.
The problem is that it cannot send email with more than 100 recipients, so bulk email sending is still the problem here. no solution so far apart from sending the email in multiple times in a chunk of 99 recipients.
Pingback: Exchange 2010 Service Pack 3 Update Rollup 7 Released
Pingback: Exchange 2010 - Upgrade SP1 to SP3 | Notes @ Budak Kuala
Pingback: Exchange 2010 Service Pack 3 Update Rollup 6 Released
After Drain stop and stopping the server in Cluster, when i updated the server, and attempted to start host in NLB, and cause to automatically stop another host in the cluster
?
thank you for your nice post
Pingback: Update Rollup 1 for Exchange Server 2010 SP3 Released
Pingback: Update Rollup 4 para Exchange 2010 SP2. - CiudadanoZero
Pingback: Installing Exchange 2010 SP3 | Jack Tracey Online
Pingback: Updating Client Access Servers to Exchange 2010 SP3 | wintelvm
Pingback: Installing Exchange Server 2010 Service Pack 3
Hi Paul
Nice consie doc for exchnage rollups, any guidance/process on Windows, IIS and Exchange patches together and how the affect one another(Excluding the lengthy KB articles).
Sorry for spamming but how long would it take for the NLB to activate the cas server again? It is now converging with status pending.. and it feels like it is either taking its time or quite its job and when awol.
I read the fsutility /disable step but unless my eyes are decieving me there is no /enable step mentioned?
Other then that it is a excellent how-to. Kudo’s, +1, like and what else the kids nowadays hand out to say thanks 🙂
Okay i admit.. too late too little sleep + lack of coffee.. i read the fsutility /enable now *sheepish grin*
Hi
I am planning to upgrade exchange server( 2 MB, 2 Hub/Cas) from sp1 to sp2 , rollup 4. Is there any important things to be considered before upgrading.. Why iam asking this now my exchange working fine without no issues but for safety purposes to match the latest patch iam planning to upgrade…
Please give me your suggestion…
Thanks in advance…..
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Yes, every Service Pack and Update Rollup contains some “gotchas”.
I’d recommend reading the release notes that Microsoft publishes with each one, reviewing the fixes that are included and the new features that are added.
Pingback: Update Rollup 7 for Exchange Server 2010 Service Pack 1
Pingback: Update Rollup 4 for Exchange 2010 Service Pack 2
Help!! I just noticed my CAS/HUB servers are running SP1 Rollup 2 but I must have forgot to apply RU 2 to my MBX servers, I can just do passive node first, failover, then active node? I know there is a particular order when it comes to updating Exchange so I just wanted to confirm, should be CAS, then HUB and finally MBX right? Thanks!!!!
Hi Paul,
After givng “drainstop” command, how much time do we need to wait? In my case NLB is saying “stopping” but it is not going to “stopped”.
Thanks
Vignesh
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Drainstop will wait until any active client connections have closed. So if you’ve still got clients connected it won’t finish stopping that server.
Pingback: How to Install Updates on Exchange Server 2010 CAS Arrays | Insiko – WebBlog
Pingback: Update Rollup 3 for Exchange 2010 Service Pack 2
Pingback: Update Rollup 2 for Exchange 2010 Service Pack 2
Great article,
I only think you made a little typo error.
It should be fscutility /disable (instead of fsutility).
greetz
Yves
Pingback: Exchange Server 2010 SP2 Update Rollup 1
Pingback: Upgrading to Exchange Server 2010 Service Pack 2
Paul
I’m having a big issue here and I’m pretty much out of MS support hours – I created a DAG on my mailbox server Friday night, everything worked fine, sat at 9:54 at my BB goes down and I’m basically being told by BB support that its a MS issue. My BES, is resolving the CAS/ names and IP address but it resolves it to the CASE ARRAY name correct, well I tried adding a host file on my BES and none of this worked, my outlook suddenly started doing te same thing on Sat, where it is resolving to the CAS ARRAY name – can you shed some light on this for me please
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Brenda, I don’t really understand your question the way it has been written. But it seems like it doesn’t relate to the article above, so I would suggest that you start a thread in the forums instead.
Pingback: Microsoft Releases Update Rollup 6 for Exchange Server 2010 SP1
Pingback: Microsoft Exchange 2010 / Rollup 4 SP1. « DÃa a DÃa con la TecnologÃa
Pingback: Update Rollup 5 for Exchange 2010 SP1 Released
Pingback: Exchange Server 2010 SP1 Update Rollup 4 Released
Pingback: Update Rollup 3 for Exchange 2010 SP1 and Exchange 2007 SP3 - Exchange Server Pro
Pingback: Upgrading to Exchange Server 2010 Service Pack 1 | Exchange Server Pro