Part 1
So recently a client had some WAN issues where the WAN kept dropping packets on the primary link, meaning some network traffic would work fine and other network traffic would fall off the face of the planet!!
Unfortunately the system affected the most was Exchange 2010!! The site had Exchange 2010 DAG set-up between two sites. One on Site A and another on another site B with the File share witness on Site A. So because some traffic was flowing nicely and some wasn’t some of the cluster services were up and some went down meaning the DAG got in a pickle!
Site B ended up taking ownership of the DAG and wouldn’t let go because he file share witness wasn’t available. So as people could still get mail, I patiently waited for the WAN to be fixed (in the end someone pulled the primary link which forced the secondary to take over and everything became hunky dory).
So now the WAN was back and happy but still the cluster service wouldn’t appear, so after digging the arp entries were missing (no idea why), so ended up creating another IP for the cluster service….Done. It’s up, time to reseed!
Update: Well shit me, MS know about this:
http://support.microsoft.com/kb/2743761
Well anyway I wish the reseed was that easy! During the issues, the mail store was un-mounted and now it won’t mount!! Booo, so another admin decided to run eseutil and recover the DB which worked but then decided to seed the wrong way meaning site B now has an old mail store with no mail from that day!! BUGGER!
Part 2
So I need to rerun all the mail and then import and luckily all the missing mail logs are still on Site B!
I know what you’re thinking:
Restore the EDM file from a backup,
Rerun the Logs via ESEUTIL,
Mount the recovery DB,
Merge users,
Bam.
Well the business was very panicky now and didn’t want anyone touching anything on Exchange till things settled down, fair enough a day’s worth of mail is missing at the moment so people are touchy.
So I decide to make use of the most excellent Veeam feature SureBackup, which will allow me to restore the entire exchange live setup including a DC in lab environment without touching the live environment!! Brilliant!
So I set it up, add a host, some storage, make sure I have a good backup of Exchange and it runs, great!! Be careful though as it uses snap shots heavily so every change adds to a delta file meaning storage gets eaten!! So here I am with the live environment in a lab environment, there some issues as I cant restore site B, so I have to remove the DAG and the other site which I do via ADSI edit (quick and easy but dangerous)!
So I copy over the recovered DB I restored with file level restore on veeam (copied by creating a HD in ESX and mounting it on veeam and copying file to it then mounting in my lab),
Create a recovery DB:
new-mailboxdatabase –recovery –name databasename –server servername –edbfilepath pathtodatabase –logfolderpath pathtologfile
Checked the logs:
Eseutil /ml “d:\logs\E00”
Replayed:
Eseutil /r E00 /d d:DB /l “d:\LOGS” /i /s “d:\LOGS”
(May of run ESEUTIL /p too, cant remember)
Mounted!
Emails are all there but in the disconnected items section, reconnect to a test db and GREAT the emails are there!!
So connect all the mailboxes (manually unfortunately), then export them to pst (to my mounted HD):
New-MailboxExportRequest -ContentFilter {((Received -le '12/12/2012') -and (Received -ge '01/12/2012')) -or ((Sent -le '12/12/2012') -and (Sent -ge '01/12/2012'))} -Mailbox “User1” -Name MailboxNameExp -FilePath \\Server\psts\User1.pst
Un-mounted the HD, added the HD to the live environment and imported the PSTS:
Dir \\Server\psts\*.pst | %{ New-MailboxImportRequest -Name BulkPSTImport -BatchName Recovered -BadItemLimit 250 -AcceptLargeDataLoss -Mailbox $_.BaseName -FilePath $_.FullName} >C:\Scripts\resultsBulkCalendar.txt 2>&1
