Tech Blog

CAG drops off network after updates to ESXi hosts, reboot fixes for a short time but then it drops off again.

Background

https://communities.vmware.com/thread/492945

Detailed fix in Citrix KB

http://support.citrix.com/article/CTX200278

Came across this today, if you have a VMDK file that is over 1.98TB you will not be able to create a snapshot of the server. Please see articles below.

Essentially you need to allow for a 16GB overhead so the maximum size the VMDK can be is 2032GB, 1.99TB calculates to 2037GB and 1.98TB calculates to 2027GB.

http://www.virtualtothecore.com/en/the-real-maximum-size-of-a-vmdk-file/

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1012384

Part 1

So recently a client had some WAN issues where the WAN kept dropping packets on the primary link, meaning some network traffic would work fine and other network traffic would fall off the face of the planet!!

Unfortunately the system affected the most was Exchange 2010!!  The site had Exchange 2010 DAG set-up between two sites.  One on Site A and another on another site B with the File share witness on Site A.  So because some traffic was flowing nicely and some wasn’t some of the cluster services were up and some went down meaning the DAG got in a pickle!

Site B ended up taking ownership of the DAG and wouldn’t let go because he file share witness wasn’t available.  So as people could still get mail, I patiently waited for the WAN to be fixed (in the end someone pulled the primary link which forced the secondary to take over and everything became hunky dory).

So now the WAN was back and happy but still the cluster service wouldn’t appear, so after digging the arp entries were missing (no idea why), so ended up creating another IP for the cluster service….Done.  It’s up, time to reseed!

Update: Well shit me, MS know about this:

http://support.microsoft.com/kb/2743761

Well anyway I wish the reseed was that easy! During the issues, the mail store was un-mounted and now it won’t mount!! Booo, so another admin decided to run eseutil and recover the DB which worked but then decided to seed the wrong way meaning site B now has an old mail store with no mail from that day!!  BUGGER!

Part 2

So I need to rerun all the mail and then import and luckily all the missing mail logs are still on Site B!

I know what you’re thinking:

Restore the EDM file from a backup,

Rerun the Logs via ESEUTIL,

Mount the recovery DB,

Merge users,

Bam.

Well the business was very panicky now and didn’t want anyone touching anything on Exchange till things settled down, fair enough a day’s worth of mail is missing at the moment so people are touchy.

So I decide to make use of the most excellent Veeam feature SureBackup, which will allow me to restore the entire exchange live setup including a DC in lab environment without touching the live environment!!  Brilliant!

So I set it up, add a host, some storage, make sure I have a good backup of Exchange and it runs, great!!  Be careful though as it uses snap shots heavily so every change adds to a delta file meaning storage gets eaten!!  So here I am with the live environment in a lab environment, there some issues as I cant restore site B, so I have to remove the DAG and the other site which I do via ADSI edit (quick and easy but dangerous)!

So I copy over the recovered DB I restored with file level restore on veeam (copied by creating a HD in ESX and mounting it on veeam and copying file to it then mounting in my lab),

Create a recovery DB:

new-mailboxdatabase –recovery –name databasename –server servername –edbfilepath pathtodatabase –logfolderpath pathtologfile

Checked the logs:

Eseutil /ml “d:\logs\E00”

Replayed:

Eseutil /r E00 /d d:DB /l “d:\LOGS” /i /s “d:\LOGS”

(May of run ESEUTIL /p too, cant remember)

Mounted!

Emails are all there but in the disconnected items section, reconnect to a test db and GREAT the emails are there!!

So connect all the mailboxes (manually unfortunately), then export them to pst (to my mounted HD):

New-MailboxExportRequest -ContentFilter {((Received -le '12/12/2012') -and (Received -ge '01/12/2012')) -or ((Sent -le '12/12/2012') -and (Sent -ge '01/12/2012'))} -Mailbox “User1” -Name MailboxNameExp -FilePath \\Server\psts\User1.pst

Un-mounted the HD, added the HD to the live environment and imported the PSTS:

Dir \\Server\psts\*.pst | %{ New-MailboxImportRequest -Name BulkPSTImport -BatchName Recovered -BadItemLimit 250 -AcceptLargeDataLoss -Mailbox $_.BaseName -FilePath $_.FullName} >C:\Scripts\resultsBulkCalendar.txt 2>&1

Had an annoying error today while trying to add a ESX 4.1 host to DataCenter:

Call “datacenter.queryconnectioninfo” for object “DC or Cluster Name” on vCenter Server “vCenter” failed.

Firstly I thought my password may be wrong for the host, it wasn’t!

Then, maybe my new host is in maintenance mode, it wasn’t!

Finally, and really it should of been obvious to me:

I’d changed the IP address twice while building the box via the console and had forgotten to enter a new default gateway!

And Bam, added the host to the Datacenter!!

So I have a test ESX 4.1.0 server that I use for all sorts of test servers and what not!

As its a test server that I wont be keeping forever I’m using the trail version of ESX (there’s a few reasons I’m not using esxi but I cant be arsed to get into it!)… So every now and again I need to extend the license for ESX and here’s how I do it:

Log onto the ESX console as root

Go to:  cd /etc/vmware
Then Delete vmware.lic and license.cfg with the following commands:
rm -r vmware.lic
rm -r license.cfg

Reboot the server with the following command:

Reboot

And thats it!!