Tech Blog

I have recently logged a call with Veeam as we had jobs failing with a message of “Unable to remove backup file by retention policy, because it still contain restore points”

They have now got back to me with a new update for Veeam 2b which has been released yesterday as per this KB: http://www.veeam.com/kb2024

As you can see there are lots of significant changes included within this update.

Doing a FLR (File Level Restore) of a file that is on a deduped VM is only possible if the system running VB&R is installed on an identical or higher OS that has dedupe enabled. Meaning you can only restore a file which is on a Win2012 deduped volume if Veeam is also on Win2012 or 2012 R2 with dedupe enabled. You can’t restore a file from a Win2012 R2 volume if Veeam is installed on a Win2012 because the dedupe driver is different in R2. It is enough to install the Deduplication Role Service, as long as the service is running on the Veeam server file level restores will work.

I knew something was wrong this morning before I’d even sat down, then it hit…a colleague raise his head and said those dreaded words “all the Veeam jobs failed”….urrrrggghhhh, brilliant.

I love Veeam 95% of the time!!  When it works its ace but when it fails it’s a total bitch to sort.

Having said that I did manage a fairly quick fix today, so here was the error on every job:

[VeeamServer1] Failed to connect to Installer service.
[VeeamServer1] Failed to discover Installer service.
Backup Console version '6.5.0.128' is not compatible with Installer service on host ' VeeamServer1’.

So from the above error it looks like a service has been updated and Veeam can’t communicate.

So to confirm this let do to the Backup Infrastructure and see whats what:

There’s definitely a issue, if I try a rescan I get the following:

Here’s the text:

Fail Disks and volumes discovery for Windows server 'Server1.Doamin' failed Error: Backup Console version '6.5.0.128' is not compatible with Installer service on host ''Server1.Doamin'.
Fail Session failed Error: Backup Console version '6.5.0.128' is not compatible with Installer service on host ''Server1.Doamin'.

Veeam doesn’t make it particularly easy to work out version within Veeam itself so I’ve found the best thing to do is go to Control Panel and Programs and Features:

So as you can see the Veeam Backup Transport agent is newer than the rest of Veeam which is obviously where the problem is.

The reason this happened is a admin on a remote site on a different Veeam server running a newer version of Veeam must of opened Veeam and noticed an agent was out of date and decided to update it (Veeam can be used to copy jobs and proxy backups between sites and servers).

So once he did this the message went away his site and he was happy, the problem is we’re not!! Also just as a bonus we can prove this by digging in the event log:

So what do we do?  Restore Veeam to an older version on the remote site? Well that seems silly, so lets just update Veeam to match the other.

So I downloaded the patch and updates the server:

You’ll notice I was prompted to update the remote agents on other proxy servers (this is what the other admin did).

Double check the version in Veeam:

Run a test job:

And we’re fixed!!

So one more Veeam issue for you!

Had a problem where the backups to SAN where SLOW, I mean really slow….a 1TB server was taking 12 hours to complete and killing the servers performance!

So after talking with Veeam they checked out our logs and mentioned when we switched vCenters and reregistered our VM’s it seemed to have changed the ID (oldChangeId and curChangeId differed) meaning change block tracking or CBT was set to false (IsChangeTrackingAllowed: FALSE) and so tracking of what blocks had changed wouldn’t work and so meaning our backups crawled!!

To fix this we had to whats laid out here: http://www.veeam.com/KB1113

Which cleared up our issues.

Recently we had some major system changed!!

We built a new VMware vCenter Server and upgraded all our ESX hosts to 5.1 then shifted all our VM’s to the new vCenter!

All was good in the world….until the witching hour and the Veeam backups kicked in!!

FAIL!  Massive Fail!!

Ok so my fault, Veeam can’t see the new vCentre…so I added it to the vCenter and made sure Veeam could see the VMs!  Gave it a try and they still failed with:

Task failed Error: Object SERVER not found.

Why, you ask…well the hosts have the same name but are also different due to the upgrade and the new vCenter so Veeam can work out how to route to the VM and most probably it’s going to try and route to the old host which technically doesn’t exist (on that vCenter)!!

So we need to edit the SQL Database and delete the duplicate host entries!

Here’s how:

--Find the duplicates
 SELECT name, count(name) as _dupes from [Hosts]
where type in (0,6)
group by name
having count(name) > 1
order by _dupes desc

-- Delete the duplicates where they match the old parent id of your old vCentre.
delete from [hosts]
where name = 'Host1.domain.local' and parent_id != '3242B2CC-022B-42C8-8AB2-BB77B2F2B5F2' or
name = 'Host2.domain.local' and parent_id != '3242B2CC-022B-42C8-8AB2-BB77B2F2B5F2'

Gave it another run and the backups worked!!

If you’re a Veeam user you may feel the same way as I do when it comes to the product, that is:

“It’s absolutely brilliant and I really love it but when it goes wrong, it really goes wrong!”

So we recently we had a backup failure where were getting the following error:

Unable to release guest. Error: Unfreeze error (over VIX): [Backup job failed.]

On the exchange server we were seeing these kind of errors in the event log:

Information Store (3424) Shadow copy instance  freeze ended.

&

Volume Shadow Copy Service error: Unexpected error calling routine ConvertStringSidToSid (S-2-6-11-2001077031-0230750062-1115881885-1111.bak).

Also when I ran “vssadmin list writers” in a command prompt  I found:

Writer name: 'Microsoft Exchange Writer'
    Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
    Writer Instance Id: {88d385e0-c3c0-4c1a-9122-2dca8b3d797f}
    State: [9] Failed
    Last error: Timed out

So after some Googling, I found fixes such as:

Reboot the server,

Uninstall any Symantec backup exec agents on the machine,

Re-install VMware tools,

Run a batch script to re-register VSS DLL’s (I’ll post it below),

Make sure Windows firewalls are off.

Well none of these ended up being our issue, ours was down to our physical internal firewalls!!  A change had been made meaning the Exchange VSS writer couldn’t write directly to the SAN meaning by the time it routed there via an alternative path the default 20 seconds VSS Microsoft Exchange Writer had timed out!  Bugger.

So the firewall man, reverted his change and Bobs your uncle, successful backup!!

Script as promised:

net stop "System Event Notification Service"
net stop "Background Intelligent Transfer Service"
net stop "COM+ Event System"
net stop "Microsoft Software Shadow Copy Provider "
net stop "Volume Shadow Copy"
cd /d %windir%\system32
net stop Vss
net stop swprv
regsvr32 /s ATL.DLL
regsvr32 /s comsvcs.DLL
regsvr32 /s credui.DLL
regsvr32 /s CRYPTNET.DLL
regsvr32 /s CRYPTUI.DLL
regsvr32 /s dhcpqec.DLL
regsvr32 /s dssenh.DLL
regsvr32 /s eapqec.DLL
regsvr32 /s esscli.DLL
regsvr32 /s FastProx.DLL
regsvr32 /s FirewallAPI.DLL
regsvr32 /s kmsvc.DLL
regsvr32 /s lsmproxy.DLL
regsvr32 /s MSCTF.DLL
regsvr32 /s msi.DLL
regsvr32 /s msxml3.DLL
regsvr32 /s ncprov.DLL
regsvr32 /s ole32.DLL
regsvr32 /s OLEACC.DLL
regsvr32 /s OLEAUT32.DLL
regsvr32 /s PROPSYS.DLL
regsvr32 /s QAgent.DLL
regsvr32 /s qagentrt.DLL
regsvr32 /s QUtil.DLL
regsvr32 /s raschap.DLL
regsvr32 /s RASQEC.DLL
regsvr32 /s rastls.DLL
regsvr32 /s repdrvfs.DLL
regsvr32 /s RPCRT4.DLL
regsvr32 /s rsaenh.DLL
regsvr32 /s SHELL32.DLL
regsvr32 /s shsvcs.DLL
regsvr32 /s /i swprv.DLL
regsvr32 /s tschannel.DLL
regsvr32 /s USERENV.DLL
regsvr32 /s Vss_ps.DLL
regsvr32 /s wbemcons.DLL
regsvr32 /s wbemcore.DLL
regsvr32 /s wbemess.DLL
regsvr32 /s wbemsvc.DLL
regsvr32 /s WINHTTP.DLL
regsvr32 /s WINTRUST.DLL
regsvr32 /s wmiprvsd.DLL
regsvr32 /s wmisvc.DLL
regsvr32 /s wmiutils.DLL
regsvr32 /s wuaueng.DLL
Cd /d %systemroot%\syswow64
regsvr32 /s ole32.dll
regsvr32 /s Vss_ps.dll
regsvr32 /s es.dll
regsvr32 /s stdprov.dll
regsvr32 /s msxml3.dll
regsvr32 /s msxml.dll
regsvr32 /s msxml4.dll
net start "COM+ Event System"
sfc /SCANFILE=%windir%\system32\catsrv.DLL
sfc /SCANFILE=%windir%\system32\catsrvut.DLL
sfc /SCANFILE=%windir%\system32\CLBCatQ.DLL
net start "COM+ Event System"

Like me you may want to stat a backup exec job once something else has complete, for example when Veeam has finished backing up my VM environment I want BE to backup the Veeam files generated….so here’s how to execute Backup Exec jobs via a script:

1. Create your Backup Exec job with no scheduling, and make a note of the job name. (your_job_name)

2. Create a new batch file, using the following statements;

X:\Y\Z\bemcmd.exe -c”YOURBACKUPSERVER” -n”LOCALADMINISTRATOR” -p”PASSWORD” -o1 -j”your_job_name”

Where;

X:\Y\Z is the location of your BackupExec install directory if running locally, or the location of your BEMCMD.exe and supporting DLL’s if a remote server.

“YOURBACKUPSERVER” – FQDN or Netbios name of your BackupExec server

“LOCALADMINISTRATOR” – Local administrator account on the BE server. Consider using a local service account for this task.

“PASSWORD” – Local Admin account password. This is stored in plain text.

“your_job_name” – The name of your backup job in Backup Exec.

3. Configure your thrid party appkication to execute the script, and your backup job will be executed on the BE server.

Consider added alerting to your batch file if your 3rd party application does not feature reporting.