Tech Blog

So we’ve been getting a very annoying error when trying to monitor Azure with vrops:

“Unable to establish a valid connection to the target system. javax.net.ssl.SSLHandshakeException: General SSLEngine problem”

I found this article that states how to fix it, which gave me a clue but didnt actually fix the problem:

https://kb.vmware.com/s/article/50122228

Basically the certs for Azure had expired so it couldn’t check the status.

The fix for me was to download all the new Azure Certs:

https://docs.microsoft.com/en-us/azure/security/fundamentals/tls-certificate-changes

Convert them to pem files:

https://cheapsslsecurity.com/p/convert-a-certificate-to-pem-crt-to-pem-cer-to-pem-der-to-pem/

Now upload the pem files to vrops:

/usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/

Now ssh to vrops and get the keystore password:

cd /storage/vcops/user/conf/ssl/

keytool -list -keystore tcserver.truststore

and finally import your pem files:

cd /storage/vcops/user/conf/ssl/
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/Microsoft_RSA_Root_Certificate_Authority_2017.pem -alias azure1 -keystore tcserver.truststore
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/Microsoft_ECC_Root_Certificate_Authority_2017.pem -alias azure2 -keystore tcserver.truststore
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/D-TRUST_Root_Class_3_CA_2_2009.pem -alias azure3 -keystore tcserver.truststore
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/BaltimoreCyberTrustRoot.pem -alias azure4 -keystore tcserver.truststore
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/DigiCertGlobalRootCA.pem -alias azure5 -keystore tcserver.truststore
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/DigiCertGlobalRootG2.pem -alias azure6 -keystore tcserver.truststore

We recently replaced our estate of trusty PowerEdge R620 Dell servers, with shiny new Cisco UCS C220 M5SX Servers.

On each server, VMware ESXi was installed. We used LAG with 4 10 GB NIC’s, each server had 10 7200rpm drives setup in a RAID 6 (no SSDs, don’t ask why).

All looked great, but when moving the VM’s to the Cisco servers with vMotion, the process was super slow.

After a lot of digging, we discovered the issue was with the way the Wite Policy was setup on the RAID, we’d used Write Through, and we should have used Write Back Bad BBU:

– Write Through— Data is written through the cache and to the physical drives. Performance is improved, because subsequent reads of that data can be satisfied from the cache.

– Write Back— Data is stored in the cache, and is only written to the physical drives when space in the cache is needed. Virtual drives requesting this policy fall back to Write Through caching when the BBU cannot guarantee the safety of the cache in the event of a power failure.

– Write Back Bad BBU—With this policy, write caching remains Write Back even if the battery backup unit is defective or discharged.

Once we changed it, everything worked superfast.

Tried to upgrade ESX on a Cisco server today but kept getting the following today:

missing_dependency_VIBS ERROR. Found=[Qlogic_bootbank_scsi-qla2xxx….. These vibs on the host are missing dependency if you continue to upgrade.

So to fix, put the host in Maintenance mode then check if the device blocking is in use:

esxcfg-scsidevs -a 

esxcfg-nics -l 

esxcli software vib list | grep -i scsi-ql

In my case it wasnt so bin it off:

esxcli software vib remove -n scsi-qla2xxx

All done, reboot and try and upgrade

To resize VMs using PowerShell with PowerCLI from a csv list, first install the software:

https://my.vmware.com/web/vmware/details?downloadGroup=PCLI650R1&productId=614

Then create a list of servers to resize and save it as a CSV file in C:\temp\VMs.csv:

Server01

Server02

Server03

Save the below as Something.PS1 and run from PowerCLI

Note: Change VCENTRE to your vCentre, this script will TURN THE SERVER OFF then give each VM two CPUs, one socket and 5GBs of RAM.

$me = Get-Credential

connect-viserver "VCENTRE" -User $me

$vms = get-content C:\Temp\VMs.csv

ForEach ($vm in $vms){

$vms | Shutdown-VMGuest –Confirm:$False

Sleep 60

$vms | Set-VM –MemoryGB 8 –NumCpu 2 –Confirm:$False

$vms | Start-VM

}

Had an issue where I’d deleted a VMDK from a LUN and was left with orphaned VM in vSphere, no problem I thought…right click and remove from inventory.  The problem is, all options were greyed out:

So what to do?

Enable SSH on the host.

Connect wit Putty and run:

List all registered VMs

vim-cmd /vmsvc/getallvms

Then to unregistered:

vim-cmd /vmsvc/unregister <id>

I run a HP G7 N54L which has ESXi 6.5 installed, its getting old but its brilliant.

Anyhow, I ran a Windows 10 VM which I had Plex installed on, for some reason Plex gave up the ghost so I decided to install Ubuntu and was planning on installing Plex on there.

Grabbed the ISO and installed and 5 minutes after install the Ubuntu server froze, so rebooted, froze again.

After some digging I post a post that mentioned its a hardware issue sorted in ESXi 6.5 update 1.

So downloaded and update my server (note there was a warning that future releases of ESXi wont be supported on this CPU) and ran Ubuntu and it worked flawlessly.

Today we needed to mirror a port on a Cisco switch in a country far far away meaning we couldn’t just wander down with a laptop.

So to get around this we decided to mirror the port to a VM that’s on site, here’s how we did it:

There’s two switches between the VM and the port we want to mirror so first we have to setup the port mirroring on every switch using RSPAN (Remote Switched Port Analyser) and a new vlan.

Add an RSPAN vlan to both the switch with the port to mirror, and to the switch that has the packet capture device on.

Conf t
Vlan 9999
Remote-span

Then make sure that RSPAN vlan is trunked between the 2 switches and on the VMWARE interface.

switchport trunk allowed vlan add 9999

Assuming the following;

You use vlan 999 for the RSPAN vlan.

The port you want to mirror is on switch 1 port g1/0/2.

You want to send the mirrored traffic to switch 2 port 1/0/23 (the port that connects to VMWARE).

You are going to use monitor session 1 on both switches. (this can be any session number between 1-66, and can be different on each switch).

On Switch 1 (mirror port 2 and punt out the traffic to 9999)

Conf t
Monitor session 1 source interface g1/0/2 both
Monitor session 1 destination remote vlan 999

On Switch 2 (suck in all traffic from 9999 and punt it to port 23)

Conf t
Monitor session 1 source remote vlan 999
Monitor session 1 destination interface g1/0/23

You can see that the monitoring is set up with;

Show monitor session

Network Diagram:

Next, we need to do the VMware side of things:

Setup a new port group on a vswitch:

Although we chose vlan 9999 when its pushed to the new port it will not be tagged so choose all:

Next edit the port group and allow Promiscuous mode, this will allow traffic not destined for the VMs MACs (normal behaviour, any traffic not destined to a VM MAC will be dropped):

Add a NIC to your VM using the port group:

Don’t worry about a IP etc:

Fire up Wireshark:

Look at all these glorious packets:

Today I noticed a NIC was down on a VM, had a look and noticed it was disconnected:

 So I ticked the box and clicked ok and got the follow error:

Failed to connect virtual device ethernet0.

Weird, so I thought I’d check the port ID for that virtual switch:

Looks like something else is on that port id:

Scroll down the list to find a free port ID and then go back to the VM and change the port ID and boom you should be able to connect.

Tried to increase a disk past 2TB today on a VM on a ESXi 6.0.0 (via vSphere) host but kept getting this:

Hot-extend was invoked with size (6442450944 sectors) >= 2TB. Hot-extend beyond or equal to 2TB is not supported. 
The disk extend operation failed: msg.disklib.INVAL

Well it was driving me nuts until I turned the VM off and tried again…..it worked.

If you are building a Client/Server that is going to be used in a VDI environment don’t install the VMware SVGA WDDM driver as part of the VMware tools install.

It can cause issues like Black screens, Users not connecting etc and the first step Citrix get you to perform is to remove them, with XenDesktop this is fairly easy but time consuming with VDI-in-a-Box you can’t…

For XenDesktop, If you have installed them you can us the follow these steps to remove the driver:

• Remove VDA agent(s)
• Reboot
• Remove VMware Tools
• Reboot
• Install VMware tools (Custom install without SVGA driver)
• Reboot
• Install VDA agent(s)
• Reboot
• Update the Catalogue

For VDI-in-a-Box, If you have installed them you can us the follow these steps to remove the driver:

• Start again with new image.

Notes:

If you already have it installed and it’s not causing any issues you can leave it as is.

If you don’t remove the VDA agent before Uninstall/reinstall VMware tools, the guest will break.

VDI-in-a-Box installs the VDA agents when you import the image, you can’t get them any other way. (confirmed by Citrix Support)

Both these links mention VMware 4.1 U1 but Citrix have confirmed it is still affects later versions.

http://support.citrix.com/article/CTX123952

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011709