vmware

VMware vRops Azure error

Posted by robd on February 23, 2022
vmware / No Comments

So we’ve been getting a very annoying error when trying to monitor Azure with vrops:

“Unable to establish a valid connection to the target system. javax.net.ssl.SSLHandshakeException: General SSLEngine problem”

 

vrops error

I found this article that states how to fix it, which gave me a clue but didnt actually fix the problem:

https://kb.vmware.com/s/article/50122228

Basically the certs for Azure had expired so it couldn’t check the status.

The fix for me was to download all the new Azure Certs:

https://docs.microsoft.com/en-us/azure/security/fundamentals/tls-certificate-changes

azure certs

Convert them to pem files:

https://cheapsslsecurity.com/p/convert-a-certificate-to-pem-crt-to-pem-cer-to-pem-der-to-pem/

Now upload the pem files to vrops:

/usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/

 

Now ssh to vrops and get the keystore password:

cd /storage/vcops/user/conf/ssl/

keytool -list -keystore tcserver.truststore

and finally import your pem files:

cd /storage/vcops/user/conf/ssl/
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/Microsoft_RSA_Root_Certificate_Authority_2017.pem -alias azure1 -keystore tcserver.truststore
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/Microsoft_ECC_Root_Certificate_Authority_2017.pem -alias azure2 -keystore tcserver.truststore
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/D-TRUST_Root_Class_3_CA_2_2009.pem -alias azure3 -keystore tcserver.truststore
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/BaltimoreCyberTrustRoot.pem -alias azure4 -keystore tcserver.truststore
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/DigiCertGlobalRootCA.pem -alias azure5 -keystore tcserver.truststore
keytool -import -file /usr/lib/vmware-vcops/user/plugins/inbound/microsoftazure_adapter3/conf/certs/DigiCertGlobalRootG2.pem -alias azure6 -keystore tcserver.truststore

 

 

 

Tags: , ,

vMotion Slow

Posted by robd on January 11, 2022
vmware / No Comments

We recently replaced our estate of trusty PowerEdge R620 Dell servers, with shiny new Cisco UCS C220 M5SX Servers.

On each server, VMware ESXi was installed. We used LAG with 4 10 GB NIC’s, each server had 10 7200rpm drives setup in a RAID 6 (no SSDs, don’t ask why).

All looked great, but when moving the VM’s to the Cisco servers with vMotion, the process was super slow.

After a lot of digging, we discovered the issue was with the way the Wite Policy was setup on the RAID, we’d used Write Through, and we should have used Write Back Bad BBU:

 

– Write Through— Data is written through the cache and to the physical drives. Performance is improved, because subsequent reads of that data can be satisfied from the cache.

– Write Back— Data is stored in the cache, and is only written to the physical drives when space in the cache is needed. Virtual drives requesting this policy fall back to Write Through caching when the BBU cannot guarantee the safety of the cache in the event of a power failure.

– Write Back Bad BBU—With this policy, write caching remains Write Back even if the battery backup unit is defective or discharged.

Once we changed it, everything worked superfast.

 

Tags: , , ,

Vib Error updating ESXi from 6.0 to 6.7

Posted by robd on March 05, 2020
vmware / No Comments

Tried to upgrade ESX on a Cisco server today but kept getting the following today:

missing_dependency_VIBS ERROR. Found=[Qlogic_bootbank_scsi-qla2xxx….. These vibs on the host are missing dependency if you continue to upgrade.

So to fix, put the host in Maintenance mode then check if the device blocking is in use:

esxcfg-scsidevs -a 

esxcfg-nics -l 

esxcli software vib list | grep -i scsi-ql

In my case it wasnt so bin it off:

esxcli software vib remove -n scsi-qla2xxx

 

All done, reboot and try and upgrade

 

Tags: , ,

Edit VMs using PowerShell and PowerCLI

Posted by robd on January 28, 2019
powershell, vmware / No Comments

To resize VMs using PowerShell with PowerCLI from a csv list, first install the software:

 

https://my.vmware.com/web/vmware/details?downloadGroup=PCLI650R1&productId=614

 

Then create a list of servers to resize and save it as a CSV file in C:\temp\VMs.csv:

 

Server01

Server02

Server03

 

Save the below as Something.PS1 and run from PowerCLI

Note: Change VCENTRE to your vCentre, this script will TURN THE SERVER OFF then give each VM two CPUs, one socket and 5GBs of RAM.

 

$me = Get-Credential

connect-viserver "VCENTRE" -User $me

$vms = get-content C:\Temp\VMs.csv

ForEach ($vm in $vms){

$vms | Shutdown-VMGuest –Confirm:$False

Sleep 60

$vms | Set-VM –MemoryGB 8 –NumCpu 2 –Confirm:$False

$vms | Start-VM

}

 

Tags: , ,

Orphaned VM in VMWare 6.5

Posted by robd on January 24, 2019
vmware / No Comments

Had an issue where I’d deleted a VMDK from a LUN and was left with orphaned VM in vSphere, no problem I thought…right click and remove from inventory.  The problem is, all options were greyed out:

So what to do?

Enable SSH on the host.

Connect wit Putty and run:

List all registered VMs

vim-cmd /vmsvc/getallvms

Then to unregistered:

vim-cmd /vmsvc/unregister <id>

 

Tags: , , ,

Citrix and vCentre

Posted by robd on September 05, 2018
Citrix, vmware / No Comments

Annoyingly our venctre broke recently meaning our Citrix clients wouldnt boot which had the knock on affect users couldnt logon.

To easily check the connection status of citrix and vcentre, you can run the following PowerShell command on a Citrix delivery server (or whereever Citrix PS is installed):

This is what it looks like when its broken, notice the State:

PS C:\Program Files\Citrix\Desktop Studio> Get-BrokerHypervisorConnection


Capabilities                           : {PowerOn, PowerOff, SuspendResume, Re
                                         et...}
ExplicitPreferredController            : True
HypHypervisorConnectionUid             : a17e1707-98f2-43e6-8d5b-42302ccbae53
IsReady                                : True
MachineCount                           : 564
MaxAbsoluteActiveActions               : 2
MaxAbsoluteNewActionsPerMinute         : 15
MaxAbsolutePvdPowerActions             : 100000
MaxPercentageActiveActions             : 10
MaxPvdPowerActionsPercentageOfDesktops : 50
MetadataMap                            : {}
Name                                   : KIL VMware vSphere
PreferredController                    : BohemianGrove\CitrixServer01
State                                  : Unavailable
Uid                                    : 1

Fixing vcentre and rebooting the citrix server it then looks like this:

PS C:\Program Files\Citrix\Desktop Studio> Get-BrokerHypervisorConnection


Capabilities                           : {PowerOn, PowerOff, SuspendResume, Res
                                         et...}
ExplicitPreferredController            : True
HypHypervisorConnectionUid             : a17e1707-98f2-43e6-8d5b-42302ccbae53
IsReady                                : True
MachineCount                           : 564
MaxAbsoluteActiveActions               : 2
MaxAbsoluteNewActionsPerMinute         : 15
MaxAbsolutePvdPowerActions             : 100000
MaxPercentageActiveActions             : 10
MaxPvdPowerActionsPercentageOfDesktops : 50
MetadataMap                            : {}
Name                                   : KIL VMware vSphere
PreferredController                    : BohemianGrove\CitrixServer01
State                                  : On
Uid                                    : 1

 

Tags: , , ,

Ratio of Physical CPUs to Virtual CPUs in VMware

Posted by robd on August 06, 2018
powershell, vmware / 1 Comment

My colleague Welsh Dai made this sweet bit of PowerShell to see the ratio of physical CPUs to Virtual CPUs:

$allhosts = @()
$cluser2hosts = Get-VMHost | where {$_.Parent -LIKE "ClusterName"} 
foreach ($vmhost in $cluser2hosts)
{
   $vms = $vmhost | Get-VM | select name,numcpu | measure -Property numcpu -Sum
   $hostload = New-Object psobject -Property @{
            hostname = $vmhost.Name
            PhysicalCPUs = $vmhost.NumCpu
            vCPUs = $vms.Sum
            hostratio = $vms.sum / $vmhost.NumCpu
    }
    $allhosts += $hostload 
}

$allhosts  | select hostname,physicalCPUs,vCPUs,hostratio | sort hostratio

 

Here’s a picture

Tags: , , ,

How to setup Cisco port mirroring to a VM

Posted by robd on December 06, 2017
Networking, vmware / 1 Comment

Today we needed to mirror a port on a Cisco switch in a country far far away meaning we couldn’t just wander down with a laptop.

So to get around this we decided to mirror the port to a VM that’s on site, here’s how we did it:

There’s two switches between the VM and the port we want to mirror so first we have to setup the port mirroring on every switch using RSPAN (Remote Switched Port Analyser) and a new vlan.

Add an RSPAN vlan to both the switch with the port to mirror, and to the switch that has the packet capture device on.

Conf t
Vlan 9999
Remote-span

Then make sure that RSPAN vlan is trunked between the 2 switches and on the VMWARE interface.

switchport trunk allowed vlan add 9999

Assuming the following;

You use vlan 999 for the RSPAN vlan.

The port you want to mirror is on switch 1 port g1/0/2.

You want to send the mirrored traffic to switch 2 port 1/0/23 (the port that connects to VMWARE).

You are going to use monitor session 1 on both switches. (this can be any session number between 1-66, and can be different on each switch).

On Switch 1 (mirror port 2 and punt out the traffic to 9999)

Conf t
Monitor session 1 source interface g1/0/2 both
Monitor session 1 destination remote vlan 999

On Switch 2 (suck in all traffic from 9999 and punt it to port 23)

Conf t
Monitor session 1 source remote vlan 999
Monitor session 1 destination interface g1/0/23

You can see that the monitoring is set up with;

Show monitor session

Network Diagram:

Next, we need to do the VMware side of things:

 

Setup a new port group on a vswitch:

Although we chose vlan 9999 when its pushed to the new port it will not be tagged so choose all:

Next edit the port group and allow Promiscuous mode, this will allow traffic not destined for the VMs MACs (normal behaviour, any traffic not destined to a VM MAC will be dropped):

Add a NIC to your VM using the port group:

Don’t worry about a IP etc:

Fire up Wireshark:

Look at all these glorious packets:

 

 

Tags: , , , ,

Failed to connect virtual device ethernet0

Posted by robd on May 04, 2017
vmware, vSphere / 2 Comments

Today I noticed a NIC was down on a VM, had a look and noticed it was disconnected:

 

 So I ticked the box and clicked ok and got the follow error:

Failed to connect virtual device ethernet0.

Weird, so I thought I’d check the port ID for that virtual switch:

Looks like something else is on that port id:

Scroll down the list to find a free port ID and then go back to the VM and change the port ID and boom you should be able to connect.

Tags: , ,

VMware – Increase disk past 2TB

Posted by robd on February 16, 2017
vmware, vSphere / 1 Comment

Tried to increase a disk past 2TB today on a VM on a ESXi 6.0.0 (via vSphere) host but kept getting this:

Hot-extend was invoked with size (6442450944 sectors) >= 2TB. Hot-extend beyond or equal to 2TB is not supported. 
The disk extend operation failed: msg.disklib.INVAL

Well it was driving me nuts until I turned the VM off and tried again…..it worked.

Tags: , ,