cisco

Cisco ISE – Live Logs Broken

Posted by robd on February 24, 2022
Cisco / No Comments

Hello,

After upgrading to Cisco ISE 3.0 and updating some certs we noticed the Radius Live Logs broke:

So to fix, I changed certs and rebooted nodes and basically spent hours trying everything.

I dint get anywhere so I raised a Cisco TAC and they fixed it by doing the following (which took two seconds), they un-checked:

 

 

Tags: , ,

Firmware Update Cisco SG350

Posted by robd on January 12, 2022
Cisco / No Comments

I had to update a Cisco SG350 recently, which should have been really easy but ended up being a bit of a pain.

Here’s how I’d do it again:

  • Download the firmware:

https://www.cisco.com/c/en/us/support/switches/sg350-28-28-port-gigabit-managed-switch/model.html

  • Ignore the GUI, pretend it never existed, as best I can work out it just doesn’t work well.
  • Download a tftp software, I suggest tftpd64 as it brilliant:

https://pjo2.github.io/tftpd64/

  • Run the tftp software and point it at the firmware, make a note of the IP.
  • SSH to the switch with putty and run the following to check the version:
show ver
  • Then run this command on the switch (The IP if the IP of the tftp software):
boot system tftp://192.168.1.2/image_tesla_hybrid_2.5.8.15_release_cisco_signed.bin

  • Now reboot:
reboot
  • Once it’s rebooted, check the version, and you are done!
show ver

Tags: , ,

vMotion Slow

Posted by robd on January 11, 2022
vmware / No Comments

We recently replaced our estate of trusty PowerEdge R620 Dell servers, with shiny new Cisco UCS C220 M5SX Servers.

On each server, VMware ESXi was installed. We used LAG with 4 10 GB NIC’s, each server had 10 7200rpm drives setup in a RAID 6 (no SSDs, don’t ask why).

All looked great, but when moving the VM’s to the Cisco servers with vMotion, the process was super slow.

After a lot of digging, we discovered the issue was with the way the Wite Policy was setup on the RAID, we’d used Write Through, and we should have used Write Back Bad BBU:

 

– Write Through— Data is written through the cache and to the physical drives. Performance is improved, because subsequent reads of that data can be satisfied from the cache.

– Write Back— Data is stored in the cache, and is only written to the physical drives when space in the cache is needed. Virtual drives requesting this policy fall back to Write Through caching when the BBU cannot guarantee the safety of the cache in the event of a power failure.

– Write Back Bad BBU—With this policy, write caching remains Write Back even if the battery backup unit is defective or discharged.

Once we changed it, everything worked superfast.

 

Tags: , , ,

Upgrade Cisco ISE from 2.6 to 2.7

Posted by robd on February 03, 2021
Cisco, Wireless / No Comments

Hello,

Upgrading Cisco ISE is pretty straight forward, there’s a pretty GUI that makes sure you back everything up before you start and then you specify a repository and the files get downloaded and bobs your uncle.

FYI – This is good blog on backing up ISE before you start:  https://www.letsconfig.com/how-to-backup-cisco-ise-2-7/

My issue is I have a ISE node in a DMZ which cant contact the main repository on the network due to it being ultra secure.

So I had to do it manually via the Cisco ISE CLI on the node in the dmz.

First find a server that can access the DMZ on port 21 – Note, I tried tftp but the transfer would fail every time.

Then download this portable ftp server: https://www.xlightftpd.com/download.htm

Run the ftp server, setup the NIC and create a user with a home directory (a folder on the server)

Download the ISE upgrade file and put it in the home directory: ise-upgradebundle-2.2.x-2.6.x-to-2.7.0.356.SPA.x86_64.tar.gz

Next logon to your ISE node,

Create a repository on the ISE node:

conf t
repository dmzf
url ftp://172.25.61.42
user FTPAdmin password plain FTPPassword

Now you can pull the upgrade file – Note, this will just download and unpackaged the file, NOT run the update.

application upgrade prepare ise-upgradebundle-2.2.x-2.6.x-to-2.7.0.356.SPA.x86_64.tar.gz dmzf

Wait for that to finish:

Now you’re ready to actually upgrade.

application upgrade proceed

Wait for the reboot and update:

Then  you are done!!

Double check

Show Version

 

Done. Boom

Tags: , , ,

Updating Cisco Prime 3.4 to 3.7

Posted by robd on March 06, 2020
Cisco / 2 Comments

Well what a fun few days its been.  I’ve been on a mad mission to update our Cisco products, first I did Cisco ISE (I’ll try and blog about that later) and then Cisco Prime.

Here’s the Cisco Prime tac I took:

SSH to Prime

First thing first, backup Prime.  In my case that was on the prime server in the default repo.

copy NAMEofBACKUP.tar.gz ftp://10.1.1.2/

Next now ftp the update the file to prime:

copy FTP://10.1.1.2/PI-Upgrade-31x_32x_33x_34x_to_3.7.0.0.159.tar.gz disk:/defaultRepo

next check its there:

show repository defaultRepo

Now update:

application upgrade PI-Upgrade-31x_32x_33x_34x_to_3.7.0.0.159.tar.gz defaultRepo

Oh no ERROR:

ERROR : Please run the application upgrade from the system console to monitor upgrade progress. Use system monitor, serial terminal or a virtual console to initiate the upgrade.

You have to open the VM console!!!!!!!

Try again!

application upgrade PI-Upgrade-31x_32x_33x_34x_to_3.7.0.0.159.tar.gz defaultRepo

Ugh, its full.  Delete the old backups

Delete disk://BACKUP.tar.gz defaultRepo

Try again

application upgrade PI-Upgrade-31x_32x_33x_34x_to_3.7.0.0.159.tar.gz defaultRepo

Arggggh now what, stop Prime….

Ncs stop

Try again:

application upgrade PI-Upgrade-31x_32x_33x_34x_to_3.7.0.0.159.tar.gz defaultRepo

Now its working!!

Tags: , ,

Dynamic vlan Assignment on Flexconnect using Cisco Wireless

Posted by robd on February 17, 2020
Wireless / No Comments

Hello,

I recently setup dynamic vlan assignment using Cisco ISE and a Cisco vWLC but had an issue where on some APs on some sites wouldnt move the devices to the correct DHCP scope.

So just make it clear what dynamic vlan assignment is, its when you have one SSID to rule them all and in the dark bind them.

So I have laptop and hand held scanners and only one SSID, I want my hand held scanner to go onto a different vlan and DHCP scope my laptops. So I use this option in profiles in ISE:

Then setup the scope option and bobs your uncle.

So back to the issue, some sites just wouldnt move scopes i.e. they’d stay on default scope.  So first thing I did was debug the client via the CLI on the vWLC:

debug client 94:fb:29:43:74:b9
*apfMsConnTask_1: Jan 30 13:09:53.561: 94:fb:29:43:74:b9 Encryption policy is set to 0x80000004
*apfMsConnTask_1: Jan 30 13:09:53.561: 94:fb:29:43:74:b9 10.51.140.17 8021X_REQD (3) Client already has IP 10.10.1.17, DHCP Not required on AP 70:79:b3:9f:4c:c0 vapId 1 apVapId 1
*apfMsConnTask_1: Jan 30 13:09:53.561: 94:fb:29:43:74:b9 Not Using WMM Compliance code qosCap 00
*apfMsConnTask_1: Jan 30 13:09:53.561: 94:fb:29:43:74:b9 Vlan while overriding the policy = 153
*apfMsConnTask_1: Jan 30 13:09:53.561: 94:fb:29:43:74:b9 sending to spamAddMobile vlanId 153 flex aclName = , flexAclId 65535

So the client knows it should be on vlan 153 but isnt moving…….So after much googling I found that my flex connect groups hadnt been setup properly.

I was missing the vlans from the vlans from AAA VLAN-ACL Mapping.  Added them in and everything started working on every site!!!

Very weird how it ever worked but there you go.

 

Tags: , , ,

SNMP v3 – Cisco Catalyst 9500

Posted by robd on August 07, 2019
Networking / No Comments

Had this frustrating issues with a Cisco Catalyst 9500 and enabling SNMP with AES 256.

Basically 256 AES encryption wont wor:

snmp-server user USER1 GROUP1 v3 auth sha PASSWORD1 priv aes 256 PASSWORD2
snmp-server group GROUP1 v3 priv

But these would:

snmp-server user USER1 GROUP1 v3 auth sha PASSWORD1 priv aes 128 PASSWORD2
snmp-server group GROUP1 v3 priv

To test I used following software:

Paessler SNMP Tester:

This is it working:

Paessler SNMP Tester 5.2.3 Computername: COMPUTER Interface: 192.168.1.2

07/08/2019 09:17:20 (2 ms) : Device: 192.168.1.1

07/08/2019 09:17:20 (3 ms) : SNMP V3

07/08/2019 09:17:20 (4 ms) : Uptime

07/08/2019 09:17:22 (2244 ms) : SNMP Datatype: ASN_TIMETICKS

07/08/2019 09:17:22 (2250 ms) : -------

07/08/2019 09:17:22 (2256 ms) : DISMAN-EVENT-MIB::sysUpTimeInstance = 300185783 ( 34 days )

07/08/2019 09:17:23 (2521 ms) : SNMP Datatype: SNMP_EXCEPTION_NOSUCHOBJECT

07/08/2019 09:17:23 (2523 ms) : HOST-RESOURCES-MIB::hrSystemUptime.0 = No such object (SNMP error # 222) ( 0 seconds )

07/08/2019 09:17:23 (2524 ms) : Done

This is it broken:

Paessler SNMP Tester 5.2.3 Computername: COMPUTER Interface: 192.168.1.2
06/08/2019 12:22:13 (2 ms) : Device: 192.168.1.1
06/08/2019 12:22:13 (3 ms) : SNMP V3
06/08/2019 12:22:13 (4 ms) : Uptime
06/08/2019 12:22:18 (4237 ms) : SNMP Datatype: ASN_PRIMITIVE
06/08/2019 12:22:18 (4242 ms) : -------
06/08/2019 12:22:18 (4245 ms) : DISMAN-EVENT-MIB::sysUpTimeInstance = No response (check: firewalls, routing, snmp settings of device, IPs, SNMP version, community, passwords etc) (SNMP error # -2003) ( 0 seconds )
06/08/2019 12:22:22 (8462 ms) : SNMP Datatype: ASN_PRIMITIVE
06/08/2019 12:22:22 (8466 ms) : HOST-RESOURCES-MIB::hrSystemUptime.0 = No response (check: firewalls, routing, snmp settings of device, IPs, SNMP version, community, passwords etc) (SNMP error # -2003) ( 0 seconds )
06/08/2019 12:22:22 (8468 ms) : Done

 

Tags: , ,

Cisco Wireless Lan Controller Update with Pre-Download

Posted by robd on June 13, 2019
Wireless / No Comments

Hello,

Had an issue joining a Cisco 2800 AP to a Cisco Wireless Controller

So the first thing to check is country code of the AP and controller and the time.

 

The AP is a -E and the country is on the controller:

https://www.cisco.com/c/dam/assets/prod/wireless/wireless-compliance-tool/index.html

Time looks ok:

 

To the console!!!

debug capwap errors enable

 

Looks like this controller version 8.0.133.0 isnt compatible with 2800s:

https://www.cisco.com/c/en/us/td/docs/wireless/compatibility/matrix/compatibility-matrix.html

Time to upgrade.

First check the APs are compatible with the version you are going too:

Looks ok.  Next download it (oh also download the code you currently have installed in case you need it!) and while you’re waiting backup the controller config:

 

Before you reboot, go to the CLI:

Check the version:

Show boot

Show ap image all

 

Pre-image the APs:

config ap image predownload primary all

Check the progress:

Reboot the controller via the GUI.

Done:

 

Tags: , ,

iPerf

Posted by robd on May 08, 2019
Networking / No Comments

I’ve been running iperf a lot recently, which is a tool to measure TCP, UDP and bandwidth performance across a network.  I’ve been running this specially from the users computer to a local server i.e. not across a WAN or MPLS etc.

The users computer has a 1GB connection to the switch and from there the switch is connected to the server via a 10GB NIC.

On the server I ran:

iperf3.exe -S

On the client I ran:

iperf3.exe -c 10.52.7.73 -R

Below are my initial results, as you can see the connection is nearly maxing out its 1GB connection:

[ ID] Interval           Transfer     Bandwidth

[  4]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec             sender

[  4]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec             receiver

 

I then ran the same test but increasing the TCP connections to better simulate a heavy network application.  From the screen shot below the network is performing well.

iperf3.exe -c 10.52.7.73 -R -P 20

So what do all the switches mean:

-t  option used in the above command tells to transfer data for 20 seconds.

-w will specify your desired window size value.  Whenever two machines are communicating with each other, then each of them will inform the other, about the amount of bytes it is ready to receive at one time. In other words, the maximum amount of data that a sender can send the other end, without an acknowledgement is called as Window Size.

-P 20 will increase the number of parallel TCP connections i.e. to 20 (Parallel TCP connections).

-R enabled reverse connections to and from the server

-I Changes the interval between periodic bandwidth tests. For example, -i 60 will make a new bandwidth report every 60 seconds. The default is zero, which performs one bandwidth test.

-B Binds iPerf to a specific interface or address. If passed through the server command, the incoming interface will be set. If passed through the client command, the outgoing interface will be set.

-i {time in seconds} #### update interval on screen; default is 1 second. Can be useful when running large numbers of concurrent tests to increase the interval, or disable with “-i 0”

–logfile {filename} #### outputs what would normally be in your terminal to a log file of your choosing.

 

Want to test QoS?  Specifically EF, then is the is bad boy:

iperf3.exe -c 10.52.7.73 -w 1M -P 5 -t 60 -R -i 5 -S 0xB8

 

Tags: , ,

Symbol RF Scanners and Cisco WLC

Posted by robd on November 13, 2018
Wireless / No Comments

Had a roaming issues with Symbol MC9090 RF scanners on a Cisco virtual WLC (AIR-CTVM-K9) but weirdly only at one site, even though the same setting were applied across all sites.

The issue was the scanners would drop their SSH connection when moving between APs.

Here’s all my findings:

  • Update the scanner firmware, do this, it’s a pain but the newer firmware has so many features that are beneficial.
  • Some Scanner firmware would not allow them to connect using the security method WPA2, so enable WPA /TKIP or a better option, update the scanner firmware.
  • Secondly change the Scanners to CAM Mode = constant awake mode.
  • Thirdly, Cisco TAC recommended using these settings:

Ensure the fast transition is set to adaptive (if you don’t see this then update the code on your WLC):

The Symbol RF scanners support CCKM according to the manual so enable this:

Weird one this one, Cisco told us to disable “Enable Session Timeout” (also disable Aironet IE)

Tags: , , , ,