Tech Blog

Had a very frustraiting issue recently where our Zebra RF Scanners werent getting DHCP addresses on certain Cisco Access Points.

Only the scanners were not working, everything else seemed fine!

So I checked a heap of things:

Data Rates

Some of RF scanners are OLD, so its important to find out what data rates they require and then match your RF profile.

I suggest you profile the scanner using sometime like a WLANPi first just so you dont have to enable any older data rates.

Or use:

Show client detail <MAC Address>

Read more about old data rates here.

Port Config

We run FlexConennect so was every port in a Trunk and did every port have have the correct vlans tags?

Trunk

Where all the vlans trunked up to the core switch?

DHCP Server

Rebooted it and everything seems fine, lots of DHCP requests from other devices etc.

To be sure I did run wireshark and there were no requests from the scanners while on the “broken” APs.

Debug, Debug, Debug

I then started these debugs and waited forced the client to join again:

From AP:
config ap client-trace address add 5c:87:9c:93:da:4b
config ap client-trace filter all enable 
config ap client-trace output console-log enable 
config ap client-trace start 
term mon

#when 
config ap client-trace stop


From WLC:
Debug client 11:22:33:44:55:66
Show client detail 11:22:33:44:55:66

So the results showthis:

When it works it looks like this:

DHCP request,

DOT11 Auth

DOT11 Association

ARP

DHCP Request

DHCP ACK

Dec 1 09:02:39 kernel: [*12/01/2020 09:02:39.6821] [1606813359:682125] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] DHCP_DISCOVER : TransId 0xefec6e9f
Dec 1 09:02:39 kernel: [*12/01/2020 09:02:39.6821] [1606813359:682163] [AP16] [11:22:33:44:55:66] <apr1v0> [U:C] DHCP_DISCOVER : TransId 0xefec6e9f
Dec 1 09:02:43 kernel: [*12/01/2020 09:02:43.2458] [1606813363:245845] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] DOT11_DISASSOC : (.)
Dec 1 09:02:43 kernel: [*12/01/2020 09:02:43.2465] [1606813363:246587] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] DOT11_DEAUTHENTICATION : (.)
Dec 1 09:02:43 kernel: [*12/01/2020 09:02:43.9712] [1606813363:971275] [AP16] [11:22:33:44:55:66] <apr1v0> [U:W] DOT11_AUTHENTICATION : (.)
Dec 1 09:02:43 kernel: [*12/01/2020 09:02:43.9721] [1606813363:972101] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] DOT11_AUTHENTICATION : (.)
Dec 1 09:02:43 kernel: [*12/01/2020 09:02:43.9829] [1606813363:982985] [AP16] [11:22:33:44:55:66] <apr1v0> [U:W] DOT11_REASSOC_REQUEST : (.)
Dec 1 09:02:43 kernel: [*12/01/2020 09:02:43.9839] [1606813363:983901] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] DOT11_REASSOC_RESPONSE : (.)
Dec 1 09:02:44 kernel: [*12/01/2020 09:02:44.0783] [1606813364: 78316] [AP16] [11:22:33:44:55:66] <wired0> [D:E] EAP_PACKET.Request : Id 0x01 type 1 Identity
Dec 1 09:02:44 kernel: [*12/01/2020 09:02:44.0784] [1606813364: 78397] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] EAP_PACKET.Request : Id 0x01 type 1 Identity
Dec 1 09:02:44 kernel: [*12/01/2020 09:02:44.1278] [1606813364:127862] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] EAP_PACKET.Response : Id 0x01 type 1 Identity
Dec 1 09:02:44 kernel: [*12/01/2020 09:02:44.1279] [1606813364:127968] [AP16] [11:22:33:44:55:66] <wired0> [U:E] EAP_PACKET.Response : Id 0x01 type 1 Identity
Dec 1 09:02:44 kernel: [*12/01/2020 09:02:44.1745] [1606813364:174565] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] EAP_PACKET.Request : Id 0xa7 type 25 Other
Dec 1 09:02:44 kernel: [*12/01/2020 09:02:44.1773] [1606813364:177337] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] EAP_PACKET.Response : Id 0xa7 type 25 Other
Dec 1 09:02:45 kernel: [*12/01/2020 09:02:45.8440] [1606813365:843995] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] EAPOL_KEY.M1 : DescType 0x02 KeyInfo 0x008a
Dec 1 09:02:45 kernel: [*12/01/2020 09:02:45.8906] [1606813365:890656] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] EAPOL_KEY.M2 : DescType 0x02 KeyInfo 0x010a
Dec 1 09:02:46 kernel: [*12/01/2020 09:02:46.0282] [1606813366: 28207] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] ARP_QUERY : Sender 10.10.10.1 TargIp 10.20.20.1
Dec 1 09:02:46 kernel: [*12/01/2020 09:02:46.0282] [1606813366: 28252] [AP16] [11:22:33:44:55:66] <apr1v0> [U:C] ARP_QUERY : Sender 10.10.10.1 TargIp 10.20.20.1
Dec 1 09:02:46 kernel: [*12/01/2020 09:02:46.0291] [1606813366: 29096] [AP16] [11:22:33:44:55:66] <wired0> [D:E] ARP_REPLY : Sender 10.10.10.1 HwAddr 66:55:44:33:22:11
Dec 1 09:02:46 kernel: [*12/01/2020 09:02:46.0291] [1606813366: 29138] [AP16] [11:22:33:44:55:66] <wired0> [D:C] ARP_REPLY : Sender 10.10.10.1 HwAddr 66:55:44:33:22:11
Dec 1 09:02:46 kernel: [*12/01/2020 09:02:46.0291] [1606813366: 29187] [AP16] [11:22:33:44:55:66] <wired0> [D:C] ARP_REPLY : Sender 10.10.10.1 HwAddr 66:55:44:33:22:11
Dec 1 09:02:47 kernel: [*12/01/2020 09:02:47.0520] [1606813367: 52031] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] DHCP_REQUEST : TransId 0xa68db1f1
Dec 1 09:02:47 kernel: [*12/01/2020 09:02:47.0520] [1606813367: 52070] [AP16] [11:22:33:44:55:66] <apr1v0> [U:C] DHCP_REQUEST : TransId 0xa68db1f1
Dec 1 09:02:47 kernel: [*12/01/2020 09:02:47.0555] [1606813367: 55585] [AP16] [11:22:33:44:55:66] <wired0> [D:C] DHCP_ACK : TransId 0xa68db1f1
Dec 1 09:02:47 kernel: [*12/01/2020 09:02:47.0556] [1606813367: 55636] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] DHCP_ACK : TransId 0xa68db1f1

When it doesnt, everything looks good until the end, no ACK from DHCP:

Dec  1 09:02:39 kernel: [*12/01/2020 09:02:39.6821] [1606813359:682125] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] DHCP_DISCOVER : TransId 0xefec6e9f
Dec  1 09:02:39 kernel: [*12/01/2020 09:02:39.6821] [1606813359:682163] [AP16] [11:22:33:44:55:66] <apr1v0> [U:C] DHCP_DISCOVER : TransId 0xefec6e9f
Dec  1 09:02:43 kernel: [*12/01/2020 09:02:43.2458] [1606813363:245845] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] DOT11_DISASSOC : (.)
Dec  1 09:02:43 kernel: [*12/01/2020 09:02:43.2465] [1606813363:246587] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] DOT11_DEAUTHENTICATION : (.)
Dec  1 09:02:43 kernel: [*12/01/2020 09:02:43.9712] [1606813363:971275] [AP16] [11:22:33:44:55:66] <apr1v0> [U:W] DOT11_AUTHENTICATION : (.)
Dec  1 09:02:43 kernel: [*12/01/2020 09:02:43.9721] [1606813363:972101] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] DOT11_AUTHENTICATION : (.)
Dec  1 09:02:43 kernel: [*12/01/2020 09:02:43.9829] [1606813363:982985] [AP16] [11:22:33:44:55:66] <apr1v0> [U:W] DOT11_REASSOC_REQUEST : (.)
Dec  1 09:02:43 kernel: [*12/01/2020 09:02:43.9839] [1606813363:983901] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] DOT11_REASSOC_RESPONSE : (.)
Dec  1 09:02:44 kernel: [*12/01/2020 09:02:44.0783] [1606813364: 78316] [AP16] [11:22:33:44:55:66] <wired0> [D:E] EAP_PACKET.Request : Id 0x01 type 1 Identity
Dec  1 09:02:44 kernel: [*12/01/2020 09:02:44.0784] [1606813364: 78397] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] EAP_PACKET.Request : Id 0x01 type 1 Identity
Dec  1 09:02:44 kernel: [*12/01/2020 09:02:44.1278] [1606813364:127862] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] EAP_PACKET.Response : Id 0x01 type 1 Identity
Dec  1 09:02:44 kernel: [*12/01/2020 09:02:44.1279] [1606813364:127968] [AP16] [11:22:33:44:55:66] <wired0> [U:E] EAP_PACKET.Response : Id 0x01 type 1 Identity
Dec  1 09:02:44 kernel: [*12/01/2020 09:02:44.1745] [1606813364:174565] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] EAP_PACKET.Request : Id 0xa7 type 25 Other
Dec  1 09:02:44 kernel: [*12/01/2020 09:02:44.1773] [1606813364:177337] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] EAP_PACKET.Response : Id 0xa7 type 25 Other
Dec  1 09:02:45 kernel: [*12/01/2020 09:02:45.8440] [1606813365:843995] [AP16] [11:22:33:44:55:66] <apr1v0> [D:W] EAPOL_KEY.M1 : DescType 0x02 KeyInfo 0x008a
Dec  1 09:02:45 kernel: [*12/01/2020 09:02:45.8906] [1606813365:890656] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] EAPOL_KEY.M2 : DescType 0x02 KeyInfo 0x010a
Dec  1 09:02:46 kernel: [*12/01/2020 09:02:46.0282] [1606813366: 28207] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] ARP_QUERY : Sender 10.10.10.1 TargIp 10.20.20.1
Dec  1 09:02:46 kernel: [*12/01/2020 09:02:46.0282] [1606813366: 28252] [AP16] [11:22:33:44:55:66] <apr1v0> [U:C] ARP_QUERY : Sender 10.10.10.1 TargIp 10.20.20.1
Dec  1 09:02:46 kernel: [*12/01/2020 09:02:46.0291] [1606813366: 29096] [AP16] [11:22:33:44:55:66] <wired0> [D:E] ARP_REPLY : Sender 10.10.10.1 HwAddr 66:55:44:33:22:11
Dec  1 09:02:46 kernel: [*12/01/2020 09:02:46.0291] [1606813366: 29138] [AP16] [11:22:33:44:55:66] <wired0> [D:C] ARP_REPLY : Sender 10.10.10.1 HwAddr 66:55:44:33:22:11
Dec  1 09:02:46 kernel: [*12/01/2020 09:02:46.0291] [1606813366: 29187] [AP16] [11:22:33:44:55:66] <wired0> [D:C] ARP_REPLY : Sender 10.10.10.1 HwAddr 66:55:44:33:22:11
Dec  1 09:02:47 kernel: [*12/01/2020 09:02:47.0520] [1606813367: 52031] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] DHCP_REQUEST : TransId 0xa68db1f1
Dec  1 09:02:47 kernel: [*12/01/2020 09:02:47.0520] [1606813367: 52070] [AP16] [11:22:33:44:55:66] <apr1v0> [U:C] DHCP_REQUEST : TransId 0xa68db1f1
Dec  1 09:02:47 kernel: [*12/01/2020 09:02:47.0520] [1606813367: 52031] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] DHCP_REQUEST : TransId 0xa68db1f1
Dec  1 09:02:47 kernel: [*12/01/2020 09:02:47.0520] [1606813367: 52070] [AP16] [11:22:33:44:55:66] <apr1v0> [U:C] DHCP_REQUEST : TransId 0xa68db1f1
Dec  1 09:02:47 kernel: [*12/01/2020 09:02:47.0520] [1606813367: 52031] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] DHCP_REQUEST : TransId 0xa68db1f1
Dec  1 09:02:47 kernel: [*12/01/2020 09:02:47.0520] [1606813367: 52070] [AP16] [11:22:33:44:55:66] <apr1v0> [U:C] DHCP_REQUEST : TransId 0xa68db1f1
Dec  1 09:02:47 kernel: [*12/01/2020 09:02:47.0520] [1606813367: 52031] [AP16] [11:22:33:44:55:66] < wifi1> [U:W] DHCP_REQUEST : TransId 0xa68db1f1
Dec  1 09:02:47 kernel: [*12/01/2020 09:02:47.0520] [1606813367: 52070] [AP16] [11:22:33:44:55:66] <apr1v0> [U:C] DHCP_REQUEST : TransId 0xa68db1f1

So what does this tell us? The DHCP requests are not getting to the DHCP server.

The Fix

So based on the above, I doubled checked the switches.

Trunks and ports were fine BUT I had missed something!!

Show VLAN Brief

Showed me I hadnt actually added the sodding vlan on the switch…… 🙁

Why did other devices work?

Well we use one SSID and Cisco ISE moves RF scanners to a different vlan when they’ve authed. Other devices dont use our special RF scanners VLAN.

The Lesson

Its never Wireless, its always something else!

Today was an interesting, over the bank holiday I demoted an old 2003 domain controller, let’s call it Server1.

All went to plan, changed the IP address of the DNS on the network card and ran DCPROMO, nexting through all the options.

Reboot, ran a few tests:

DCDIAG on all the remaining domain controllers,
REPADMIN /REPLSUMMARY to test replication,
Logged onto a few PC’s to check they could authenticate ok.

Well that all seemed fine and all the results were great.

Tuesday comes round and I turn up to bedlam!!

Around 20 or 30 machines were referencing Server1 for DNS and since the dcpromo the DNS server only had Active Directory integrated DNS running meaning users couldn’t get to a host of sites!!

The question was though, why on earth were they referencing Server1?

To try and resolve I did the normal client side:

 Rebooted the client,
IPCONFIG /Release,
IPCONFIG /Renew,
IPCONFIG /FlushDNS,
Changed the settings in the registry:  HKLM/CCS/Services/TCP/Parameters

But no matter what I did the DNS server reverted back.

So after some thought I logged onto Sercver1 and checked DHCP because as you know DHCP assigns DNS and the default gateway etc to clients.

The service was enabled which worried me slightly and the Scopes were disabled but more importantly the Server options were all referencing the wrong server!!  So I quickly changed the options then disabled the DHCP service.

Rebooted the clients and Bam, everything was back up and running as it should.

For some strange reason the clients were using the  wrong DHCP server along with its incorrect settings.

So I urge you if you have this problem check all your DHCP server first!!