Networking

Loops

Posted by robd on August 06, 2014
Networking / No Comments

So recently we had a total network failure, urrrgghhh!

HP Intelligent Management and VMware VCentre started to report that there was a network issue. The reports included a lost connection to every switch on the network and a critical temperate warning on our ESX server of 101 Celsius!!

Well after rebooting some switches and rebuilding a few more the issue was still present, no traffic over the switches….

So we started unplugging cables from the core switch as a last ditched attempt to fix the issue.  After unplugging about 30 suddenly the network came back to life!!

Long story short we had two days of total network failure because a cleaner plugged a one network cable into two active ports! NETWORK LOOP and no we didnt have spanning tree enabled.

So the next job was to biwire everything to separate switches and enable Spanning tree on every switch:

Or on a Vlan:

Note: When you configure a VLAN, the VLAN inherits the global STP settings. However, once you begin to define a VLAN, you can no longer configure standard STP parameters globally using the CLI. From that point on, you can configure STP only within individual VLANs.

Tags: , , , ,

The Meru AP to VPN to HP Switches to Controller issue

Posted by robd on April 08, 2014
Networking, Wireless / No Comments

Hi all,

As well as our main site we have a remote site, lets call it Remote1. Remote1 is on a basic ADSL line, the site connects to the main site via a site to site VPN between two SonicWall’s.  Remote1 has two Meru Access Points (AP332e) which are configured to communicate with the Meru controller at the main site which is where our issue was.

Here’s a pretty picture to help see what I’m on about:

MeruIssue

With the help of Meru support who were brilliant I carried out the follow analysis:

So normally Meru AP’s talk to the controller via UDP broadcast packets i.e. UDP port 9292, 9393.  If that doesnt work it uses layer 3 IP routing.

From the remote site I can ping (IP address, server name and broadcast address), telnet and http access the Meru Controller via the VPN. Great Layer 3 is good to go.

From the Controller I can ping the Access Points. Again great.

We have two AP’s on the remote site, to test one is set to L3 and one to L2 but neither work…hmmmm

From connecting to the AP’s via a cable we can see the packets are broadcasting and the AP’s have a valid IP address,

A packet trace on the firewalls show the UDP broadcast packets arrive and leave the remote firewall, are ingested and forwarded at the main site,

Wireshark

A port mirror on the controller shows no traffic from the remote site subnet.

A port mirror of the Main Sites firewall show the packets entering the network but when you connect to the next switch and port mirror I cant see any traffic (see wireshark results below):

wireshark2

 

So what the hell is going on???  Well it turned out I hadnt drawn the network diagram properly (above), here’s the proper topography:

MeruIssue2

Between the firewall and the first switch we have a Lightspeed Rocket that does a great job of email protection and website filtering.  Well after looking on the main web filtering page I noticed a tick box under “Block all unidentified UDP connections, Skype, UltraSurf type traffic, and file-sharing networks such as BitTorrent.”….well bugger!!

LightspeedBlock

So I un-ticked this section and Boom the AP’s came one line!!

Now this isnt great as users could start using P2P so I re-ticked the box and added a exception for AP’s and we have a winner!!!

Big thanks to Meru Support, Lightspeed Support, SonicWall Support, HP Support and Commercial LTD (who in the end helped find my missing piece in the diagram).

Tags: , , , , , , , ,

HP SAN P4300 Performance

Posted by robd on April 04, 2014
Networking, SAN, Server / No Comments

We had some issues a while back where there was latency between a PC and SAN so I started to look at all the layers to try and find a problem, here’s my findings in case anyone finds it a interesting read.

To rule out the PC, I’ve tested the performance of the SAN and network throughput.

The performance of the SAN which is based in IOP’S (Input/Output Operations Per Second) and the current average total is 800 (found on the SAN info page).  To put this prospectus, a poor performance would be in the 2000’s.

The below graph only shows output from 17:28 but has been running all day meaning the average should be accurate:

SAN1

Looking at the performance of switches can be difficult, but we’ve started using HP Intelligent Management Centre which is great at collating stats. The switches reported low bandwidth, CPU, memory and I/O seemed normal:

Switch1

We know from experience the throughput on these switches is limited by infrastructure in our case 1GB fibre.

These are the theoretical Max Sequential (SEQ) write limits we could obtain from our connection to the SAN (in practice there is a  5%-20% overhead involved):

SAN2

I’ve managed to very roughly test this write limit from a client to the SAN SAS disks:

SAN3

A result of 81.12MB/s is very positive, considering we can realistically over ever achieve 125MB on our current setup.

What this meant in my opinion was the SAN and network were not to blame meaning it was either client or server….Long story short the AV on the server was causing our latency not the “network”.

Tags: , , , ,

HP Switch – Access control Lists

Posted by robd on April 04, 2014
Networking / No Comments

Hi All,

We’ve recently implemented a BYOD wireless SSID for end users to check facebook or what ever it is they do.  As we’re short on resources we had to use existing DC’s to doll out IP’s but we were obviously worried around users “hacking” into the system.  So first thing we did was create a VLAN and assigned it to the wireless then applied several access control lists (ACL’s) to the core switch limiting the access to only the DC’s, proxy, core switch, each other and firewall then blocked the rest:

Firstly check the subnet mask of the VLAN you want to apply the ACL too as the wildcard/subnet address in the ACL will change dependant of the subnet.

The below example will use a subnet mask of 255.255.255.0 (the bits in red are notes only – do not try and apply them)

Logon to a switch and go into config mode:

 

Finally test!

Ok so what if you have a network that has a subnet of 255.255.252.0, well the wildcard changes in the ACL or above we had 0.0.0.255 where as in a 255.255.252.0 subnet we’d need 0.0.3.255.

Example:

 

So what happens if you want to delete a ACL from a VLAN:

 

 

Tags: , , , , ,

IP Routes

Posted by robd on March 29, 2014
Networking / No Comments

Hi All,

Started at a new company recently and they’ve had a ongoing issues for ages where they could only access one of two switches in their server room if they were on the same subnet.

So Switch A was the problem switch and had a IP of 10.0.0.123,

Switch B they could access from anywhere and had a IP of 10.0.0.124.

My Computer had a IP of 192.168.111.1.

So initially I started thinking the issue was down to vlans as we have around 50 vlans and I kind of just presumed it would be a issue with cross vlan routing.  Well actually I was wrong.

Firstly I pinged both switches:

SwicthA – Can not ping.

SwitchB – Can ping.

Next Trace route (TraceRT from a win 7 PC)

SwitchA – Could not trace

SwitchB – two hop trace, Core switch then the switch.

Ok so what next…..Lets try the above from the switches (so after connecting to server on the same subnet I ran the tests):

Ping my PC:

SwicthA – could not ping,

SwitchB – could ping.

Trace route:

SwitchA – No route,

SwitchB – two hop route.

So after this I decided it must be a routing issue, as both switches could ping the core switch and both switches could trace route the core switch.  So lets have a butchers at the routing.

 

Hold the phone, there’s no sodding return route to the core switch (10.0.0.2)

So I added the route like so:

And bobs your uncle it worked!!

Tags: , , , ,

HP Swicth – Show everything to a TFTP Server

Posted by robd on March 27, 2014
Networking / No Comments

So How do I send the output of the “show tech all ” command to a TFTP server?

Easy.

Launch the TFTP server (i.e. TGTPD64 from HERE), and issue this command at the CLI of the switch:

Tags: , , ,

Force traffic through a Network interface

Posted by robd on November 06, 2012
Networking / No Comments

Occasionally you may want to force traffic through a specific network interface.

For example, you have a VPN and you don’t want Chrome or IE (really, you use IE?)  to use the VPN as its slow and certain web sites may be restricted (because you shouldn’t be looking at facebook all day!)!

So what you need to do is change the network interface for the VPN to have a HIGH metric in the Advanced properties of the Networking interface.

This will force all traffic through your other network connection BUT and this is a big BUT, do the below after you have connected to what ever it is you want to connect to via your VPN (does that make sense? if not let me know in the comments)!

Turn your machine on, connect to the internet and then connect to the VPN, connect to what ever it is you do on your VPN,

Then Go to Network connections

Properties of undesirable network interface (VPN in this case but could be a Ethernet connection if you want to use your wireless for internet rather than Ethernet)

Properties > Double Click Internet Protocol Version 4 > Advanced

Deselect Automatic Metric

and enter a high number like 500

Ok Out,

and open Chrome or IE or what ever and check the IP to see what interface your routing out off!

 

Done

Tags: , , , , ,

Copy Protected by Chetan's WP-Copyprotect.