UCS Networking Adventure: A tale of CoS and The Vanishing Frames

The Problem

This week I had to connect an additional NetApp Storage System to my existing UCS environment through a different path than a similar shared storage platform that we utilize here. This shouldn’t be a big deal but there were a few caveats:

  • The Storage System was attached to a dedicated Nexus 5k for this customer
  • The VLAN configured on the customer switch collided with one configured in UCS so VLAN translation was necessary through access ports.
  • The traffic takes a different switching path from normal NFS traffic in this environment

I configured everything as one normally does when connecting to IP storage, jumbo frames and all. There was only one problem:

Standard Frames

~ # vmkping -s 1400 -d
PING ( 1400 data bytes
2508 bytes from icmp_seq=0 ttl=255 time=0.232 ms
2508 bytes from icmp_seq=1 ttl=255 time=0.198 ms
2508 bytes from icmp_seq=2 ttl=255 time=0.265 ms

--- ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.198/0.232/0.265 ms
~ #

Jumbo Frames

~ # vmkping -s 2500 -d
PING ( 2500 data bytes

--- ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
~ #


Now, normally this is just a simple issue of the MTU being set incorrectly somewhere along the traffic path but as I dug deeper into this issue that turned out to not be the case, it was something much stranger and more interesting. The IPs, VLANs and hostnames have been changed or obscured to protect the innocent. Onward!


BeyondVM is a personal blog is about virtualization, system administration archetecture and the business of IT. I post research that I do into better management of virtualization and infrastructure, as well as things that I learn along the way.

The views expressed anywhere on this site are strictly mine and not the position of any employer, vendor or provider including but not limited to my employer, VMware or any of its companies. Any solutions that I offer are 'use at your own risk.'
Twitter Follow Me!
Recient Comments