For a few years, jumbo frames have been all the rage for storage and network people. However, a lot of people don’t understand what exactly jumbo frames are and when/why they can help. Some would even be shocked to learn jumbo frames are not a panacea. From vendor to vendor, people claim that turning on jumbo frames gives a 10 to 30 percent performance increase, but in many situations that performance increase only happens at network saturation and only with certain workloads. Independent studies have shown the real gain of jumbo frames to be between zero and five percent for most environments.
So should you turn jumbo frames on? I’d say definitely not everywhere and possibly not at all, but with a little exploration of what jumbo frames are and why they may or may not be a good thing, you can decide for yourself.
Intro to Jumbo Frames
For starters, what are jumbo frames? The simple answer is that they are bigger-than-normal Ethernet frames. If that doesn’t mean anything to you, a normal Ethernet frame (layer 2) has 42-1500 bytes of payload. On top of that is about 22 bytes of overhead, which means 1.4 to up to 34 percent of the frame is overhead only. By definition, a jumbo frame has more than 1500 bytes of payload. Because of that, there is some flux in how large that frame is, but 1600-9000 bytes of payload is typical. The overhead (still 22 bytes) is between 0.2 and 1.3 percent and is significantly less. Past that, there is a super jumbo frame, which – you guessed it – is over 9000 bytes (generally up to 64000 bytes).
Great, less overhead is always a good thing; let’s move everything to jumbo frames, right? Not necessarily. There are three big reasons not to: not having jumbo frames enabled everywhere in the network, not having a standard for payload size, and increasing offload capabilities of network cards and CPUs. Let’s take a look at each.
Not Having it Enabled Everywhere on a VLAN/Network
Since jumbo frames are the exception to the rule, they are turned off by default on networking equipment and hosts. This immediately tells you there must be a problem with having them on by default. Not only are jumbo frames turned off on all VLANs, but they are also dropped by default on all interfaces without manual changes.
The problem comes from the fact that turning jumbo frames on basically raises the ceiling for the largest size frame you can send, but what if another device on the path doesn’t have that higher ceiling? Well, the short answer is that device can’t process the larger frame, so the frame gets fragmented into smaller within-normal-spec frames that are then transmitted, which induces latency. After that, the receiving device has to reassemble that fragmented frame and process the data. This isn’t simply a case where the device gains nothing and just functions as it normally would. This is fragmentation — the endpoint that has to store all those fragments long enough to reassemble the data and process it. This can greatly increase load on the device as well as cause drops and lower throughput substantially.
Also, think in terms of your whole VLAN/network: where are the frames originating? What’s on the edge of your network? Hosts also have to have jumbo frames turned on if they are going to be transmitting or receiving jumbo frames. That means every host on your jumbo frame network will have to have jumbo frames enabled manually or through scripts/GPOs. Are you going to be sending fragmented frames to your firewall for the internet? The internet isn’t kind to fragments. Since there are so many different paths for transfers to take to the other side before it can be defragmented, there will be a very high volume of drops. So if you do need jumbo frames, think about where you’ll want to enable them and what the implications of enabling them will be.
Not Having a Standard Size for Payload
Since the definition of a jumbo frame is anything larger than 1500 bytes of payload, there can be many different sizes defined for the jumbo frames. If a host defines one at 9020 but a switch defines it at 9000, fragmentation – along with all the issues listed above – will occur. Make sure that the jumbo frame size is the same everywhere, or at least larger on the networking gear than on the hosts originating the load.
Increased Offload Capabilities of Network Cards and CPUs
The final thing to keep in mind with regard to jumbo frames is how much better NICs and CPUs are at offloading overhead now. The hardware is so much more powerful and capable at offloading that it imposes a much lower load on the CPU than it used to … on top of being much faster. This means that the penalty for having extra frames isn’t as large as it used to be. Additionally, just because jumbo frames are enabled doesn’t mean that everything is going to leverage them. If the data being passed doesn’t fill more space than a normal frame, there is zero benefit to having jumbo frames. This is why you normally only see them as a big concern on storage or video networks where there are large streams of data.
So in short, when looking at jumbo frames, make sure to think through WHERE to enable them and if doing so will actually provide a benefit. From there, make sure to enable them properly at all parts of the VLAN/Network. An improper or incomplete implementation of jumbo frames can result in far more harm than help.