While vSphere 6.5 came out with a bunch of great enhancements and features, Veeam may take a really large performance impact due to some fundamental changes. So, if you’re a Veeam shop and you’re planning to upgrade to vSphere 6.5, here are a few things you should be made aware of before you upgrade.
Network (NBD) Transport Mode Changes Cause Slow Backups
If you do NBD backups over the network, you’ll notice a large change in backup performance. vSphere 6.5 enforces NBD SSL across the board, and there is no longer an option to run NBD backups without it. In other words, your backups are all going through SSL. We’ve been testing this with our clients of varying sizes and ages of hardware and have been seeing a 30 to 50 percent reduction in Veeam backup performance via NDB transport mode.
VMware claimed that during their initial testing they did not experience such a performance hit. Personally, I don’t think they fully tested it across different environment types. However, VMware is aware of this large performance issue, so you probably can expect a focus on improvement here in subsequent updates.
You may also notice a decrease in the amount of concurrent connections on NDB. With vSphere 6.0 and Backup & Replication 9.5, we could reliably do 28 NBD connections at a time. However, with vSphere 6.5, it’s showing to be unstable at higher connection rates, so Veeam has lowered that maximum down to 20. While this may not seem like a big deal, if you’re in a large environment with lots of backup jobs, this could also negatively impact your backup time.
One new, cool feature is NBD Traffic Compression. This occurs on the ESXi host, and while it’s nice, be prepared to add some significant CPU overhead to run the compression. However, with the second level compression, we have seen up to 50 percent reduction in NBD traffic size. If you’re backing up on the WAN, this could be beneficial to you. Even if it takes up resources, it could minimize your bandwidth traffic a significant amount. This is not enabled by default within Veeam so you need to go manually turn it on.
Keep in mind that if you have a large backup window and you do it all via NBD, you’ll need to make adjustments. If you usually have an eight-hour window, add at least an additional couple of hours at first, and then add on time for each NBD connection you lost. Plan accordingly!
New Issues with Snapshotting (SEsparse)
There’s a new VDDK version with vSphere 6.5 that implements new features, but as any new version, it has some bugs you need to test before pushing it to production. One issue we’ve seen is that you can no longer enable CBT on a VM that has previous snapshots, so make sure that you’re always starting with a good copy of a server that does not have existing snapshots before you upgrade to vSphere 6.5.
The reason for the change in snapshot functionality is because it changes in vSphere 6.5 to SEsparse as the default snapshot for all VMs. In vSphere 6.0, we used a traditional snapshot for any disks below two terabytes and SEsparse for anything two terabytes and larger (which was not really recommended in the first place).
If the first section didn’t really apply to you since you’re not using NBD, you’re probably using Virtual HotAdd Proxies. Well, with vSphere 6.5 these proxies are showing to be up to ten times more demanding IOps-wise compared to running NBD. This can become an issue because it puts stress on your production storage that you are not used to having. With that being said, if you cannot withstand the IOps, Virtual HotAdd could even be slower than backing up via NBD.
SEsparse is actually more efficient in snapshotting, so as a whole it’s a positive feature. But, with anything relatively new, we have some bugs.
Changes and/or Discontinued Features within vSphere 6.5
The VIX API guest application-aware processing API was discontinued. This is important if you were using this to do your guest-aware processing or system indexing. There is a new one out there called the vSphere API for Guest Interaction and it only works on hosts on vSphere 6.5.
This new API is brand new, so be a bit suspicious and test it before trusting production with it.
Similarly, vSphere Tags has a new API and is only functional on vSphere 6.5. VMware says it is compatible with vSphere 6.0, but expect to have issues or bugs with that since it’s completely rewritten code. Again, test before pushing it to production.
If you do upgrade to vSphere 6.5, verify afterwards that your tags are still working properly. If a backup job is supposed to back up 10 VMs from a particular tag and only nine got updated with the tag, you could be missing backing up important data. There have been reports of this occurring.
Encrypted VMs
vSphere 6.5 now has encrypted VMs, but it comes with a few gotchas. It doesn’t necessarily have to do with Veeam itself, but it will impact your storage and your storage connections so it’s important to note here.
The biggest part of that is that it requires a Key Management Server that has to interact with your VMware environment in order to allow access to these encrypted VMs. So, if you have an unreliable or poorly configured KMS, you could lose your entire VMware environment. Direct storage access modes will be impossible for encrypted VMs, including VDDK SAN, Direct NFS and backup from storage snapshots (these are specific for Nimble, NetApp, HP , EMC and X-IO). Basically, if it directly runs outside of VMware, you’ll lose that functionality once you encrypt the VMs.
However, you can still do Hot Add but the proxy itself must also be encrypted, otherwise the backup will failover to NBD.
It is possible that the VDDK can fetch content as unencrypted even if it was encrypted in the VMDK, so consider enabling backup file encryption within Veeam. And regarding restores, it’s important to note that the restore process doesn’t change since the encryption occurs with VMware and not Veeam. You will still be able to restore encrypted VMs normally though Veeam, but restore them to encrypted VMware datastores to ensure continued protection.
vSphere 6.5 introduces a lot of changes. It’s a significant upgrade for your VMware environment, but make sure you’re thinking through these upgrades carefully before you do them. If you need help, we’re here.