A KB article by VMware was published earlier this month outlining a CBT bug in all the versions of ESX(i). The bug seems to have appeared after block tracking was introduced.
Backups and replicas for all VMs that had virtual disks expanded beyond 128 GB may be unrecoverable.
Our vendor, Veeam, is working hard at a hotfix for both the 7.0 Patch 4 and 8.0 code branches that will reset CBT automatically upon detecting source virtual disk size change. In the meantime, you must do a CBT manual reset for all VMs that had their virtual disks expanded. You must disable CBT and then the following backup run job should re-enable it automatically.
As long as the total amount the volume has been extended in its whole life exceeds 128 GB total, this bug will potentially affect you.
UPDATE: To clarify, here’s a table…
| 300 GB extend by 20 GB (10 times) | May have an issue |
| 300 GB extend by 20 GB, then 60 GB twice | May have an issue |
| 300 GB extend by 100 GB | Should not have an issue |
| 50 GB extend by 50 GB | Should not have an issue |
Disabling CBT
Before you start this process, ensure that there are no snapshots on your virtual machine. If you need to consolidate snapshots, VMware has instructions.
Be sure to notice that the “ctkEnabled” parameter must be set to false.
You could also disable CBT on all VMs using a PowerCLI script but the following job runs will take a while so it might be a job to do before the weekend:
From IT Walk Thru:
Get the VMs with CBT enabled:
$vms=get-vm | ?{$_.ExtensionData.Config.ChangeTrackingEnabled -eq $true}
Create a VM Specification to apply with the desired setting:
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.ChangeTrackingEnabled = $falseApply the specification to each VM, then create and remove a snapshot:
foreach($vm in $vms){
$vm.ExtensionData.ReconfigVM($spec)
$snap=$vm | New-Snapshot -Name ‘Disable CBT’
$snap | Remove-Snapshot -confirm:$false}Check for success:
get-vm | ?{$_.ExtensionData.Config.ChangeTrackingEnabled -eq $true}
SureBackup
It’s essential that you invest time into verifying your backups. Veeam makes it easier with its feature, SureBackup. Read DuRand’s post about setting up SureBackup in Veeam. The Backup Validator tool in Veeam is popular but it doesn’t do what SureBackup does and it may not even catch corruption issues like this one. Veeam reports that it will be improving its Backup Validator in v8 in order to catch moe issues like this one, but for now, SureBackup is the only way you can guarantee you can fully recover.
UPDATE, 11/5/15: Obviously, there is plenty more information out there about how to avoid this CBT bug and this has mostly been addressed with hotfixes from Veeam. To learn more, check out Veeam’s KB. And remember, you should still be verifying your backups!