VMFS-6 read file error – Found stale lock

Once upon a time there was power cut and ESXi system went down.

… for these in rush, scroll down to “Fix” section…

After power-up none of details could be seen for VM which was up at the time of power cut. On VMs list page it was showing VMFS path to the file and that was it.

CLI level investigation shown that neither VM.vmx nor VM.vmdk files could be read and were raising error.

Fun…. fun… fun….

Disclaimer:
All below is against any good practice and/or anything Vmware would recommend as overwrites sensitive meta-data information of vmfs.
With that said, Vmware does not provide any way to fix it as of today and the only way would be to contact Vmware support.
All below is for educational purposes only and done at your own risk.

First steps

First steps were towards voma tool and output was similar to below (though below is taken from internet as no screenshots were taken at the time of the issue):

voma -m vmfs -f check -d <path_to_device>

returned output similar to below:

#https://www.reddit.com/r/vmware/comments/9j8k6o/esxi_vmfs6_datastore_corruption_after_host_reboot/

VOMA unfortunately does not support VMFS-6 in fix mode (as of 2018.11.02 on ESXi 6.5 and 6.7).

vmfs-tools (https://glandium.org/projects/vmfs-tools/) does not support VMFS-6 neither.

This left me in cul-de-sac… almost.

Big, big thanks to: Ulli aka continuum at communities.vmware.com helped to solve the issue.

Fix

1.
Dump heartbeat section of your VMFS-6 in question (.vf.sf file is in root folder of VMFS-6)

2.
Verify if the file contain only locks from your system (needs to be done on other system as strings is not available on ESXi, scp or any other way to get the file out of your system is your friend) :

3.
If above is confirmed, generate a clean heartbeat section using same build of ESXi and dump it to file:

4.
Transfer that clean file to your ESXi server and incorporate it into VMFS-6 with issues:

Within a minute or two earlier locked files should be accessible if not, try to reboot your ESXi.

 

Links:

Locked files with VMFS 6

Create a VMFS-Header-dump using an ESXi-Host in production

https://communities.vmware.com/thread/597513

Leave a Reply

Your email address will not be published. Required fields are marked *