I have the strangest phenomenon I would like to share and hopefully get solved.
For a couple of weeks I have one host in my cluster wich doesn't show his cpu in the vcenter or even in the viclient. When I do esxtop it is really slow.
This host gives me a lot of warnings in the vmkernel log. (I show below)
When I stop the vmware management service the problem and the logs shift towards another host in my cluster. When I start the vmware management service again the problem comes back.
The biggest issue I have is that the host with this errors is very unresponsive and vmotion and snapshoting doesn't hardly work because the communication between the vcenter and the host is bad. A vmotion can work. I see the host in the esxtop of the host I'm vmotioning towards but the vcenter doesn't know it and stays at 78% and sometimes after 10-30 minutes it succeerds or fails.
Nov 3 15:31:40 svresx023 vmkernel: gen 31737, mode 1, owner 4e906734-179edb35-0b0b-0017a477fca8 mtime 3469311] on vol 'ESX_DS3_LUN3_750_PathA'
Nov 3 15:31:44 svresx023 vmkernel: 34:03:00:21.011 cpu16:4126)FS3: 3099: [Req mode: 1] Lock [type 10c00001 offset 72628224 v 1747, hb offset 3833856
Nov 3 15:31:44 svresx023 vmkernel: gen 4445, mode 1, owner 4e8a01c4-31f70f49-4e00-0017a477fc34 mtime 1635804] is not free on vol 'ESX_DS2_LUN2_750_PathA'
Nov 3 15:31:44 svresx023 vmkernel: 34:03:00:21.031 cpu1:4128)FS3: 3099: [Req mode: 1] Lock [type 10c00001 offset 47818752 v 6053, hb offset 4083712
Nov 3 15:31:44 svresx023 vmkernel: gen 89, mode 1, owner 4e9161fe-59317166-128d-0017a477fc5c mtime 137991] is not free on vol 'ESX_DS3_LUN3_750_PathA'
Nov 3 15:31:44 svresx023 vmkernel: 34:03:00:21.054 cpu1:4127)FS3: 3099: [Req mode: 1] Lock [type 10c00001 offset 22554624 v 2957, hb offset 4128768
Nov 3 15:31:44 svresx023 vmkernel: gen 31737, mode 1, owner 4e906734-179edb35-0b0b-0017a477fca8 mtime 3469311] is not free on vol 'ESX_DS3_LUN3_750_PathA'
Nov 3 15:31:44 svresx023 vmkernel: 34:03:00:21.100 cpu19:4126)FS3: 3001: [Req mode: 1] Checking liveness of [type 10c00001 offset 64999424 v 6421, hb offset 4128768
Nov 3 15:31:44 svresx023 vmkernel: gen 31737, mode 1, owner 4e906734-179edb35-0b0b-0017a477fca8 mtime 4459595] on vol 'ESX_DS3_LUN3_750_PathA'
Nov 3 15:31:44 svresx023 vmkernel: 34:03:00:21.157 cpu7:4128)FS3: 3001: [Req mode: 1] Checking liveness of [type 10c00001 offset 4593664 v 3115, hb offset 4063232
Nov 3 15:31:44 svresx023 vmkernel: gen 1281, mode 1, owner 4e8cddbf-8cfbaa63-bd0b-0017a477fc7c mtime 558216] on vol 'ESX_DS3_LUN3_750_PathA'
Nov 3 15:31:44 svresx023 vmkernel: 34:03:00:21.197 cpu7:4127)FS3: 3001: [Req mode: 1] Checking liveness of [type 10c00001 offset 72634368 v 1744, hb offset 3833856
Nov 3 15:31:44 svresx023 vmkernel: gen 4445, mode 1, owner 4e8a01c4-31f70f49-4e00-0017a477fc34 mtime 1635797] on vol 'ESX_DS2_LUN2_750_PathA'
Nov 3 15:31:48 svresx023 vmkernel: 34:03:00:25.107 cpu13:4126)FS3: 3099: [Req mode: 1] Lock [type 10c00001 offset 64999424 v 6421, hb offset 4128768
Nov 3 15:31:48 svresx023 vmkernel: gen 31737, mode 1, owner 4e906734-179edb35-0b0b-0017a477fca8 mtime 4459595] is not free on vol 'ESX_DS3_LUN3_750_PathA'
Nov 3 15:31:48 svresx023 vmkernel: 34:03:00:25.159 cpu2:4128)FS3: 3099: [Req mode: 1] Lock [type 10c00001 offset 4593664 v 3115, hb offset 4063232
Nov 3 15:31:48 svresx023 vmkernel: gen 1281, mode 1, owner 4e8cddbf-8cfbaa63-bd0b-0017a477fc7c mtime 558216] is not free on vol 'ESX_DS3_LUN3_750_PathA'
Nov 3 15:31:48 svresx023 vmkernel: 34:03:00:25.207 cpu2:4127)FS3: 3099: [Req mode: 1] Lock [type 10c00001 offset 72634368 v 1744, hb offset 3833856
Nov 3 15:31:48 svresx023 vmkernel: gen 4445, mode 1, owner 4e8a01c4-31f70f49-4e00-0017a477fc34 mtime 1635797] is not free on vol 'ESX_DS2_LUN2_750_PathA'
Nov 3 15:31:48 svresx023 vmkernel: 34:03:00:25.266 cpu5:4127)FS3: 3001: [Req mode: 1] Checking liveness of [type 10c00001 offset 59633664 v 2647, hb offset 3715072