Hello All,
I have a Sun x4150 running ESX 4.0.0 164009 (going to patch this weekend) in pre-production.
I recently noticed that VMs were running sluggish and the console was loaded down. The VI client does not show anything other than the names of the VMs. Rebooting the host seems to clear the problem for about 20 hours.
After checking /var/logs, I noticed that my 5 Gig /var partition was nearly full on an install less than a month old. The vmkwarning file has two repeating messages over and over:
Oct 22 21:05:50 esx-host-3 vmkernel: 0:21:06:09.674 cpu2:4206)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device "naa.600144f04aa71afa000021283d7a3200" - failed to issue command due to Not found (APD), try again...
Oct 22 21:05:50 esx-host-3 vmkernel: 0:21:06:09.674 cpu2:4206)WARNING: NMP: nmp_DeviceAttemptFailover: Logical device "naa.600144f04aa71afa000021283d7a3200": awaiting fast path state update...
Oct 22 21:05:51 esx-host-3 vmkernel: 0:21:06:10.437 cpu1:4248)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba39:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"
Oct 22 21:05:51 esx-host-3 vmkernel: 0:21:06:10.437 cpu1:4248)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess ISID: 00023d000001 TARGET: iqn.1986-03.com.sun:02:4213f0b0-9c16-c8d8-c731-bb6d87d01a07 TPGT: 1 TSIH: 0
Oct 22 21:05:51 esx-host-3 vmkernel: 0:21:06:10.437 cpu1:4248)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn CID: 0 L: 192.168.1.22:62888 R: 192.168.1.10:3260
Oct 22 21:05:51 esx-host-3 vmkernel: 0:21:06:10.674 cpu2:4206)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device "naa.600144f04aa71afa000021283d7a3200" - issuing command 0x410006050a00
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1460: Heap vmklinux (20968056/20971520): Maximum allowed growth (3464) too small for size (65536)
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1645: Heap_Align(vmklinux, 312/312 bytes, 64 align) failed. caller: 0x418005e5f27e
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1460: Heap vmklinux (20968056/20971520): Maximum allowed growth (3464) too small for size (65536)
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1645: Heap_Align(vmklinux, 312/312 bytes, 64 align) failed. caller: 0x418005e5f27e
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1460: Heap vmklinux (20968056/20971520): Maximum allowed growth (3464) too small for size (65536)
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1645: Heap_Align(vmklinux, 312/312 bytes, 64 align) failed. caller: 0x418005e5f27e
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1460: Heap vmklinux (20968056/20971520): Maximum allowed growth (3464) too small for size (65536)
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1645: Heap_Align(vmklinux, 312/312 bytes, 64 align) failed. caller: 0x418005e5f27e
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1460: Heap vmklinux (20968056/20971520): Maximum allowed growth (3464) too small for size (65536)
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1645: Heap_Align(vmklinux, 312/312 bytes, 64 align) failed. caller: 0x418005e5f27e
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1460: Heap vmklinux (20968056/20971520): Maximum allowed growth (3464) too small for size (65536)
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1645: Heap_Align(vmklinux, 312/312 bytes, 64 align) failed. caller: 0x418005e5f27e
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1460: Heap vmklinux (20968056/20971520): Maximum allowed growth (3464) too small for size (65536)
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1645: Heap_Align(vmklinux, 312/312 bytes, 64 align) failed. caller: 0x418005e5f27e
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1460: Heap vmklinux (20968056/20971520): Maximum allowed growth (3464) too small for size (65536)
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1645: Heap_Align(vmklinux, 312/312 bytes, 64 align) failed. caller: 0x418005e5f27e
Oct 22 23:03:48 esx-host-3 vmkernel: 0:23:04:07.206 cpu7:4314)WARNING: Heap: 1460: Heap vmklinux (20968056/20971520): Maximum allowed growth (3464) too small for size (65536)
After Googling the Internet machine, it seems the above errors were experienced by admins of ESX 3.0.1 and VMware supposedly fixed a bug related to this message in ESX 3.0.2
I also noted that in forum posting the errors were associated with insufficient settings on VMFS3.MaxHeapSizeMB and possibly COW.COWMaxHeapSizeMB.
I currently have VMFS3.MaxHeapSizeMB = 128 and COW.COWMaxHeapSizeMB = 192
Can anyone shed some light on this issue? Is this a configuration issue on my part? Has anyone else seen this on 4.0.0? Could this possibly be hardware/memory?
Thanks,
Phil