All hosts in our vSphere cluster, we are seeing "Physical Device Write Latency" with many peaks on several VMFS stores between 200ms and 3000ms (yes, 3 seconds). As you would expect, our users are seeing poor VM performance, guest disk errors, non-responsive VMs, database diconnections. But the NetApp storage statistics show performance is OK.
In some cases, switching the path for a VMFS to the other HBA makes the latency drop to 0 instantly. In other cases, this has no effect, but if we v-Motion the affected guest to another host, the latency and performance are fine, at least for a short time.
We firest saw this issue when we upgraded the cluster to ESX 4.1 U1. But we have since downgraded back to ESX 4.0 U2 and still see the same problems. Around the same time we also upgraded the NetApp from ONTAP 7.3.3 to 8.0.1P3 7-Mode.
Has anybody seen similar issues before?
Environment:
3 Clusters in the Datacenter. Cluster A has 4 x ESX 3.5 U5 hosts. Cluster B has 5 x ESX 3.5 U5 hosts. Cluster C has 10 x ESX 4.0 U2 hosts.
The SAN is a pair of NetApp FAS3140 filers in Active/Active mode running ONTAP 8.0.1P3 7-Mode. Some aggregates are FC disks, some are SATA disks.
All hosts in all 3 clusters are connected to all VMFS datastores. However we intend to separate out the 3.5 and 4.0 clusters to have their own dedicated datastores so that we can enable ALUA and Round Robin for the vSphere hosts.
ALUA is currently disabled on all NetApp iGroups.
HA and DRS are enabled and DRS is Fully Automated.
We have run the NetApp Virtual Storage Console 2.1 to set the recommended settings (timeouts and MPIO) on the ESX hosts. All paths are Fixed and are correctly set to the correct filer.
The Fibre Switches were recently upgraded to the latest firmware in an attempt to fix this issue, no effect.