We are seeing issues when vmotioning a VM using vmxnet3 drivers. The VM is running CentOS 5.6 and we are using the vmxnet3 driver to take advantage of 10000mb (10gb). When we vmotion, the VM gets to 78% and then loses network connectivity. We are seeing that the mac address is not being set on the physical port that the VM is being vmotioned to. We can tell this by looking at the arp table on the ex4500 switch. We can see the mac address being deleted from the physical source port but not being set on the physical destination port. This is what we believe is breaking the network connectivity after the vmotion.
We have 2 10gb ex4500 switches which are not in a chassis - this is not yet supported. We have 2 10gb physical NICs in the ESX hosts and each pNIC has a cable plugged in to each of the physical EX4500 switches to provide redundancy. We have a vswitch which has the 2 10gb pNICs and have NIC teaming set on the vswitch - we have created port groups and have been changing various NIC teaming settings on the port group but have not been able to vmotion successfully. However, if we change promiscious mode to accept, we can vmotion successfully, bi-directionally to any ESX host which has the port group promiscious mode set to accept. We understand that placing a guest adapter in promiscuous mode causes it to detect all frames passed on the vswitch that are allowed under the VLAN policy for the port group that the adapter is connected to.
However if we use the e1000/vmxnet2, we can vmotion without any issues with promiscious mode set to reject. We also see the mac addrss being removed on the pysical source port and being set on the physical destination port, exactly how we believe it should work. The problem with the e1000 driver is that it does not support 10000mb. We do not feel turning on promiscious mode to accept is a feasible workaround.
Does anyone have any suggestions as to why vmotion does not work correctly using the vmxnet3 driver? We have been banging our heads against the wall. We will need to have all our VMs (~100 in this cluster) running vmxnet3.
Any help, advice or pointers would be appreciated. It's a tough one!
Oli