Quantcast
Channel: VMware Communities : Popular Discussions - VMware ESX 4
Viewing all articles
Browse latest Browse all 36074

ESX host drops out of communication

$
0
0

Ok this one is a little unusual.

 

Here are the symptoms that i've been seeing.

 

The host has had two losses of communication in the past 7 days.  One loss of communication was at about 7:30 pm and the other 4 days later at about 8:30 pm.  The environment has two ESX hosts and a vcenter server which is a physical machine.  The drop out has been on the same host, we'll call it ESX1 for simplicity's sake.

 

When i look at the logs for ESX1 i see a loss of communication with the iSCSI datastore which is hosted by SVSAN.  At the same time as the communication drop off occurs there is a spike in disk latency over 20000 ms that lasts until ESX1 is rebooted. When i first logged in to the vcenter server from the vsphere client i could see the machines on ESX1 but could not console into them, nor could i rdp into them.  I had the same results if i logged into the local host with the vsphere client.

 

I'm thinking this is a network issue but i'm not sure.  Its happened to the same host twice with the exact same symptoms.  There are no data recovery activities running at all and there doesnt appear to be any functions happening on the network at that time. I have pasted some of the event logs below to show what is happening.  Basically what is posted is repeated until the issue is resolved with a hard reboot of the host.

 

Any help diagnosing this would be appreciated.  Even if you can just point me in the right direction.

 

Thanks

 

GI

 

 

Type      Time                    User                    Description        
warning   2/15/2012 5:51:05 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddf. Path vmhba33:C1:T2:L0 is down. Affected datastores: "WRSVMSAN1_Datastore3".
warning   2/15/2012 5:51:04 AM                            Lost path redundancy to storage device eui.0003398db20796e1. Path vmhba33:C0:T0:L0 is down. Affected datastores: "WRSVMSAN1_DataStore1".
info      2/15/2012 5:51:03 AM                            Successfully restored access to volume 4c2cb19e-aea13fd4-9bb0-0010186a63a2 (WRSVMSAN1_Datastore3) following connectivity issues.
info      2/15/2012 5:51:03 AM                            Successfully restored access to volume 4c28aa95-be6ce21a-de6e-0010186a63a0 (WRSVMSAN1_DataStore1) following connectivity issues.
info      2/15/2012 5:50:47 AM                            Lost access to volume 4c2cb19e-aea13fd4-9bb0-0010186a63a2 (WRSVMSAN1_Datastore3) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
warning   2/15/2012 5:50:47 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddf. Path vmhba33:C0:T2:L0 is down. Affected datastores: "WRSVMSAN1_Datastore3".
info      2/15/2012 5:50:47 AM                            Successfully restored access to volume 4c28addc-12e3a95b-9e7d-0010186a63a0 (WRSVMSAN2_Datastore2) following connectivity issues.
info      2/15/2012 5:50:32 AM                            Lost access to volume 4c28addc-12e3a95b-9e7d-0010186a63a0 (WRSVMSAN2_Datastore2) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
warning   2/15/2012 5:50:32 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddd. Path vmhba33:C0:T1:L0 is down. Affected datastores: "WRSVMSAN2_Datastore2".
warning   2/15/2012 5:50:31 AM                            Lost path redundancy to storage device eui.0003398db20796e1. Path vmhba33:C1:T0:L0 is down. Affected datastores: "WRSVMSAN1_DataStore1".
warning   2/15/2012 5:51:15 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddd. Path vmhba33:C1:T1:L0 is down. Affected datastores: "WRSVMSAN2_Datastore2".
error     2/15/2012 5:51:13 AM                            Lost connectivity to storage device eui.000339fbf07c9ddf. Path vmhba33:C1:T2:L0 is down. Affected datastores: "WRSVMSAN1_Datastore3".
error     2/15/2012 5:51:12 AM                            Lost connectivity to storage device eui.0003398db20796e1. Path vmhba33:C0:T0:L0 is down. Affected datastores: "WRSVMSAN1_DataStore1".
warning   2/15/2012 5:51:10 AM                            Lost path redundancy to storage device eui.0003398db20796e1. Path vmhba33:C1:T0:L0 is down. Affected datastores: "WRSVMSAN1_DataStore1".
info      2/15/2012 5:51:10 AM                            Successfully restored access to volume 4c28addc-12e3a95b-9e7d-0010186a63a0 (WRSVMSAN2_Datastore2) following connectivity issues.
warning   2/15/2012 5:51:10 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddf. Path vmhba33:C0:T2:L0 is down. Affected datastores: "WRSVMSAN1_Datastore3".
info      2/15/2012 5:51:10 AM                            Lost access to volume 4c28addc-12e3a95b-9e7d-0010186a63a0 (WRSVMSAN2_Datastore2) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
warning   2/15/2012 5:51:10 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddd. Path vmhba33:C0:T1:L0 is down. Affected datastores: "WRSVMSAN2_Datastore2".
info      2/15/2012 5:50:01 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 5:50:01 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 5:49:32 AM                            Alarm 'Health status changed alarm' on entity Datacenters send SNMP trap
info      2/15/2012 5:49:32 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 5:49:32 AM                            Alarm 'Health status changed alarm' on Datacenters triggered an action
info      2/15/2012 5:49:31 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 3:48:08 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 3:48:08 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 3:47:38 AM                            Alarm 'Health status changed alarm' on entity Datacenters send SNMP trap
info      2/15/2012 3:47:38 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 3:47:38 AM                            Alarm 'Health status changed alarm' on Datacenters triggered an action
info      2/15/2012 3:47:38 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 2:26:32 AM                            Alarm 'Virtual machine cpu usage' on Citrix Access Gateway changed from Yellow to Red
info      2/15/2012 2:26:12 AM                            Alarm 'Virtual machine cpu usage' on Citrix Access Gateway changed from Green to Yellow
info      2/15/2012 1:47:46 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 1:47:46 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 1:46:45 AM                            Alarm 'Health status changed alarm' on entity Datacenters send SNMP trap
info      2/15/2012 1:46:45 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 1:46:45 AM                            Alarm 'Health status changed alarm' on Datacenters triggered an action
info      2/15/2012 1:46:45 AM                            Alarm 'Health status changed alarm' on Datacenters changed from Gray to Gray
info      2/15/2012 1:39:16 AM                            Alarm 'Cannot connect to storage' on wrsvmsan2.holmdel.k12.nj.us changed from Gray to Gray
info      2/15/2012 1:39:15 AM                            Removed datastore WRSVMSAN1_DataStore1 from wrsvmsan2.holmdel.k12.nj.us in DataCenter
warning   2/15/2012 1:35:42 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddf. Path vmhba33:C1:T2:L0 is down. Affected datastores: "WRSVMSAN1_Datastore3".
info      2/15/2012 1:37:30 AM                            Alarm 'Cannot connect to storage' on wrsvmsan1.holmdel.k12.nj.us changed from Gray to Gray
info      2/15/2012 1:37:30 AM                            Alarm 'Cannot connect to storage' on entity wrsvmsan1.holmdel.k12.nj.us send SNMP trap
info      2/15/2012 1:37:30 AM                            Alarm 'Cannot connect to storage' on wrsvmsan1.holmdel.k12.nj.us changed from Gray to Gray
info      2/15/2012 1:37:30 AM                            Alarm 'Cannot connect to storage' on wrsvmsan1.holmdel.k12.nj.us changed from Gray to Gray
info      2/15/2012 1:37:30 AM                            Alarm 'Cannot connect to storage' on wrsvmsan1.holmdel.k12.nj.us triggered an action
info      2/15/2012 1:37:30 AM                            Alarm 'Cannot connect to storage' on wrsvmsan1.holmdel.k12.nj.us changed from Gray to Gray
warning   2/15/2012 1:37:23 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddf. Path vmhba33:C1:T2:L0 is down. Affected datastores: "WRSVMSAN1_Datastore3".
info      2/15/2012 1:37:22 AM                            Successfully restored access to volume 4c2cb19e-aea13fd4-9bb0-0010186a63a2 (WRSVMSAN1_Datastore3) following connectivity issues.
warning   2/15/2012 1:37:21 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddf. Path vmhba33:C0:T2:L0 is down. Affected datastores: "WRSVMSAN1_Datastore3".
warning   2/15/2012 1:37:08 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddd. Path vmhba33:C1:T1:L0 is down. Affected datastores: "WRSVMSAN2_Datastore2".
info      2/15/2012 1:37:05 AM                            Lost access to volume 4c28aa95-be6ce21a-de6e-0010186a63a0 (WRSVMSAN1_DataStore1) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
warning   2/15/2012 1:37:04 AM                            Lost path redundancy to storage device eui.0003398db20796e1. Path vmhba33:C1:T0:L0 is down. Affected datastores: "WRSVMSAN1_DataStore1".
info      2/15/2012 1:37:00 AM                            Lost access to volume 4c2cb19e-aea13fd4-9bb0-0010186a63a2 (WRSVMSAN1_Datastore3) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
info      2/15/2012 1:37:00 AM                            Successfully restored access to volume 4c28addc-12e3a95b-9e7d-0010186a63a0 (WRSVMSAN2_Datastore2) following connectivity issues.
info      2/15/2012 1:37:00 AM                            Lost access to volume 4c28addc-12e3a95b-9e7d-0010186a63a0 (WRSVMSAN2_Datastore2) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
info      2/15/2012 1:37:20 AM                            Alarm 'Cannot connect to storage' on wrsvmsan2.holmdel.k12.nj.us changed from Gray to Gray
info      2/15/2012 1:37:20 AM                            Alarm 'Cannot connect to storage' on wrsvmsan2.holmdel.k12.nj.us changed from Gray to Gray
info      2/15/2012 1:37:20 AM                            Alarm 'Cannot connect to storage' on entity wrsvmsan2.holmdel.k12.nj.us send SNMP trap
info      2/15/2012 1:37:20 AM                            Alarm 'Cannot connect to storage' on wrsvmsan2.holmdel.k12.nj.us changed from Gray to Gray
info      2/15/2012 1:37:20 AM                            Alarm 'Cannot connect to storage' on wrsvmsan2.holmdel.k12.nj.us changed from Gray to Gray
info      2/15/2012 1:37:20 AM                            Alarm 'Cannot connect to storage' on wrsvmsan2.holmdel.k12.nj.us triggered an action
info      2/15/2012 1:37:19 AM                            Alarm 'Cannot connect to storage' on wrsvmsan2.holmdel.k12.nj.us changed from Gray to Gray
warning   2/15/2012 1:37:08 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddd. Path vmhba33:C1:T1:L0 is down. Affected datastores: "WRSVMSAN2_Datastore2".
warning   2/15/2012 1:37:06 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddf. Path vmhba33:C0:T2:L0 is down. Affected datastores: "WRSVMSAN1_Datastore3".
warning   2/15/2012 1:37:03 AM                            Lost path redundancy to storage device eui.0003398db20796e1. Path vmhba33:C1:T0:L0 is down. Affected datastores: "WRSVMSAN1_DataStore1".
warning   2/15/2012 1:37:03 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddf. Path vmhba33:C0:T2:L0 is down. Affected datastores: "WRSVMSAN1_Datastore3".
warning   2/15/2012 1:37:03 AM                            Lost path redundancy to storage device eui.000339fbf07c9ddd. Path vmhba33:C0:T1:L0 is down. Affected datastores: "WRSVMSAN2_Datastore2".
info      2/15/2012 1:36:59 AM                            Successfully restored access to volume 4c28addc-12e3a95b-9e7d-0010186a63a0 (WRSVMSAN2_Datastore2) following connectivity issues.

Viewing all articles
Browse latest Browse all 36074

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>