VM's not starting after reboot vSphere 4 update 2 server

Hello,

After reboot of a vSphere 4 update 2 server the VM's are not started and connecting vSphere client not possible.

If running the vmware-cmd command following error comes: vim.fault.HostConnectFault: Failed to acquire session manager while querying for local ticket. ((111, 'Connection refused'))

The hostd log:

Log for VMware ESX, pid=22941, version=4.0.0, build=build-261974, option=Release, section=2

Compression of /var/log/vmware//hostd-1.log to /var/log/vmware//hostd-1.log.gz took 1 ms

Current working directory: /var/log/vmware

Thread info: Min Io, Max Io, Min Task, Max Task, Max Thread, Keepalive, thread kill, max fds: 2, 200, 2, 10, 25, 8, 600, 2048

Setting system limit of 2048

Set system limit to 2048

Executable directory: /usr/lib/vmware/bin

Plugin base directory: /usr/lib/vmware/bin

Trying blklistsvc

Plugin 0 statically linked

Block List Service Plugin initialized

Trying cimsvc

Plugin 1 statically linked

Initializing cimsvc plugin

Created CimManagerImpl

Trying directorysvc

Plugin 2 statically linked

Directory service initialized

Command: Esxtop created

Service added: Esxtop:

Plugin initialized

Trying hostsvc

Plugin 3 statically linked

CreateHostsvcFactory: mockup mode is false

Storage data synchronization policy set to invalidate_change

undoAttempts(15), timeoutMsec(2000)

FSVolumeVmkImplProvider::Start() called

FSVolumeVmkImplProvider::Start() called

Initializing resource pool tree

Resource pool configuration: /etc/vmware/hostd/pools.xml

Looking up VMkernel resource group 'host/user'

Initializing vms in resource pool tree

Fetch all physical NICs....

Fetched physical NIC: vmnic0.

Fetched physical NIC: vmnic1.

Loading DVS information

DvsVmkProvider::Refresh called

DVPortgroups file not found. Assuming no DVPortgroups present

Fetch all virtual switches....

Successfully fetched virtual switch: vSwitch0.

RefreshVmkernelNic: called

InstallNics: Fetch all VmKernel NICs....

RefreshConsoleNic: Fetch all Console NICs....

vswif0: Network::IpConfig { IsValid() : 1 IsIpv4Valid() : 1 GetUseDhcp() : 0 GetDhcpDns() : 0 GetIpv4Address() : 172.20.149.130 GetIpv4Netmask() : 255.255.255.0 GetIpv4UseNone() : 0 IsIpv6Valid() 0 V6 Addresses:  total = 0 GetUseIpv6Dhcp() : 0 GetUseIpv6RouterAdvertised() : 0 }

ProcessConsoleNic: nic vswif0 added 1 item to NetworkInfo:ConsoleVnic array

Successfully fetched console NIC: vswif0.

All vnics refreshed, found 0 disabled.

Obtaining DNS Configuration....

Dns Configuration: Network::DnsConfig { GetHostname(): RB031149, GetPreferredHostname():, GetDomain(): retail.local, 172.20.149.27, 10.254.1.221, retail.local, GetSearchDomainString(): retail.local, GetName(): DnsConfigImpl} }

Obtaining Route Configuration....

SetIPv6Capability: IPv6 is disabled

Datastore auto-refresh is disabled.

Datastore alias configuration file is /etc/vmware/hostd/datastores.xml

ReconcileVMFSDatastores called: refresh = true, rescan = false

RefreshVMFSVolumes called

RefreshVMFSVolumes: refreshed volume, id 4c5698ac-ac8e0416-fd7d-001a4bf07116, name Storage2

RefreshVMFSVolumes: refreshed volume, id 4616d939-d7efd4db-3cc1-001a4bae203c, name storage1

RefreshOneVmfsVolume on /vmfs/volumes/4c5698ac-ac8e0416-fd7d-001a4bf07116

Refresh: VMFS Datastore name was changed from '4c5698ac-ac8e0416-fd7d-001a4bf07116' to 'Storage2'.

Looking up object with name = "ha-vmsvc" failed.

4c5698ac-ac8e0416-fd7d-001a4bf07116-datastorebrowser::IntializeCOSDirectory: COS path: /vmimages/

Looking up object with name = "ha-view-mgr-impl" failed.

4c5698ac-ac8e0416-fd7d-001a4bf07116-datastorebrowser::Constructor

Looking up object with name = "ha-view-mgr-impl" failed.

DiscoverNewDatastores: Storage2 created.

RefreshOneVmfsVolume on /vmfs/volumes/4616d939-d7efd4db-3cc1-001a4bae203c

Refresh: VMFS Datastore name was changed from '4616d939-d7efd4db-3cc1-001a4bae203c' to 'storage1'.

Looking up object with name = "ha-vmsvc" failed.

4616d939-d7efd4db-3cc1-001a4bae203c-datastorebrowser::IntializeCOSDirectory: COS path: /vmimages/

Looking up object with name = "ha-view-mgr-impl" failed.

4616d939-d7efd4db-3cc1-001a4bae203c-datastorebrowser::Constructor

Looking up object with name = "ha-view-mgr-impl" failed.

DiscoverNewDatastores: storage1 created.

ReconcileVMFSDatastores: Done discovering new filesystem volumes.

ReconcileNASDatastores: Discovering new filesystem volumes.

RefreshNASVolumes called

ReconcileNASDatastores: Done discovering new filesystem volumes.

Looking up object with name = "ha-view-mgr-impl" failed.

Looking up object with name = "ha-vmsvc" failed.

ha-host-datastorebrowser::IntializeCOSDirectory: COS path: /vmimages/

Looking up object with name = "ha-view-mgr-impl" failed.

ha-host-datastorebrowser::Constructor

VM autostart configuration: /etc/vmware/hostd/vmAutoStart.xml

Looking up object with name = "ha-view-mgr-impl" failed.

bind to socket. /var/run/vmware/hostd-vprob.sock

Created vprob socket: 8

Host Timezone 'Europe/Amsterdam' loaded

Loading ntp server list '/etc/ntp.conf'

Adding NTP server nl.pool.ntp.org

PciPassthru info at startup

Loading service configuration file '/etc/vmware/service/services.xml'

Invoking command /etc/init.d/sshd status 

Command finished with status 0

Invoking command /etc/init.d/vmware-vpxa status 

Command /etc/init.d/vmware-vpxa failed

Invoking command /etc/init.d/ntpd status 

Command finished with status 0

Invoking command /etc/init.d/vmware-webAccess status 

Command finished with status 0

Loading firewall configuration file '/etc/vmware/firewall/symantecNetBackup.xml'

Loading firewall configuration file '/etc/vmware/firewall/esxupdate.xml'

Loading firewall configuration file '/etc/vmware/firewall/legatoNetworker.xml'

Loading firewall configuration file '/etc/vmware/firewall/ibmTSM.xml'

Loading firewall configuration file '/etc/vmware/firewall/symantecBackupExec.xml'

Loading firewall configuration file '/etc/vmware/firewall/caARCserve.xml'

Loading firewall configuration file '/etc/vmware/firewall/services.xml'

Loading firewall configuration file '/etc/vmware/firewall/commvaultGalaxy.xml'

Invoking command /sbin/chkconfig vmware-webAccess off

Command finished with status 0

UpdateService: stop vmware-webAccess

Invoking command /etc/init.d/vmware-webAccess stop 

Command finished with status 0

Looking up object with name = "ha-view-mgr-impl" failed.

Resource pool instantiated

Looking up object with name = "ha-view-mgr-impl" failed.

Refresh interval: 60 seconds

Loaded resources for root resource pool 'ha-root-pool'

Registered advanced option 'VMkernel.Boot.ACPI'

Registered advanced option 'VMkernel.Boot.acpiDbgLevel'

Registered advanced option 'VMkernel.Boot.assumeCommonBusClock'

Registered advanced option 'VMkernel.Boot.assumePerNodeBusClock'

Registered advanced option 'VMkernel.Boot.baudRate'

Registered advanced option 'VMkernel.Boot.buddyPhysicalMemoryDebugStruct'

Registered advanced option 'VMkernel.Boot.busSpeedMayVary'

Registered advanced option 'VMkernel.Boot.busSpeedMayVaryPerNode'

Registered advanced option 'VMkernel.Boot.checkCPUIDLimit'

Registered advanced option 'VMkernel.Boot.clockMultiplierMayVary'

Registered advanced option 'VMkernel.Boot.clockMultiplierMayVaryPerNode'

Registered advanced option 'VMkernel.Boot.coresPerPkg'

Registered advanced option 'VMkernel.Boot.correctBSPMTRRMasks'

Registered advanced option 'VMkernel.Boot.custom'

Registered advanced option 'VMkernel.Boot.debugBreak'

Registered advanced option 'VMkernel.Boot.debugLogToSerial'

Registered advanced option 'VMkernel.Boot.debugPort'

Registered advanced option 'VMkernel.Boot.disableC1E'

Registered advanced option 'VMkernel.Boot.disableCFOH'

Registered advanced option 'VMkernel.Boot.disableTurbo'

Registered advanced option 'VMkernel.Boot.dumpDiag'

Registered advanced option 'VMkernel.Boot.executePOST'

Registered advanced option 'VMkernel.Boot.fakeNUMAnodes'

Registered advanced option 'VMkernel.Boot.fastHzEstimate'

Registered advanced option 'VMkernel.Boot.forceCRS'

Registered advanced option 'VMkernel.Boot.fsCheck'

Registered advanced option 'VMkernel.Boot.generalCriticalMemory'

Registered advanced option 'VMkernel.Boot.hyperthreading'

Registered advanced option 'VMkernel.Boot.ioapic'

Registered advanced option 'VMkernel.Boot.ipmiEnabled'

Registered advanced option 'VMkernel.Boot.logicalApicId'

Registered advanced option 'VMkernel.Boot.logOnScreen'

Registered advanced option 'VMkernel.Boot.maxPCPUS'

Registered advanced option 'VMkernel.Boot.mcaClearBanksOnMCE'

Registered advanced option 'VMkernel.Boot.mcaEnableAllErrorSources'

Registered advanced option 'VMkernel.Boot.mcaEnableScrubber'

Registered advanced option 'VMkernel.Boot.mcaFindDIMMFromAddr'

Registered advanced option 'VMkernel.Boot.measureNUMALatency'

Registered advanced option 'VMkernel.Boot.memCheckEveryWord'

Registered advanced option 'VMkernel.Boot.memLowReservedMB'

Registered advanced option 'VMkernel.Boot.memmapMaxPhysicalMemMB'

Registered advanced option 'VMkernel.Boot.memmapStressHighBitMPNs'

Registered advanced option 'VMkernel.Boot.mpsIntRouting'

Registered advanced option 'VMkernel.Boot.netDVSSyncEnabled'

Registered advanced option 'VMkernel.Boot.netESX2LegacyMode'

Registered advanced option 'VMkernel.Boot.netMaxPCPUPktCacheSize'

Registered advanced option 'VMkernel.Boot.netNetqueueEnabled'

Registered advanced option 'VMkernel.Boot.netNumGPBuffers'

Registered advanced option 'VMkernel.Boot.netNumPortsets'

Registered advanced option 'VMkernel.Boot.netPanicBadDevOpen'

Registered advanced option 'VMkernel.Boot.netPktBufUseSlab'

Registered advanced option 'VMkernel.Boot.netPktHeapMaxSize'

Registered advanced option 'VMkernel.Boot.netPktHeapMinSize'

Registered advanced option 'VMkernel.Boot.netPktNumSlabSizes'

Registered advanced option 'VMkernel.Boot.nmiAction'

Registered advanced option 'VMkernel.Boot.noIOMMU'

Registered advanced option 'VMkernel.Boot.oem'

Registered advanced option 'VMkernel.Boot.overrideSerialPortAddr'

Registered advanced option 'VMkernel.Boot.pageSharing'

Registered advanced option 'VMkernel.Boot.powerManagement'

Registered advanced option 'VMkernel.Boot.realNMI'

Registered advanced option 'VMkernel.Boot.serialPort'

Registered advanced option 'VMkernel.Boot.sharePerNode'

Registered advanced option 'VMkernel.Boot.storageHeapMaxSize'

Registered advanced option 'VMkernel.Boot.storageHeapMinSize'

Registered advanced option 'VMkernel.Boot.techSupportMode'

Registered advanced option 'VMkernel.Boot.timerEnableACPI'

Registered advanced option 'VMkernel.Boot.timerEnableHPET'

Registered advanced option 'VMkernel.Boot.timerEnableMPMC'

Registered advanced option 'VMkernel.Boot.timerEnableTSC'

Registered advanced option 'VMkernel.Boot.timerForceTSC'

Registered advanced option 'VMkernel.Boot.tty1Port'

Registered advanced option 'VMkernel.Boot.tty1Term'

Registered advanced option 'VMkernel.Boot.tty2Port'

Registered advanced option 'VMkernel.Boot.tty2Term'

Registered advanced option 'VMkernel.Boot.updateBusIRQ'

Registered advanced option 'VMkernel.Boot.usbBoot'

Registered advanced option 'VMkernel.Boot.useNHCC'

Registered advanced option 'VMkernel.Boot.useNUMAInfo'

Registered advanced option 'VMkernel.Boot.uwSwap'

Registered advanced option 'VMkernel.Boot.vmkKeyboard'

Registered advanced option 'VMkernel.Boot.vmkLoadEntry'

Registered advanced option 'VMkernel.Boot.vmkLoadSeed'

Registered advanced option 'VMkernel.Boot.vmkTerminals'

Registered advanced option 'VMkernel.Boot.xapicAMDDetect'

Registered advanced option 'VMkernel.Boot.xapicForce'

Registered advanced option 'Misc.HostAgentUpdateLevel'

Registered advanced option 'Config.Defaults.security.host.ruissl'

Registered advanced option 'Config.Defaults.cpuidMask.mode.0.eax'

Registered advanced option 'Config.Defaults.cpuidMask.val.0.eax'

Registered advanced option 'Config.Defaults.cpuidMask.mode.0.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.val.0.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.0.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.val.0.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.0.edx'

Registered advanced option 'Config.Defaults.cpuidMask.val.0.edx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.1.eax'

Registered advanced option 'Config.Defaults.cpuidMask.val.1.eax'

Registered advanced option 'Config.Defaults.cpuidMask.mode.1.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.val.1.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.1.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.val.1.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.1.edx'

Registered advanced option 'Config.Defaults.cpuidMask.val.1.edx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000000.eax'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000000.eax'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000000.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000000.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000000.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000000.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000000.edx'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000000.edx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000001.eax'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000001.eax'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000001.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000001.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000001.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000001.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000001.edx'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000001.edx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000008.eax'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000008.eax'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000008.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000008.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000008.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000008.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.80000008.edx'

Registered advanced option 'Config.Defaults.cpuidMask.val.80000008.edx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.8000000A.eax'

Registered advanced option 'Config.Defaults.cpuidMask.val.8000000A.eax'

Registered advanced option 'Config.Defaults.cpuidMask.mode.8000000A.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.val.8000000A.ebx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.8000000A.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.val.8000000A.ecx'

Registered advanced option 'Config.Defaults.cpuidMask.mode.8000000A.edx'

Registered advanced option 'Config.Defaults.cpuidMask.val.8000000A.edx'

Registered 49 options.

Registered advanced option 'Config.GlobalSettings.guest.commands.sharedPolicyRefCount'

Registered 1 options.

Plugin initialized

Trying httpnfcsvc

Plugin 4 statically linked

Trying to load disk lib: diskLibWrapper.so,

Successfully loaded disk lib.

Plugin initialized

Trying internalsvc

Plugin 5 statically linked

Looking up object with name = "ha-view-mgr-impl" failed.

Looking up object with name = "ha-view-mgr-impl" failed.

Looking up object with name = "ha-view-mgr-impl" failed.

Looking up object with name = "ha-view-mgr-impl" failed.

Looking up object with name = "ha-view-mgr-impl" failed.

Looking up object with name = "ha-view-mgr-impl" failed.

Looking up object with name = "ha-view-mgr-impl" failed.

Trying nfcsvc

Plugin 6 statically linked

Breakpoints disabled

Using system libcrypto, version 9080CF

File Services initialized fmgr()

Trying ovfmgrsvc

Plugin 7 statically linked

Plugin initialized

Trying partitionsvc

Plugin 8 statically linked

Initializing partitionsvc plugin

Trying proxysvc

Plugin 9 statically linked

Proxy config filepath is /etc/vmware/hostd/proxy.xml

Another process is listening on the proxy port.

Please make sure other instances of hostd are not running

Did anyone know what's going wrong.

Greets