Hi,
I have begun converting an application that runs on multiple servers from physical to virtual. The application is a Content management system with the clustering implemented inside the app so no special configuration has to be done to the servers.
The first step in the conversion was converting the search servers. These are:
HP DL380 G5 with two dual core processors and 32 GB RAM
HP DL580 G5 with four quad core CPUs and 32GB RAM
Having monitored the CPU utilization and RAM on both servers we concluded that the OS and application was consuming 29-30GB RAM on each server, CPU utilization averaged at 4% with peak utilization of 44% CPU.
The search application consists of 64 java applications that load up 29-30GB of data of a CIFS share on a NetApp NAS. On physical it takes about 15 minutes before Search is up. Once loaded up the handle requests for indexed content. Some content is served from RAM and additional content is brought off the CIFS share.
The VMs were converted to virtual using vmware converter. They are hosted on two HP DL580 G5 with four quad core processors and 100GB RAM each. As they are currently in testing no other VMs contend for resources.
After the initial conversion each vm was given 4 virtual CPUs, 32GB RAM each and powered on. They powered on in a reasonable time however it took Search 1hour and 30 minutes to start. Also the average search times have increased significantly. CPU was constantly at 100% on both VMs. The host never show more than 25% utilization(4 cores out of 16 in use).
I then increased the number of cores to the maximum of 8 on each vm. The Search startup times now went to 1 hour and 10 minutes so still not acceptable and the average search times are still bad. CPU utilization of the VM still hovers in the 80% - 100% utilization. As expected the hosts only experience up to 50% CPU utilization as on 8 cores are in use on each host.
As a total the physical systems do have a total of 20CPU cores available however the 64 Search engines are distributed equally between the 4 core server and the 16 core server yet none ever experience high CPU. In comparison the two VMs have 16 cores between them so at least the one converted from 4 physical to 8 physical should fare better.
I checked HAL and the CPU is set to multiprocessor. I removed all HP software from add/remove programs. This hasn't made a visible dent in CPU utilization.
The hosts do not support hyperthreading so it is not enabled in the VMs. It seems that the biggest consumer of CPU resources are "Hardware Interrupts" with 18-30% utilization followed by "Deferred Procedure Calls" and then each individual search engine consuming about about 0.5% each.
Performance is so poor that we are now considering shelving the virtualization project so any assistance in fixing the performance would be greatly appreciated.
Thank you