I was deploying virtualized Citrix XenApp Servers on HP BL460c G6 servers and somehow the storage (direct attached) responded very slowly.

I had expected reduced performance (see my earlier post) since I didn’t have the  Battery Backed Write Cache module installed.
I did order them but had to start deployment before they arrived.

I did not however expect such an extreme bad performance. Deployment took ages or sometimes failed completely and when logging in to a VM it responded very sluggish.

Disk Latency

I looked in the vSphere console what the Disk Latency was. Latency under 10ms is usually considered good while a latency between 10 and 20ms is a potential performance problem.

I was shocked to notice that the Disk Latency was much higher with peaks toward 2.000 ms (2 seconds!):

DiskLatency

 

At the same time the transfer rates were very low:

DiskKbps

 

CPU Ready time

A look at the CPU Ready and CPU Wait counters was the final confirmation that storage was the bottleneck.

To view these counters, select the VM in question and go to the Performance Tab and click Advanced:

image

Then add the counters:

SNAGHTML15276d9f

 

These counters indicate that the CPU is waiting for something (the storage in this case).
You will want these numbers to be as low as possible and they are very high here:

image


Console Errors

On the VMWare console I noticed the following errors:

  • scsi_cmd_alloc returned NULL!
  • WARNING: NMP: nmp_DeviceRetryCommand
  • WARNING: NMP: nmp_DeviceAttemptFailover

image

I went back from deploying multiple machines at once to just one but this didn’t improve the performance at all.

Using the error messages above I found this knowledge base article from Hewlett Packard: To Obtain Optimal Disk Subsystem Performance on ProLiant G6 Servers Configured with Smart Array P410i/P410/P411/P412/P212/P712m Controllers Running VMware.

The article states: When the Battery Backed Write Cache is not included in the configuration, even moderate disk I/O can negatively impact server performance.

In my opinion the article actually says: performance without the battery backed cache module is is absolutely horrible and there is no way you can successfully run any Virtual Machine on the hardware without it!

And then, to my great relief,  the upgrade hardware (including the BBWC) was delivered. So I immediately built in the BBWC because I wanted to include the results in this post:

image

After placing the BBWC module it will not be activated until the battery has been sufficiently charged:

image


Results

And then the results are amazing:

image

And finally a successful deployment:

image

Especially the time needed for restoring the base image went down a lot (this includes not only the actual restore but also sysprep, network config and several reboots).

Related links