The application had been thinapped and the error only appeared when starting the thinapped version.

Using Process Monitor I noticed that the application wrote logfiles to the C:\Temp\HarmonyClient\Log folder (which of course it shouldn’t write there.).

My next step was to trace using the Thinapp Log Monitor but unfortunately the error doesn’t occur when tracing.

This makes me believe it’s a timing issue; upon exit the application is cleaning up memory (objects) but destroys a certain object (eventhandler) while it’s still being accessed.

Upon further inspection of the Process Monitor log I noticed access to a file called HARMONY_Client.exe.9e3c5c50.ini.inuse which was in the ThinApp SandBox (%Local AppData%\ApplicationHistory\HARMONY_Client.exe.9e3c5c50.ini.inuse).

The application creates this file when it’s started and when it exists it copies this to HARMONY_Client.exe.9e3c5c50.ini.

I deleted this file and this made the error go away. Perhaps this fault will surface again in the future, further testing will need to tell us.

Workaround

Delete *.inuse from the SandBox, preferably using a (vbs) script in the thinapp package each time the app is launched.

Solution

Contact Vendor and get them to fix their software and while they’re at it, have them fix access to things such as C:\Temp as well.

Please note that at this time the code is not publicly available so please don’t ask for it.

EDIT 2: I added a video that I received from someone who tried my Excel Sheet with AppSense Application Manager.

EDIT: I wanted to clarify a couple of things regarding this post.

First of all I would like to explain why I wrote this code and why I choose to test it with RES WM.

I had the idea about this approach a long time ago but I never got around to actually do it. The main reason was that I needed to convert Delphi code to VBA and especially converting some Windows headers was a lot of work. Then suddenly I noticed that someone had already converted the headers, so I all I had to do was rewrite the code that used it to VBA.

The choice for RES was made because of two reasons:

1. If you want to beat something, you want to beat the best and I most certainly consider RES WM to be one of the top products.
2. At the time I wrote the POC code I had access to an enviroment with RES in it.

I would like to emphasize that RES contacted me very quickly after publishing this blog. I’ve had contact with RES and they showed a very constructive approach with their primary goal being a fix or guidance for their customers. Hats of to RES taking a constructive approach and I will be working together with RES on this issue.

Finally I would like to state that I didn’t expect this post to draw this much attention, if I did I would have probably taken another approach.

Same demo but now with AppSense:

I received a reference document from a user, a rather complex Excel sheet. When selecting multiple tabs it took almost a minute to generate a print preview in Excel 2007 running on Windows 2003 with XenApp 5.

I was aware of a bug in the Xerox Universal Driver where almost 9.000 files were copied into the user’s profile directory (I wrote about that in an earlier post). But this seemed to be another problem.

I made a trace with Process Monitor while generating the print preview and this made clear that the driver tries to create the key HKLM\Software\Xerox. This is denied because a user has of course no permissions to write in HKLM:

The Registry Summary shows us that the driver tries to access HKLM\Software\Xerox almost 8.000 times before finally giving up:

I changed the permissions on the Xerox key to see what the driver writes to this key. It creates a subkey with the SID of the User under HKLM\Software\Xerox\V5.0\Low.

Then it gives the user special permissions on this key and subkeys so the user is allowed to write there.

Finally it stores a few settings in that key:

This looks like a serious design flaw because not only is a standard user not allowed to create keys under the HKLM hive but it also means that user settings are stored on a single machine and are not roaming with the user.

The good news was that the time to generate the print preview went back to around 4 seconds which is acceptable considering the complexity of the reference Excel sheet.

I tested in total 3 versions of the Xerox Universal Print Driver and they all exhibited this particular issue:

• 5.185.42.0N 2011.03.02
• 5.216.12.0N 2011.06.10
• 5.246.7.0 2011.12.07

When starting the application, the WOW subsystem (NTVDM) crashed with the message: “NTVM encountered a hard error.”:

After spending some time troubleshooting I remembered a similar issue from a few years ago where a DOS application worked fine from the Console but refused to work from an RDP or ICA session.

And indeed the application works perfectly when run from the Console but not from a Console session. I noticed that the application switched to full screen mode after it was launched (even when I set it to Windowed mode) and presumably this is why ntvdm errors: full-screen mode is disallowed for DOS apps in RDP (and ICA) sessions as documented in Q192190.

I looked for a way to force the application to run in windowed mode but I was unable to find such a solution. So I decided to test the application in DOSBox, an x86 PC emulator.

And that worked perfectly, no changes were needed at all to make the application run.

As an added bonus, DOSBox takes care of typical issues with DOS applications running on Citrix XenApp such as keyboard polling and 100% cpu usage.

I was even more impressed that the application runs fine with DOSBox on my Windows 7 64 bit machine!

There is one thing I didn’t like though, DOSBox always shows a Splashscreen that fades in and out:

This is typically something that is not desirable on a XenApp (or RDS) environment because it causes many unnecessary screen updates. This may be a non issue on a fast LAN but on a slower WAN or high latency connection it may matter. Do how do we get rid of it?

There is no commandline argument or config setting that disables the splash so I figured that my only option would be to compile the DOSBox source and leave out the splash screen.

So I downloaded the source files from the sourceforge project page and launched Visual Studio 2010.

The Splashscreen is in sdlmain.cpp but I noticed this comment:

This presented me with a dilemma: I really think the creators deserve their credit but at the same time I want to get rid of the splash.

So I decided to change the code in a way that the Splash screen is shown when run from the Console but not when run from an RDP or ICA session. This change was very easy, I surrounded the Splash screen code with a conditional statement:

In order to compile the code with Visual Studio I followed the Building DOSBox with Visual C 2008 Express article from the Wiki.

Below you will find two downloads:

• A binary package, containing my compiled DOSBox.exe and it’s dependancies.
• A source code package, containing the modified source and the SDL Development Libraries. The modified source is licensed under the GNU GPL license.

If you are going to use DOSBox I highly encourage you to make a donation to Support the DOSBox project.

A few other causes for the “NTVM encountered a hard error” message may be:

• The %TEMP% or %TMP% variable point to a directory that is not in a short (8.3) format. See also CTX110996.
• Error message when you run a 16-bit program in Windows Server 2003: "NTVDM has encountered a hard error” (Hotfix KB937932).
• Check that HKLM\System\CurrentControlSet\Control\WOW\DisallowedPolicyDefault is set to 0
• Check that HKLM\SYSTEM\CurrentControlSet\Control\VirtualDeviceDrivers\VDD doesn’t contain non existing files.

DOSBox 0.74 (23)

DOSBox 0.74 Source Code (8)

On my testmachine it ran nicely but on a customer machine the script failed with the error 2312 “The Citrix online plug-in received a corrupt ICA File. The ICA File has no [ApplicationServer] section“:

I couldn’t find any errors in my script so I fired up Process Monitor and noticed that the Ica Client Object creates a temporary .ica file in the %temp% folder. When it tried to write to this file this fails because access is denied:

I Checked the temporary .ica file and it was empty (0 bytes). Then I used the Stack View option from Process Monitor on the first ACCESS DENIED event:

From the stack we can see that fltMgr.sys is the last to touch the file (stack is from bottom to top). fltMgr is the File System Filter Driver which makes it likely that the Virus Scanner is blocking access. So I checked the Anti Virus log, McAfee VirusScan in my case:

The text is in Dutch, it says: Blocked by the access rule: maximum Anti-Spyware: prevent script execution from the temp folder.

So McAfee considers the .ica a script since it’s created by the process cscript.exe.

Unfortunately the Ica Client Object doesn’t offer a method or property to change the folder where the temporary ica file is created. I decided to have look at Wfica.ocx with Ida Pro and noticed that the GetTempPath and GetTempFilename API’s are used to assemble the filepath.

In the remarks section of the GetTempPath documentation on MSDN states that it looks first to the %TMP% environment variable.

So we can easily workaround this issue by changing the %TMP% variable before we run our script:

I had expected reduced performance (see my earlier post) since I didn’t have the  Battery Backed Write Cache module installed.
I did order them but had to start deployment before they arrived.

I did not however expect such an extreme bad performance. Deployment took ages or sometimes failed completely and when logging in to a VM it responded very sluggish.

Disk Latency

I looked in the vSphere console what the Disk Latency was. Latency under 10ms is usually considered good while a latency between 10 and 20ms is a potential performance problem.

I was shocked to notice that the Disk Latency was much higher with peaks toward 2.000 ms (2 seconds!):

At the same time the transfer rates were very low:

CPU Ready time

A look at the CPU Ready and CPU Wait counters was the final confirmation that storage was the bottleneck.

To view these counters, select the VM in question and go to the Performance Tab and click Advanced:

Then add the counters:

These counters indicate that the CPU is waiting for something (the storage in this case).
You will want these numbers to be as low as possible and they are very high here:

Console Errors

On the VMWare console I noticed the following errors:

• scsi_cmd_alloc returned NULL!
• WARNING: NMP: nmp_DeviceRetryCommand
• WARNING: NMP: nmp_DeviceAttemptFailover

I went back from deploying multiple machines at once to just one but this didn’t improve the performance at all.

Using the error messages above I found this knowledge base article from Hewlett Packard: To Obtain Optimal Disk Subsystem Performance on ProLiant G6 Servers Configured with Smart Array P410i/P410/P411/P412/P212/P712m Controllers Running VMware.

The article states: When the Battery Backed Write Cache is not included in the configuration, even moderate disk I/O can negatively impact server performance.

In my opinion the article actually says: performance without the battery backed cache module is is absolutely horrible and there is no way you can successfully run any Virtual Machine on the hardware without it!

And then, to my great relief,  the upgrade hardware (including the BBWC) was delivered. So I immediately built in the BBWC because I wanted to include the results in this post:

After placing the BBWC module it will not be activated until the battery has been sufficiently charged:

Results

And then the results are amazing:

And finally a successful deployment:

Especially the time needed for restoring the base image went down a lot (this includes not only the actual restore but also sysprep, network config and several reboots).

Related links

• To Obtain Optimal Disk Subsystem Performance on ProLiant G6 Servers Configured with Smart Array P410i/P410/P411/P412/P212/P712m Controllers Running VMware
• Slow virtual machine storage when using HP Smart Array P410 with ESX 3.5 (also on 4.1).
• Slow power on and storage operations with the HP Smart Array P410i controller
• HP 512MB P-Series Battery Backed Write Cache Upgrade

• Printing in general was slow, especially when a user connects to a printer for the first time.
• User Profiles were rapidly growing in size (from the expected 1-2 MB to over 40 MB).
• Logons took much longer then in the testing period (and since we use a Full Screen Desktop the user doesn’t see any progress).
• Performance monitoring showed CPU spikes in Word, Excel and IE processes.

I took a look at the profiles first and noticed that the size growth was due to a Xerox subfolder in %APPDATA%:

An even bigger problem is the huge amount of files in this folder (almost 9000) which of course causes the delay in the logon (while the server is loading the profile).

This is very bad considering that this driver, the Xerox Global Print Driver is advertised by Xerox as the Terminal Server/Citrix ready printer driver!

What makes it even worse is that the driver exhibiting this bug, is still the one offered for download on the Xerox Site.

For the record: I found this bug in version 5.185.19 dated oct. 14, 2010 in the PCL5 and PCL6 versions for both x86 and x64.

The good news is that I found a newer version that was linked on the Xerox forum, version 5.185.32 dated jan. 5, 2011.

I tested this version and the bug was fixed there, I cannot yet tell if the CPU Spikes issue was also fixed.

Here are the links:

• Xerox Global Print Driver x86 PCL5
• Xerox Global Print Driver x64 PCL5
• Xerox Global Print Driver x86 PCL6
• Xerox Global Print Driver x64 PCL6

But after installing this driver we also need to cleanup the (roaming) profiles to get rid of the Xerox folder.

I wrote a very simple PowerShell script for that:

The idea is that half of the servers will reboot in a night and the other half the following night.

The TSSHUTDN tool is handy since it can issue a warning to logged on users, log them out after a certain period and finally issue the reboot.

Since I needed to add a scheduled task to many servers I wanted to do this with a script.

WMI Exposes the Win32_ScheduledJob Class and it’s Create Method.

The parameters, especially StartTime, are constructed very odly and I can never remember them.

So I wrote a very simple wrapper in PowerShell to make it a little easier for the next time:

He received this error message:

Citrix has a KB Article: “How to Allow More Memory for Session Graphics on Windows Server 2003” that explains exactly how we can solve this.

We need to change the MaxLVBMem registry value and we can use the Excel Sheet from the KB Article to calculate the proper value.

Please don’t set this value too high because a higher value means you will restrict other kernel memory pools.

You also need to deny the SYSTEM account the SetValue permission on the HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management key to prevent the Citrix IMA service from overwriting the new value.

So I wrote a small PowerShell script to change the permission and set the value:

Actually the script isn’t specific to Citrix or Terminal Server but on such environments we need to preload all drivers because users do not have the permissions to do that.

I have chosen for PowerShell because you can do it in a one-liner which makes it easy to run this script from my Altiris server on all Citrix Servers.

The idea is that we enumerate all the shared printers on a Printer Server and make a connection to each printer. This will make sure that the driver is installed if it wasn’t already present.

The script could even be scheduled to enforce that newly added printer drivers are added to each Citrix Server.

To enumerate all printers we can use the Win32_Printer WMI class like this:

It’s possible that some printers are not shared so we are going to filter that out using the -filter parameter:

Our next step is to make a printer connection and MSDN shows that the win32_printer class has an AddPrinterConnection method.

So I first tried:

I am not sure of the exact reason but obviously there is no access to the AddPrinterConnection method.

This works though:

Now we can use the For-Each Object to make a connection to each printer using the __SERVER property for the servername and the ShareName property for the Printername:

Once again I met a PowerShell oddity: I couldn’t use AddPrinterConnection(“\\$_.__SERVER\$_.ShareName”) so I used:

And finally we put the whole thing in a one liner, replacing double quote (“) with (‘) and using the gwmi alias to make the line shorter:

EDIT: In order to load new driver we need to enable the SeLoadDriverPrivilege. I have corrected the code above,  see Enabling Privileges for WMI in PowerShell for an explanation.