AppV5 – Package fails to load with error 0x79100E100-0xC (Starring Procmon)

/ /
in Blog

We were having an issue with a AppV5 package loading and we received the following error message:
Event ID 1008
Package {b8b01729-ed31-4d77-a859-dbd8b82a3372} version {e1d21ac7-84f0-4ab7-998f-e3258be91298} failed configuration in folder ‘D:AppVDataPackageInstallationRootB8B01729-ED31-4D77-A859-DBD8B82A3372E1D21AC7-84F0-4AB7-998F-E3258BE91298’ with error 0x79100E10-0xC.

This message is preceeded by this message:
Event ID 4009
machine script for event AddPackage with command line: ‘cmd.exe’ exited with failure error code: Incorrect function.. Because Rollback is set to true in the script definition, the current AppV Client operation was rolled back.

I started investigating.  The first thing I did was open a powershell window and tried to load the package via the command line:

I then looked at our DynamicDeployment XML file and examined the script it is trying to launch:

The script it’s trying to launch looks like this:

Since this is a Machine Script, I started a process monitor capture, executed my command, stopped the capture then used the “Process Tree”

and clicked on AppVClient.exe and clicked “Include Subtree”.

and used the “show process and thread activity”.  I filtered on ‘Detail’ ‘Begins with’ and set it for both Parent and Exit so I can look at the process path and exit codes:


The ‘exit code’ is ‘1’ (Exit Status: 1) which means an error occurred that caused the script to fail.  So now we dive into that script and see why it’s failing:

From looking at the script, it’s trying to create a new ‘mklink’ path.  If I try and run this command manually, I get the following:

So this is where the errorlevel (also exit code) is being set.  The last error level is 1 which becomes the exit code once the script runs ‘EXIT.

So, there are two methods I can think of to solve this problem.

1) I can set EXIT /B 0 to always set EXIT to report and error code of ‘0’.
2) Check to see if the path already exists and then EXIT

I chose to modify the script to exit if the path exists.  I changed it like so:

I cleared procmon, set it to trace again and attempted to run the add-appvclientpackage command again.

I selected ‘AppVClient.exe’ and clicked ‘Include Subtree’

This time we can see that the ‘AHS-ATOP.cmd’ script has an ‘Exit Status: 0’ which means it completed successfully.  But, the next script, ‘AHS-SoftWorksGroup-ScreenTestIII.cmd’ with the parameter ‘INSTALL’ fails with ‘Exit Status: 1’.  Again, we look into the script…

We can see it has the same flaw.  I then modified the script to add the same IF EXISTS check.  I then cleared procmon and reattempted to add the package.


Hurray!  The package loaded and the scripts ran correctly.

An alternative to all of this is I could have changed the ‘rollback’ to ‘false’ in the DeploymentConfig.xml file, but I would rather the package *not* load if the ‘INSTALL’ script fails for whatever reason.  This ensures an error is generated that needs to be dealt with rather than potentially having a half-working package.  These INSTALL scripts were simple enough that a simple directory check would suffice to ensure it’s working and that’s what I’ve done.

Read More

Help! My Citrix application is running slow (Meditech, Citrix, Imprivata)

/ / /

We have an application (Meditech) that users have reported as performing poorly in Citrix.  They were able to confirm the poor performance by comparing it to a desktop that had the same software installed.  I was brought on to try and understand these performance differences and why Citrix would be performing so much worse.  To measure the performance, the user took me to a screen where they held down a key on the keyboard to increment a counter in the software.  When holding the key down on the desktop the counter was quick and incremented at a steady pace, on Citrix it seemed OK at first but then slowed as time went on.   To illustrate these performance differences, I made a video:

So we definitely have a problem.  My first steps on troubleshooting this problem was to compare the differences between the desktop and the VM.  The VM had server processors but they were older, and at less GHz at 2.6GHz for the server.  The desktop had 3.4GHz processors with a newer generation.  If the processing is NOT latent sensitive then the faster processor’s can make a difference, and the CPUMark had them at 1500 and 2100 (~40% difference).  At first glance, this seems like it could be the difference in our timings but it’s still too drastic at ~5s vs ~20s, a 400% difference.  To try and narrow the difference I took the application to a completely bare server with no software on it what-so-ever and reran the test.  It completed in about ~5-6s.  The processor it ran on was comparable to the original server processor but ran at 2.7GHz instead.  The processor was not running at nearly enough speed to make up the difference, something else must be consuming those cycles.

At this point I procmon’ed the benchmark and came up with the following:

(PID 6660 is the process that the benchmark runs against)

Very obviously, the pattern of each key stroke on the Citrix server is present with the initial pattern highlighted:

About 400ms from each key stroke to when the next one is registered.  So what is delaying the 400ms?  From my experience, unexplained delays are CPU related.  I then looked at the process activity summary to see if I can find the bottleneck:

Again, very obviously we are seeing the CPU ramp up, and it can also explain the faster, initial iterations of the GUI as the CPU ‘ramps’ up and doesn’t spike to the it’s maximum.  None of the other graphs show any activity at the time of the benchmark so the suspect is highly on the CPU.  When hovering over the graph we see the CPU percentage.

This is a 4 core box, so ~25% equals one full core for a single threaded application.  Again, this points to the application being bottlenecked by the processor, but again, the difference is too large to consider just the CPU at this point.  We need to find what part of the process is consuming these resources.  Fortunately, ProcExp (process explorer) can help us determine what is going on within a process.

I started a new run and got properties of the process:

MGUI.DLL is consuming all of the CPU.  That is a DLL utilized by the application, clicking on ‘Stack’ gives us the hierarchy of commands being utilized by that thread.

From this, I can understand that ntoskrnl.exe, ntdll.dll are native Microsoft Windows functions, MGUI.DLL is utilized by Meditech, but what is ISXHook.dll?

Doing a search within the process shows that it’s utilized by Imprivata, a single sign-on solution we utilize to try and increase user efficiency.  It works by ‘screen scraping’ to determine fields that it needs to populate with user credentials to try and speed up user logins.  Logically, this sounds like it could be causing the delay by screen scraping every time a key is stroked or a change is registered.  To confirm this, we need to remove Imprivata from the application.  Fortunately, it’s hooked in by services that can easily be terminated.

I’m going to terminate everything that says ‘Imprivata Inc.’

With the processes terminated I reran my test.  Using Process Explorer and getting properties on the process, immediately CPU usage went from 25% down to 15% at a peak:

And getting Stack information showed a much cleaner stack:

In conclusion, I may need to investigate into Imprivata to determine if I can reduce its polling rate or find some way of allowing it to ‘stop’ polling the CSMagic process *after* the accelerated login.  Its current settings (which I’m not familiar with, sadly) is not acceptable and causes a significant slow down.  Fortunately, the root cause has been determined and we can work towards a full resolution.

Read More

AppV5 – Sequencing an application that has Com+ Applications

/ / /

AppV has a few weaknesses, one of them is Com+ Objects.  It doesn’t like Com+ Objects and sequencing an application with them generally results in this error message:

The Resolution seem easy enough:
1) Try extracting and installing the COM+ component natively on both sequencing station and the client
2) Thoroughly test your virtual application package to verify functionality.
But how do you extract the COM+ components?
1) Open ‘Component Services’
2) Expand until you find your Com+ Applications, right-click and select ‘Export’
3) Click ‘Next’
4) Enter a path to save your MSI then click ‘Next’
5) Click ‘Finish’
Since this application had two Com+ Applications I exported my second one as well:
To ensure they work, I took them to our XenApp Test server and installed them.
Both *appeared* to have installed correctly.  But this application requires the Com+ Application to run under a service account.  Checking the original install:
I can see under the ‘Identity’ tab that it is configured for ‘This user:’.  I then checked the MSI install on the XenApp Test server and looked at the Identity tab:


It appears it defaults to ‘System Account’ even though I selected ‘Export user identities with roles’.  To change the user account silently, I used this script supplied by the vendor (as a encrypted vbe):
This script changes the user account on the Com+ applications to use the proper account.  When browsing the properties of the components of the SFSQL object I saw it referenced the dll to it:



Fortunately, it seems these files are captured by the MSI export of the Com+ Application.
To test everything now, I must now create my AppV package.  Before doing the install I installed my prerequisites (the Com+ Applications) into the Sequencer so they are not captured by the package.
I ran this script to do my prereq install:
Now I startup the sequencer and do my sequencing first steps.
I selected a batch file I created for the install (per the vendor’s initial instructions):
Once the install completed I saved my package and took it to my XenApp Test server and published the application and ran it.
No, this is not correct
For some reason my application was failing.  I double checked the COM+ Applications are installed and appear to be working, so I opened Procmon.exe to see if I can find where it’s going wrong:
I’ve used procmon enough now to practically pull the old’ Matrix, “blonde, brunette, red head“.  Working from bottom to top, the message box is generated from the ‘Text’ (LanguagePack) back to the KernelBase.dll.  So we can exclude that has the cause, this is telling us *of* the issue.  So if we look at the lines preceding it, one of them must be causing our message.
Looking at the first keys, the OLE keys, I can compare them to my sequencer and they match exactly, so the likelihood of them causing our issues is none.
So we go back a little further to the AppID key and jumping to the *SUCCESS* location to look at it’s value…
I am on a server called WSCTXAPP301T but the registry key for this value is pointing to the system I did my sequencing on.  I changed the value to the actual server name and did a registry search for “WSAPVSEQ07” to see if it appeared anywhere else.  It did in one other location so I modified the value there as well to point to the local server:
And tested my application again:
Success!  The login screen.  This is as far as I can take this app until I get credentials or a user to login and try testing for me.  So, for now, I can create a preflight-script to setup some prerequisites when the application is published.  I modified my preflight script to include the two registry values being updated:

Then I updated the Deployment_Config.xml file for this application:

Lastly, I logged onto a new, fresh, system and manually tested my XML file by publishing from the command line:

Using Procmon to monitor to ensure the prerequisites get installed:

“Exit Status: 0” –> Usually a good thing 🙂

Testing the application:

Looks good

Then testing removal and monitoring with Procmon:

Com+ Application are removed cleanly 🙂

So everything looks good and I can now add this application to our AppV Management Server and deploy it to the machines that need it.

Read More

Citrix Receiver 4+ rants and “Your apps are not available at this time. Please try again in a few minutes or contact your help desk with this information”

/ / /

The environment I’m working in is a mix of XenApp 6.5, 5.0 and Presentation Server 4.5.  The 6.5 and 5.0 farm also have a nearly identical test farm.  We’ve been migrating the applications off the Presentation Server farms and are moving them to XenApp 6.5.  At this point in the migration we have around 5-10 applications left on 4.5 to move, with around 400-450 on the 6.5 farms and probably around 20-30 or so on the 5.0 farms.  These farms utilize a Citrix Webinterface 5.4.2 frontend for web interface and PNA.

We have standardized the environment on mostly Citrix Receiver 3.3 and some 3.4.  Time has marched and we’ve been tasked with getting Receiver 4+ working the Windows 7/8/10 rollout.  We were not able to do so with the earlier versions of Receiver 4 because things like sort icons into custom folders on the desktop and Start Menu.  This feature came in around 4.2.  In our environment memberships to applications are granted through group membership and Citrix PNA allowed the user to ‘roam’ from computer to computer only displaying the applications they have access to, as opposed to a bunch of applications they do not have access with the onus on them to pick and choose the correct applications.

So we started work on planning this migration and have started with the latest greatest (as of today) Citrix Receiver 4.3.  We have been able to come close to simulating all the features of the Enterprise editions of 3.3 and 3.4.  Namely:

Citrix Receiver automatically connects and populates a folder (MyApps) in the Start Menu and on the desktop.
No self-service.  Applications are automatically presented to you and defined by Group Membership.
Single sign-on.  Receiver will take your Windows logon credentials to use for authentication.

We do have some outstanding items.  We’ve set the client to have all applications as ‘MANDATORY’ which does populate the applications in the MyApps folders; but applications marked as ‘Create shortcut on the desktop’ in the applications properties in AppCenter are not created.

Anyways, onto the problem.  Now that we have Receiver 4.3 setup, SSON working, PNA working, we logged onto our system and watched the applications populate.


Really Slowly.

Eventually a dialog popped up.

Then another, and another, and another.

And these dialogs are completely custom!  They are NOT native Windows dialogs!

So if you have multiple ones of them, sometimes clicking the X (close button) or OK doesn’t work because the dialog appears ‘modal’ and you need to click the button on the ‘active’ window.  But you generally don’t know which one that is so you have to go through and select each dialog from the task bar and try clicking ok until you magically get lucky and select the one that has priority.  Then you do it all over again as the next primary window *may not be the one on top*.

1 minute 24 seconds to populate 470 applications

Just for giggles, how fast does Receiver 3.3 populate the same list?

8 seconds.  And all the icons show up.

Alright, so you’ve passed that point and are now looking at your applications.  But they are missing icons!

But not all applications are missing their icons…  Only some.

So let’s find out what’s consuming all this time and maybe, just maybe, we’ll solve our “Your apps are not available” error message.

First thing we need to do is enable Citrix Receiver Logging:

Next is to exit and restart receiver and logs will start to generate.  They are located here:

The most important log tends to be the ‘SelfService.txt’ log.  If you search that log for the “Your apps are not available” error message it pops up in locations like this:

So this dialog popped up for an application called ‘BMTServe’.  And what does BMTServe look like?

Generic icon!

But BMTServe was not the only application that encountered this dialog.  From my video it popped up numerous times.  Searching the SelfService.txt file for ‘Your apps’ and looking for the application it references points to an application with a blank icon 100% of the time.  Not every application that produces a blank icon causes this prompt, as we literally have ~250 applications with blank icons and the dialog pops up anywhere from 0 to 10 times.  Sometimes it pops up 2 times, sometimes none, sometimes 10 times.

So, why are these icons blank?

Citrix Receiver 4.3 seems to only prefer 32bit icons or icons of a particular size.  I haven’t confirmed what exactly yet, but I do know that 8bit 32×32 icons don’t seems to get ‘translated’.  The Citrix logs all but confirm this as well.

I confirmed with Citrix that icons are required to be 32bit and the order they are checked is 48×48, 32×32 then 16×16.

This is how Receiver processes icons that are formatted correctly:

The icons were processed instantly. 00:00:00. But if they are formatted in a way that Receiver decides it needs to ‘reformat’ them:

This call to get an icon took 11.6 seconds!!! If it doesn’t get the icon formatted in the way it wants, it appears the SelfService.exe setups a queue of icons that it needs and ‘re-requests’ them from the server. Could it be that Receiver is submitting too many queries? The error mentions to check the authmansvr.txt log file. This log file shows the following:

The error appears to start at “CWindowsReceiver::CallARGetConnectedVpnGateway” When this call is successful it returns:

So, I guess it’s possible that trying to re-pull the icon data is causing authmansvr.exe to crash…?  Another crazy thing is I was attempting to automate this process of terminating Receiver and relaunching it to see if I could get a gauge on the frequency of this occurrence and this is what I saw:

Ok, I thought, not so bad.  Just two messages the first couple launches?  It shouldn’t be too much of an issue…  But then I looked at my application folder:

Left is when I get all my apps (and usually the message box) the left is all those ‘successes’

It appears Receiver removed all applications producing that dialog box.  When I was terminating and relaunching receiver it was ONLY populating 195 applications as opposed to 493 it was supposed to. No wonder I wasn’t getting any messages!  On a hunch, I looked at each of the 195 applications it kept and they all had good icons.  I then took a random sampling of about 30 of the 300 or so applications that it did not keep and none of them had proper icons, all blank.  So another bullet towards icons causing my issue.

Read More

PVS Target Device Update Script — Supplemental File, Update_Tools.ps1

/ / /

This script is used to keep ProcMon.exe, ProcExp.exe and Wireshark up to date and available as tools we can use to troubleshoot our vDisks.  We store them within a folder on the C:Swinst.


Read More

IMA Service Fails with the Following Events: 3989, 3634, 3614

/ / /

We have run into an issue where we have Provisioning Service 6.1 and XenApp 6.5 working together. After we update the vDisk (say, for Windows Update) we run through a script that does things like the “XenApp Prep” to allow the XenApp 6.5 ready for imaging. It appears that there is a bug in the XenApp Prep that sometimes causes it to not fully get XenApp 6.5 ready for rejoining the farm. The initial symptoms I found were:

Event ID 4003
“The Citrix Independent Management Architecture service is exiting. The XenApp Server Configuration tool has not been run on this server.”

I found this CTX article about it, but nothing of it was applicable.

I did procmon traces and I found the following registry keys were missing on the bad system:

A broken system missing the Status Registry key

A working system with the Status key. Note Joined is “0”

After adding the Status Registry key:

I tried restarting the service and the progress bar got further but then still quit. Procmon showed me this:

That is ACCESS DENIED when trying to see that registry key. It turns out that the IMAService does not have appropriate permissions to read this key. The Magic Permissions on a working box and what you need to set here looks like this:

Notice that none of the permissions are inherited and “NETWORK SERVICE” is added with full control to this key. Now when we try and start the Citrix Independent Management Architecture service we get the following errors:

To correct these errors the local host cache needs to be rebuilt. To fix that we need to run:
Dsmaint recreatelhc
Dsmaint recreaterade

After doing that, we can start the IMA Service and MFCOM. If instead of IMA Service starting you get the following error message:

Ensure the following registry is populated:
Read More

SCVMM: Install VM components Failed

/ / /

I’m attempting to deploy a virtual machine from a template but I get this error: Install VM components: Failed.



SO…  Long story short; if you are encountering this error, I would suggest booting your VHD file in a VM and re-sysprep /generalize it.  If you’ve maxed out on sysprep’s I have a post earlier in my blog on how to get around the 3-times limit and rerun sysprep.  Alternatively, you can try what I did, but I can’t gaurantee your success and replace the BCD file in your Library VHD with a BCD you *know* has been sysprepp’ed and try redeploying it.

I’ve ensured that HTTP and HTTPS is not blocked (firewall is disabled) and the agent on my SCVMM machine is installed and running.  So this error message is somewhat useless.

So I took my two boxes, 2012-SCVMM (the SCVMM server) and S5000VXN-SERVER (the Hyper-V host) and procmon’ed them while it was attempting to “Install VM Components” to try and understand what it’s trying to do.

After reinitating the task, we can see that the vmmAgent.exe on the Hyper-V host accesses the file about 2 seconds after I submit the command to retry the job.

A few seconds after this, it appears to “CreateFile” in the WindowsTemp directory; but this is not actually correct.

What it is really doing is *mounting* the VHD into a folder in the WindowsTemp directory.  You can actually watch this in action if you view Disk Management on the Hyper-V host while you execute the task.

So now that we know it’s mounting the file as a volume this helps us narrow down on what Hyper-V is attempting to do…  And I suspect what it is attempting to do is “offline servicing” of the attached vdisk/vhd.

After attaching the vdisk the next thing it does it query the BCD file on the system.  Maybe it needs to be in a certain mode to operate?  I’m not sure…

Continuing on we can see that events are written to the Microsoft-Windows-FileShareShadowCopyProvider Operational.evtx, System.evtx, and Microsoft-Windows-Bits-CompactServer Operational.evtx event logs.  Examining each log at the time stamp showed the FileShareShadowCopyProvider and System log were just noticed of volume shadow copy starting, but the BITS event log was interesting.



It showed that it was doing something with the BCD file.  The Hyper-V host was *serving* it out.  I suspect it was serving it to the SCVMM.
Looking back at the SCVMM server we see that was executing some WinRM commands.  Sadly, we do not know what commands it was trying to send.

If I mount the vhd file and check out the BCD file I can see that it appears to be corrupted in that it doesn’t know what the proper boot device should be.


It’s not actually corrupted though; the reason why the devices are unknown is because bcdedit isn’t finding the disk signature of the volume I mounted.  But it has the disk signature because I can boot with it without issue.

Continuing on…

In order to try and find out what commands WSMAN was sending I enabled debugview on the SCVMM server:

I then reproduced the error by rerunning the Create Virtual Machine job.

DebugView gave me more information to narrow down what was happening.  It appears that the process is failing with:

Doing some googling on this took me to a Korean Microsoft page where the following was stated:

Thinking about it, I do not think my VHD was SysPrep’ed.  I find it interesting that sysprep appears to do something to the BCD file.  To find out what sysprep does to the BCD file I booted up my 2012 VHD into Windows and ran sysprep, generalize and shutdown.

And……….?  Lets go to DebugView:


Voila!  It appears much better than before.  It found the drive correctly and checked the BCD file and found it is the “Generalize” state.  The *actual* image I originally made was NOT sysprep’ed and all I did was replace the BCD file, but it allowed it to continue beyond and the machine actually completed the imaging process properly.  It joined the domain and whatever else the answer file was I gave it in SCVMM.

It appears I was correct in my earlier assumption about what SCVMM is doing.  It goes and grabs the BCD file and transfers it from the target machine to itself, “fixes it up” (not sure what it’s doing at this stage precisely), then sends it back for injection.  Certainly a bit of a complicated process with a fair bit that can go wrong, but shame on Microsoft for having such a poor error message.  I suspect it wouldn’t have required much effort to push out a real message; something to the effect of, “The BCD of this vDisk does not appear to have been through the sysprep /generalize process.  Please rerun sysprep against the image and try again”.

SO…  Long story short; if you are encountering this error, I would suggest booting your VHD file in a VM and re-sysprep /generalize it.  If you’ve maxed out on sysprep’s I have a post earlier in my blog on how to get around the 3-times limit and rerun sysprep.  Alternatively, you can try what I did, but I can’t gaurantee your success and replace the BCD file in your Library VHD with a BCD you *know* has been sysprepp’ed and try redeploying it.

Read More

Issue with WSH (Scripting.FileSystemObject 800A01AD)

/ / /

I recently had a Windows 2008 Server that was unable to execute a VBS script that works with other servers and other combinations of desktops. I decided to break out Process Monitor and try and see if I can figure out what’s going on…

To simplify this process, I found this vbs script that trys to utilize the Scripting.FileSystemObject in a script:


I ran that script on the affected server and, after clicking OK on the WSH Version dialog, I got this message:

I broke out Process Monitor and monitored on the File System. It sounds like it should be a file system error so we’ll scope that out first. I filtered for everything but wscript.exe (I executed all my command lines as wscript.exe test.vbs) and nothing appeared. So wscript.exe wasn’t even getting to the file system. So I enabled registry filtering and filtered for wscript.exe:

And I reran the script and got this result:

From here I went to another Windows 2008 server and added the missing registry keys (NAME NOT FOUND) and repeated the process again, finding more keys until all were added to the non-functioning server.

I ended up adding the following registry keys:


For some reason, it is launching the Wscript.exe in a 32bit process (as evidenced by WOW6432Node key). On the working 64bit server I have it is running as a 64bit process.

After entering those registry keys, here is my new result.

Success! Hopefully, if you encounter the same issue, you are not missing any more, or too many more, registry keys. I wonder why they disappeared, but I don’t have a way to trace that unfortunately.

Read More

invalid pxe server list format – Altiris

/ / /

I restarted our Altiris server and our PXE services wouldn’t come up. Trying to start them resulted in:

File not found
I then checked the path listed in the service and found that, indeed, our PXE files where not in the location that the service was trying to start them in:
F:\Program Files\Altiris\eXpress\Deployment Server\PXE
The missing files were:
They were located here:
F:\Program Files\Altiris\eXpress\Deployment Server\PXE\MasterImages\UpSrv\51
I don’t know why it wasn’t looking for them in the longer path, but I copied those files to the directory it wanted (PXE)
I then attempted to start the services and they all started correctly.
Then I attempted to PXE boot one of my VM’s. This failed with an error stating:
“invalid pxe server list format”
Attempting to troubleshoot this, I used procmon and saw that it was downloading bstrap.0 successfully then generating the error. I enabled logging for the PXE Server in Altiris and set the logging level for “Errors”. I then restarted all the Altiris services. When I restarted the PXE Server service, I got this error message:
E [11:32:26 05/03] (3480): Enter: SetupDHCP(…)
E [11:32:26 05/03] (3480): SetupDHCP: Auto Detect, configure option 60.
(3480)Failed to load Dll Library.
(3480)Failed to load Dll Library.
(3480)Failed to load Dll Library.
(3480)Failed to load Dll Library.
I then fired up Process Monitor and did a file trace while restarting PXE Server. It informed me it could not find the following files:
I then copied those missing DLL’s from the F:\Program Files\Altiris\eXpress\Deployment Server\PXE\MasterImages\UpSrv\51 directory to the F:\Program Files\Altiris\eXpress\Deployment Server\PXE directory and restarted the PXE Server Service.
I then attempted to PXE boot my VM and lo and behold, it worked again.
Read More