AppV

APPV5 – Sequencing FolioView ICD/CCI with AppV5. (Issues with VFS not redirecting)

2015-03-24
/ / /

When sequencing FolioView 2012 or 2015 with AppV5 SP3, launching the application after sequencing generates an error:

—————————
Folio Views
—————————
No object handler is registered for this object type.
—————————
OK
—————————

This error seems to appear because VFS either isn’t fully loaded by the time Views.exe is loaded so the program/dlls actually traverse down a non-virtual path, or they somehow break the AppV bubble and don’t find whatever it is they are looking for.  To correct this issue, you need to create a symbolic link to the VFS’d program directory from where the actual directory is supposed to reside.  For instance, I sequenced FolioView to:
C:\Program Files (x86)\CIHI
To produce the error, I launch my command like so (this is what AppV5 automatically puts into the shortcut):
%ALLUSERSPROFILE%\Microsoft\AppV\Client\Integration\C05CC38A-D146-442B-B97F-89E8867DF0CE\Root\VFS\ProgramFilesX86\CIHI\CIHI_PUB_2015\Cihi32\Views.exe -L1033 -cSoftware\CIHI\2015\ICD10\English -i”\\fileserver\CTX_APPS\FolioView2015\cci_2015_eng.sdw”
I then get the error message.
If I mklink /d “C:\Program Files (x86)\CIHI” D:\AppVData\PackageInstallationRoot\C05CC38A-D146-442B-B97F-89E8867DF0CE\0CA628F4-AC24-4981-9B52-DA
B4445B8ECF\Root\VFS\ProgramFilesX86\CIHI
Then launch the application via the ‘native’ path:
“C:\Program Files (x86)\CIHI\CIHI_PUB_2015\Cihi32\Views.exe”  -L1033 -cSoftware\CIHI\2015\ICD10\English -i”\\q9-v-citrix-lif1.healthy.bewell.ca\CTX_APPS\FolioView2015\cci_2015_eng.sdw”
It works without any error messages and without issue.
Here’s a short clip of the error and the ‘fix’:
I did a stack trace of the error message and it comes up like so:
There are several calls that appear to look for files (nfomgr4, fcctrl4, mpr, davInt) that maybe generating the error message, but, I am not skilled enough at WinDBG at this time to be able to dig deeper.  Either way, there appears to be a limitation within AppV5 that prevents this program from operating correctly.  The same program works in AppV 4.6 without issue though, so it appears AppV5 sill has some work to go to get full compatibility with PITA applications.
Here’s the application working fine in AppV 4.6.  The application was installed identically for both AppV versions.
Read More

AppV5 – Data Precache script (used for XenApp)

2015-03-20
/ / /

This is the AppV5 Data Precache script we use to load our AppV5 packages on our Citrix XenApp servers.  Because we use PVS and store the AppV package installation root folder on a persistent disk, we sometimes encounter issues that require us to ‘clean up’ the package installation root folder before the AppV5 packages will load.  What this script attempts to do is setup AppV5 as if it’s a new install, grab any packages from the publishing server, then, if sharedcontentstore is disabled, fully mount/load the packages.

AppV5DataPrecache.cmd

 

Read More

AppV5 – Package failed configuration in folder with error 0xA040132A-0x3

2015-03-04
/ / /

When publishing AppV5 applications on a PVS server we sometimes encounter an issue where packages do not load.  There are usually events logged to event viewer with the following:

Package {5075e8a4-4335-4101-991e-be88f5862575} version {940e4d49-af37-428c-a129-3bd37e3e4539} failed configuration in folder ‘D:\AppVData\PackageInstallationRoot\5075E8A4-4335-4101-991E-BE88F5862575\940E4D49-AF37-428C-A129-3BD37E3E4539’ with error 0xA040132A-0x3. with error 0xA040132A-0x3.

With another event that follows:

Part or all packages publish failed.
published: 14
failed: 10
Please check the error events of ‘Configure/Publish Package’ before this message for the details of the failure.

This issue is typically caused by two factors: folders not existing in the “PackageInstallationRoot”, in my example that is D:\AppVData\PackageInstallationRoot.  You can find this value in the registry here:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AppV\Client\Streaming /v PackageInstallationRoot /d D:\AppVData\PackageInstallationRoot

and registry entries existing here:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AppV\Client\Streaming\Packages

 

There are more values in the registry then the folder above.

What you will find is a mismatch between the number of keys in Packages and the number of folders in your PackageInstallationRoot path.  For us, our error was we had 14 folders existing in that path, but 10 of them were missing.  But we had all 24 values existing in the registry.

To fix this error I created a script to create the missing folders that were found in the registry and doing a get-appvpublishingserver | sync-appvpublishingserver

And then all applications were able to be published without issue.

Read More

APPV5 – Virtualization Template

2015-02-20
/ / /

Use if you want, or not.  This virtualization template is to be applied against the sequencer.  I’ve found it removes a lot of useless captured information that can get caught in a sequence.

 

Read More

How important is it to launch your application in a AppV package while sequencing?

2015-01-22
/ / /

For a bit I was of the mindset that sequencing a AppV package should be as clean as humanely possible.  This would include finding all configuration tweaks to files/registry keys ahead of time and implementing them so any registry keys generated would be unique to the user.  I’ve seen some applications generate a unique GUID that vendor’s would use to lock it so that the application was tied to one machine.  Since this key would be generated in HKLM so all users would be able to see the key, it prevented new launches.

But if you didn’t launch the application while sequencing, the key wouldn’t get generated until the user launched it in their bubble.  This effectively allowed multiple users to use the same application on one server.  With this new information in mind, and a new outlook on keeping the AppV registry hive to a bare minimum; forward I strode.  And hard into a wall I ran.

What I eventually ended up finding was applications that error’ed on first launch:

But would launch just fine the second time:

So what would cause it to fail the first time?  I imagine for most cases the error message is caused by missing file(s) or registry key(s) or values.  So how do you find these newly generated profile?  Well, the nice thing about AppV is it stores all these things in two places.  Your registry hive or your user profile.

Before launching the application I did a “dir /s /b C:UsersAdtest91 >>Clean.txt” and saved that to a text file.  I then launched the program twice and ran this command “dir /s /b C:UsersAdtest91 >> Working.txt”  I then compared the files and found the following new paths generated:

C:\Users\adtest91\AppData\Local\Microsoft\AppVClient\VFS\ADB25534-3FE9-44BD-9FC8-D5AAD8C0E728
C:\Users\adtest91\DesktopFax

To see if these two paths caused my issue I deleted them and relaunched the application.  The application launched just fine.  I then backed up those two folders.  To rule out the file system with some more finality, I deleted my user profile then copied my two backup folders to their recorded paths and tried launching the program and got the error message again.  With that I felt confident I could rule out files/folders as the cause.

The beautiful thing about AppV registry changes is they are recorded to your user profile.  This is stored here:
HKEY_USERS\!GUID!_Classes\AppV\Client\Packages

Export this key prior to launching your application, launch your application, and export the key again and do a difference.  Any created or modified registry keys will reside in this location for you to examine.

For this issue, this was the key that was generated:

[HKEY_USERS\S-1-5-21-38857442-2693285798-3636612711-15053136_Classes\AppV\Client\Packages\ADB25534-3FE9-44BD-9FC8-D5AAD8C0E728\REGISTRY\MACHINE\SOFTWARE\Classes\TypeLib\{3B7C8863-D78F-101B-B9B5-04021C009402}\1.2\0\win32]
@=”C:\\Windows\\system32\\Richtx32.ocx”

Deciphering the key results in the following missing value:
[HKLM\SOFTWARE\Classes\TypeLib\{3B7C8863-D78F-101B-B9B5-04021C009402}\1.2\0\win32]

Looking locally on the server, this is what I saw in the registry:

There was nothing in the Data field!  AppV5 ‘integrates’ the Classes key in your AppV package when you publish it.  I resequenced and launched the application after the install and checked the key again:

Surprise, surprise.  And now the application launches without issue.  So it appears that some application installers don’t completely register all files (OCX files are two instances of this issue happening that I noticed) until the application is launched.  So now, our policy will always to, at a minimum, launch the application while sequencing.

Read More

AppV 5 – Measuring RegistryStaging load times

2014-12-14
/ / /

Per my previous posting, I have an issue where the AppVClient.exe consumes significant CPU resources upon application launch.  From a Microsoft forum where another member did some further investigation, he discovered that the slowness and delayed launch times are related to registry staging.  To confirm and measure the impact of registry staging I wrote a script that measures the length of time it takes to finish registry staging for all AppV5 applications on your computer/server.

What this script does is iterate through all your AppV5 applications and then loads a cmd.exe with the AppV environment.  It then checks for the RegistryStagingFinished registry key, and once it is found, it moves on to the next program.  It records all this information than exports it as a CSV file.

By utilizing this script as a AppV prelaunch/startup script we can optimize our users first application startup times and reduce CPU utilization of first-run applications.

Read More

AppV 5 first launch application performance

2014-11-17
/ / /

Our AppV 5 environment is a full infrastructure implementation.  We utilize the management/streaming server to pull the applications down to our Citrix XenApp 6.5 servers.  Our XenApp servers are Citrix PVS servers, we enable the Cache on RAM with disk overflow and the write-cache intermediate mode.  To maximize CPU performance we have our ESXi hosts set to maximum performance and disable power management in the BIOS of the hosts.  We have some applications that are very latency sensitive and the switching of power states on the ESXi hosts have caused performance degradation so we have power management disabled.  We have setup our PVS servers with the secondary D: WriteCache disk where we fully mount the AppV 5 packages, removing the streaming latency that going over a network may add.

Because of some performance concerns with the Shared Content Store (SCS) I was tasked with coming up with a way of determining if there is a performance impact of switching from fully mounted applications.  In order to determine the impact my plan was to measure a baseline based on disk performance.  Our SMB share that we are storing our .appv packages actually has the same performance as the local disk.  Since AppV packages are immutable, the only performance consideration we should be concerned is READ performance from the SMB share compared to the local disk.  The writes occur in the %userprofile%appdatavfs which is stored on the C:\ drive.  The Cache to RAM with disk overflow feature would ensure that write performance into those directories are fast and should be near instant.

With that said, I’ve used the diskspd.exe application (new from Microsoft) to measure performance. AppV 5 utilizes a 64KB allocation size so that’s what we’ll set as our -b value.  We’ll measure Latency statistics comparing local disk to the file share as well.

D:\diskspd.exe -c1G -b64K -L -d60 D:test.dat
D:\diskspd.exe -c1G -b64K -L -d60 \citrixnas01ctx_images_testtest.dat

Results:
D:\

SMB share:

 

Performance of the SMB share vs local disk:
MB/s: 96%
IO per s: 96%
AvgLat: 46%

Based on these results, the local disk appears to be nearly identical to the SMB share with the average latency a little more than half on the local disk.  Although it’s half on average, we are still in the sub 1ms time range which is significantly faster than you could get with a physical server with a single local disk.

The next test I have is launching an application and getting to the splash screen to see how long it takes to load.  For this test I’ve written a AutoIt script that takes two parameters, the name of the program to launch and the window title to monitor for.

I setup a cmd file with my program (Epic) because it takes some parameters prior to launch.  I then pointed my timer application at it.

The results (with SCS):

The columns are Time Completed, Duration (in ms), Window to check for, Command executed.

After doing a Net Stop AppvClient / Net Start Appclient and then executing our AppV application it takes 196 seconds to start the application.  After that initial launch it takes 12-15 seconds to start.  Something is really dragging our initial application launch time down.  I’ve found if I stop/start the service I need to do a add/publish via Powershell for that application to reduce the 196 seconds.  This then takes first launch down to 48 seconds.  This is how long is takes to start the same application after a system restart:

First launch time after restart is 48 seconds then subsequent launches are essentially identical to just the stop/start appvclient service + add/publish.  Which makes sense as our AppV5_Data_Precache script does a add/publish.  Evidently, we’re going to have to go further into AppV to understand what’s causing it to take so long.  To start, I’m going to detail our package a bit.

The application I’m testing this with is Epic.  It’s a huge application.  AppXManifest is 72MB, FilesystemMetaData.xml is 1.7MB, Registry.Dat is 62MB.

It contains 22,000 files totalling around 2GB in size.

When AppV is “launching” the application for the first time it starts consuming memory and CPU for the 196 seconds that it’s launching, peaking at nearly 600MB RAM and 50% CPU (though most of the time it’s peaked at 25% CPU).

AppV utilization before application launch

 

Start of application launch

 

Peak during launch
Application launched

The AppV Debug logs do not give a whole lot of info as to what AppVClient.exe is doing during this time.  Most of the logs show the application “start” as they setup their components, and when the application has launched.  Almost all the logs show the first second or two of application launch and the last second or two before the GUI.

I launched the application at 12:36:24, it finally displayed the GUI at 12:39:38

The only log that shows data during the entire time is the SHARED PERFORMANCE log.  Unfortunately, the log is undecipherable to me.

Lots of PreCreate, PreCleanup, PreAcquireForSection with no relevant data.

Perfmon.exe doesn’t do a whole lot better with large gaps between file/process/network accesses:

What is it doing between 1:17:41 and 1:18:29?  CPU is pegged but no disk activity

Showing Registry accesses also shows huge gaps between the AppVClient.exe process accessing the system.

Registry information still shows huge gaps in time where the AppVClient.exe is processing

So I’m not sure what the hold up is with regard to the delay for this application.  None of the usual tools I use to monitor performance is giving me any hints or indications of why it’s delaying launch.

First launch delay when stopping/restarting AppV service.
Total time is 221s for the first launch, 15s for subsequent launches

 

 

First launch delay when starting application after a full system restart.
Total time is 38s for first launch then 13s for subsequent launches.

 

Read More

Unable to launch /appvve switch via XenApp 6.5

2014-10-16
/ / /

It appears you can’t launch an AppV environment around an application with the /appvve switch as Microsoft automatically strips the /appvve switch OR because icast.exe freaks out.

I have found an alternative method though:

This will launch a cmd.exe window in the virtual environment in citrix with the passed .exe (cmd.exe in this example).

Dan mentioned another, less character way:

 

Read More

AppV 5 – Unable to launch applications but other users can launch without issue

2014-09-16
/ / /

We have AppV 5 SP2 HF4 installed on a Citrix PVS server with our AppV PackageInstallationRoot redirected to a static drive.  I recently updated the PVS server than found my account could not launch AppV applications but other accounts could launch them without issue.  I deleted my user account from the server, tried launching applications with /appvve (which appeared to work but didn’t actually initialize the AppV environment) and procmon’ed the launch of the application.  Procmon showed me the application actually launching, but immediately terminating.

Bad Launch

 

Good Launch

We can see that the application stops launching at the moment it initializes the virtual environment.  Next we’ll dive into the event logs and see if it gives us any more relevant information.  I chose to enable a few event logs that deal with the early process of AppV5.  I can’t really tell you why I chose those ones, they are just a few that I’ve learned to monitor when I have problems launching apps.

I enabled:
Client-Integration
Client-Orchestration
Client-Vemgr
Client-VirtualizationManager
Subsystem-Venv

Launching the application gave me some error messages.  From Client-VirtualizationManager:

 

If we do the backwards HRESULT dance to try and get a useful error code we get this for the two errors:

According to the link, the object is 04 (

04
Virtualization Manager Client-VirtualizationManager Log or Client-Vemgr Log

)

with error code 0x28 and 0x2c.  Googling those HRESULT codes reveals nothing.  I then looked into Client-Vemgr and found some error messages there:

 

Well, this is interesting.  So it appears that AppV can’t do mappings; which I assume are the CoW (Copy on Write) for the filesystem…?  Procmon does not reveal anything with a access denied or failed to create directories, as a matter of fact it actually creates the %userprofile%\local\Microsoft\AppV\VFS directories.  But it is specifically targeting my user account, as referenced by the SID.

At this point I scanned Procmon for my SID and found it existed in the AppV registry path here:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AppV\Client\Virtualization\LocalVFSSecuredUsers

Procmon did not notice anything wrong (no access denied or such error messages).  I then launched another application with a runas and saw it that a new registry key SID combination was dynamically generated.  I then suspected that maybe the existence of this key maybe causing my issue…  To verify I rebooted the PVS server and without using my account, checked and saw the registry key was there through reboots.  My next test was to delete the key and try launching the application again.

Lo and behold it worked!  AppV 5 regenerated the key and launched the application.  It appears if the key exists then it will fail to create the CoW mappings in the registry (which you can see in the screenshot when it’s creating the MAV keys) and my issue is resolved.

Read More

AppV5 – Update files in an already mounted package

2014-09-04
/ /
in Blog
/

An issue I’ve been dealing with lately is we have some in-house applications baked into an AppV package.  We are encountering some issues with them and they require updating almost on a daily basis.  Since you can’t update/copy .exe’s via a preconfig script or by breaking into the environment (CoW restrictions) they need to be baked into the environment.

One of the new features of AppV 5 vs. 4.6 is that it can mount the files as actual files in the filesystem as opposed to a single binary.  This exposure of the AppV 5 packages allows for manipulation of the files *after* they are deployed, but you need to make some modifications to the mounted files.

1) Navigate to the folder you want to change, right click on it and take ownership of it.  Make sure you replace permissions on all items within the folder.

2) Set the permissions on the folder to Everyone Full.
And at this part you can now replace .exe’s within that folder and subsequent launches of the application now use the new .exe’s.  At this stage I can now replace these files without recreating the package and when another update to that .exe comes down the pipe I don’t have to open the sequencer to update a file, update the share, update the batch file with the new /appvve: GUID’s.  At least until their software becomes stable and final for this build.
Read More