AppV

AppV5 – Applications being published to “C:\App-V Packages” instead of %PACKAGEINSTALLATIONROOT%

2016-09-21
/ /
in Blog
/

The last little while we’ve had an issue with some applications seemingly being published to “C:\App-V Packages” instead of our defined path “D:\AppVData\PackageInstallationRoot”.

AppV-Packages

Notice the timestamps? Packages published during the same ‘publishing refresh’

 

Video of the issue in action that I was able to setup a repro:

So what’s going on here that would cause this to happen?

I turned to the event viewer to find the time when this package “8FE4C99E…” was published and just looked to see if there was anything surrounding it.

Highlighted is the package publish event

This is the event showing the package being published to “C:\App-V Packages”.  There is an odd thing different about this publish event compared to the rest (that you can see above).  Between ‘Configure Package’ and when the ‘Streaming’ starts there is a ‘Publishing Refresh’ event.

Publishing refresh

Why is there a Publishing Refresh event?  It turns out, it can be triggered by a group policy refresh.

GP Refresh

Notice the time stamps?

So, is there something that occurs during Group Policy refresh that could cause this?

To test this I used procmon to monitor for PackageInstallationRoot registry key and ran a Group Policy Refresh:

screen-shot-2016-09-21-at-8-27-01-am

Procmon_PackageInstallationRoot

The key is deleted then recreated and inbetween the AppV service is trying to read it.

So what is happening here?

At our organization we prefer to set Group Policies using Group Policies Preferences – Registry because of the power and flexibility of Item Level Targetting.  We prefer to only apply registry keys/policies to specific systems in certain groups rather than having a complicated OU structure.  Since I know we have a policy that configures our AppV5 values I went and looked into how this value was being configured:

GroupPolicyManagement

The ‘Action’ is ‘Replace’

screen-shot-2016-09-21-at-8-44-18-am

We have this value set to ‘Replace‘.  What does replace do?

Replace Delete and recreate a registry value or key for computers or users. If the target is a registry value, the net result of the Replace action is to overwrite all existing settings associated with the registry value. If the target is a registry key, the net result is to delete all values and subkeys in the key, leaving only a default value name with no data. If the registry value or key does not exist, then the Replace action creates a new registry value or key.

Well, process monitor is showing EXACTLY that scenario.  Replace is deleting the key and recreating it.  In between that time ‘AppVClient.exe’ is trying to read that value.  If PackageInstallationRoot doesn’t exist, then AppV defaults to ‘C:\App-V Packages” as the PackageInstallationRoot.

screen-shot-2016-09-21-at-10-01-23-am

No “PackageInstallationRoot” key -> “C:\App-V Packages” folder is where packages go.

 

It appears we have a race condition between group policy being applied and when our AppVClient is refreshing.  It is less than 600ms but that’s enough time for a package or two to start their refresh and get caught in ‘default folder’ purgatory.

How can we fix this?

The easiest solution and the one we are going to implement:

We’re going to change the action to “Update”.  When I initially tried to change the action was greyed out.  When “Remove this item when it is no longer applied” is checked, it forces the action to “Replace”.
screen-shot-2016-09-21-at-8-44-27-am

I had to ‘uncheck’ that value

screen-shot-2016-09-21-at-8-44-34-am

And then I could select ‘Update’.

screen-shot-2016-09-21-at-8-44-43-am

After updating the GPO and refreshing policies on the affected system, I ran procmon to verify that it does not delete the value anymore:

GPP_Update

It does not delete the value, it now ‘Sets’ the value to the same string we had before.  I don’t know if the individual ‘Set’ action will cause an issue; I suspect not since the value will always exist and is technically the same value throughout the entire set of actions that this issue is now ‘resolved’ for us.

Read More

AppV 5.1 Sequencer – Not capturing all registry keys

2016-09-20
/ /
in Blog
/

Video of this issue:

https://theorypc-my.sharepoint.com/personal/trententtye_theorypc_onmicrosoft_com/_layouts/15/guestaccess.aspx?guestaccesstoken=vq4pmhseZam8zGPBh6Q8bn%2bgvaZrVuFc4fXo%2fziYmeA%3d&docid=08e53da35fa314929a7ce6578c69bf5c5

The issue is when sequencing an application (100% reproducable on Epic and the VMWare Hypervisor) and then you add a large ‘update’ (for Epic this is the client pack) then not all registry keys are captured.  At 8:20 seconds you can see keys that are present in the local registry are not present in the package.

appv_bug

 

Notice the “0”, “win32” and FLAGS keys are missing in the AppV package.

This is the script I used to compare the local registry vs the package:

 

Read More

ControlUp – List AppV5 recent events on various servers

2016-08-26
/ /
in Blog
/

David Falkus just posted a blog post on using Powershell to combine multiple AppV5 logs into a single view and orders them chronologically so you can see the events as they occurred.

Since this was a PowerShell script we can use ControlUp to import it, tweak it to accept some server variables and then get the output back to us.  Here is a video of this in action:

Here is the recipe for it:

1

2

3

4

 

And the script:

 

Read More

AppV5 – Package {GUID} version {GUID} failed configuration in folder ‘%packageinstallationroot% with error 0x4C40310C-0x12

2016-03-30
/ /
in Blog
/

We experienced this error on a package on one of our Citrix servers in the AppVClientAdmin event logs.  Attempting to procmon this error didn’t reveal anything substantial for what could be causing this issue.  I then tried my last post to enable AppV debug logs to see if we can see what’s going on.

Because this server was being used by other users the log generated a lot of noise.  I stopped the log as soon as I got the error message though, which meant the error should be at the end of the generated log.  Fortunately, you can search for the error message:

[2]06B0.2DE4::‎2016‎-‎03‎-‎30 09:41:30.817 [Microsoft-AppV-Client]Package {499ed340-c809-47dc-a533-2cdeab537e93} version {3589c28b-edb7-41d6-865d-e01c4fdd4318} failed configuration in folder ‘D:AppVDataPackageInstallationRoot’ with error 0x4C40310C-0x12. 

I suspect the first bit of hexadecimal code (06B0.2DE4) are probably an identifier for a thread or some such so I suspect if I search for just this code I can be shown all events that lead up to this error:

Looks like it.

The log up to the error:

With the error time code, I can compare what Procmon says is going on.  And procmon reports back the following (at exactly 09:41:30.815):

Procmon has the AppVClient.exe doing the following action “QuerySecurityFile” and it’s looking for “Information:Owner”.  So what is this the value of this property?

Hmmm…  This does not look correct to me.  From my experience, I think the current owner ends up being ‘SYSTEM’ because that’s the account the AppVClient.exe runs under when it creates this folder.  As a hunch, I looked at a system that is operating correctly and compared it’s Owner attribute:

Sure enough, on a working system the Owner is SYSTEM.  So I’ll attempt to modify the non-working system and retry adding the package:

Success!  So it appears this error “0x4C40310C-0x12″ is DIRECTLY related to the ownership attribute on your PackageInstallationRoot folder.  If it is anything but SYSTEM it looks like it fails.  I do not know why that attribute changed, but changing it back to SYSTEM resolves this error code.

Read More

AppV5 – The trouble with AppV5 logs (and a solution!)

2016-03-24
/ /
in Blog
/

AppV5 introduced to us a slew of logs that made troubleshooting more difficult than it should have been.

For instance, say you get an error message:

How do you start debugging this?  Well, you could start by trying to decipher the error message AppV has generated to a decimal value that you can then look at the first two digits to determine which ‘part’ of AppV generated the error and you can examine that specific log.  Sounds kind of ridiculous when you spell it out.

Another thing you could do is you could enable all logs just to be sure, but then you have to sift through the logs to find the time stamp that corresponds closest to the event that generated the message.  Sometimes it would be too far or close together or the debug logs generate so much information that your event viewer is just one big blob of ‘the exact same time’.

One of these could be the cause of my issue.  Going through them all is extremely cumbersome.

Of course this isn’t true that all these events are the same time, but it’s all event viewer can display.  If you view the XML of the event you can see there is more precise time stamps, but regardless, trying to compare each event with other logs is cumbersome and very difficult.

Is there a better way?

Yes!  There is!  And Microsoft has actually documented it here:
https://support.microsoft.com/en-us/kb/3037955

This is a powershell script that generates a ETL file on your desktop then converts it to a text format.  The script is:

What does the output look like?

First thing to notice is that it’s sorted chronologically, so if you can get a narrow time range to examine the error occurring it should be easier to spot.  Second thing is all event logs are added, and events themselves are included as they are generated.  So it should be easier to finding that specific component that causes the error.

If you are encountering AppV5 errors, this may be easier to help track down the error then trying to sift through the debug logs in event viewer.

Read More

AppV5 – Package fails to load with error 0x79100E100-0xC (Starring Procmon)

2016-03-17
/ /
in Blog
/

We were having an issue with a AppV5 package loading and we received the following error message:
Event ID 1008
Package {b8b01729-ed31-4d77-a859-dbd8b82a3372} version {e1d21ac7-84f0-4ab7-998f-e3258be91298} failed configuration in folder ‘D:AppVDataPackageInstallationRootB8B01729-ED31-4D77-A859-DBD8B82A3372E1D21AC7-84F0-4AB7-998F-E3258BE91298’ with error 0x79100E10-0xC.

This message is preceeded by this message:
Event ID 4009
machine script for event AddPackage with command line: ‘cmd.exe’ exited with failure error code: Incorrect function.. Because Rollback is set to true in the script definition, the current AppV Client operation was rolled back.

I started investigating.  The first thing I did was open a powershell window and tried to load the package via the command line:

I then looked at our DynamicDeployment XML file and examined the script it is trying to launch:

The script it’s trying to launch looks like this:

Since this is a Machine Script, I started a process monitor capture, executed my command, stopped the capture then used the “Process Tree”

and clicked on AppVClient.exe and clicked “Include Subtree”.

and used the “show process and thread activity”.  I filtered on ‘Detail’ ‘Begins with’ and set it for both Parent and Exit so I can look at the process path and exit codes:

 

The ‘exit code’ is ‘1’ (Exit Status: 1) which means an error occurred that caused the script to fail.  So now we dive into that script and see why it’s failing:

From looking at the script, it’s trying to create a new ‘mklink’ path.  If I try and run this command manually, I get the following:

So this is where the errorlevel (also exit code) is being set.  The last error level is 1 which becomes the exit code once the script runs ‘EXIT.

So, there are two methods I can think of to solve this problem.

1) I can set EXIT /B 0 to always set EXIT to report and error code of ‘0’.
2) Check to see if the path already exists and then EXIT

I chose to modify the script to exit if the path exists.  I changed it like so:

I cleared procmon, set it to trace again and attempted to run the add-appvclientpackage command again.

I selected ‘AppVClient.exe’ and clicked ‘Include Subtree’

This time we can see that the ‘AHS-ATOP.cmd’ script has an ‘Exit Status: 0’ which means it completed successfully.  But, the next script, ‘AHS-SoftWorksGroup-ScreenTestIII.cmd’ with the parameter ‘INSTALL’ fails with ‘Exit Status: 1’.  Again, we look into the script…

We can see it has the same flaw.  I then modified the script to add the same IF EXISTS check.  I then cleared procmon and reattempted to add the package.

Success!!!

Hurray!  The package loaded and the scripts ran correctly.

An alternative to all of this is I could have changed the ‘rollback’ to ‘false’ in the DeploymentConfig.xml file, but I would rather the package *not* load if the ‘INSTALL’ script fails for whatever reason.  This ensures an error is generated that needs to be dealt with rather than potentially having a half-working package.  These INSTALL scripts were simple enough that a simple directory check would suffice to ensure it’s working and that’s what I’ve done.

Read More

AppV5 – Sequencing Logibec eClinibase

2016-03-11
/ /
in Blog
/

I was given a new application to sequence:
Logibec’s eClinibase.

This application has some restrictions:
1) It has an updater that will pull down .exe’s
2) It won’t launch from a script in the bubble.  Example:
cmd.exe /appvve:%PACKAGEGUID%_%VERSIONGUID%
cd /d C:LogibeceClinibaseilgi11
eclinibase.exe

Will crash.

I’m going to address point 2) first because it’s weird and interesting.

Why does it crash?  Because, somehow, it breaks out of the bubble.  The reason it crashes is it looking for some registry keys that don’t exist outside the bubble and it explodes.  The keys it looks for are:

On a local install if I delete the environment keys that matchup to the folder we’re launching eClinibase from, I get the same error.  Since those keys don’t exist outside the bubble, we get the same error…  Which means it’s trying to look outside the bubble for some reason.  Powershell, however, works correctly but only if the exe is called directly:

However, we CAN get it to stay ‘in the bubble’ if we launch the eClinibase.exe with the /appvve: switch.

This is the first time I’ve ever encountered this where launching the application from a command prompt inside the AppV bubble fails, but it will work if you launch the exe with the /appvve: switch.

So that’s weird that we need to launch it specifically that way, but it appears to work so I guess that’s a limitation we have to roll with.

So how do we address point 1)?

I’ve used this technique before and it works here.  We will create a directory symlink to a network file share where the users will have full permission to update the application.  Since these updates occur ‘as needed’ the different environments may update at different times, this keeps them in sync across all of our Citrix servers since they will all point to the same location.

I add the mklink command the DynamicDeploymentConfig.xml:

And we are good to go!

Read More

AppV5 – Sequencing an application that has Com+ Applications

2015-12-04
/ / /

AppV has a few weaknesses, one of them is Com+ Objects.  It doesn’t like Com+ Objects and sequencing an application with them generally results in this error message:

The Resolution seem easy enough:
1) Try extracting and installing the COM+ component natively on both sequencing station and the client
2) Thoroughly test your virtual application package to verify functionality.
But how do you extract the COM+ components?
1) Open ‘Component Services’
2) Expand until you find your Com+ Applications, right-click and select ‘Export’
3) Click ‘Next’
4) Enter a path to save your MSI then click ‘Next’
5) Click ‘Finish’
Since this application had two Com+ Applications I exported my second one as well:
To ensure they work, I took them to our XenApp Test server and installed them.
Both *appeared* to have installed correctly.  But this application requires the Com+ Application to run under a service account.  Checking the original install:
I can see under the ‘Identity’ tab that it is configured for ‘This user:’.  I then checked the MSI install on the XenApp Test server and looked at the Identity tab:

 

It appears it defaults to ‘System Account’ even though I selected ‘Export user identities with roles’.  To change the user account silently, I used this script supplied by the vendor (as a encrypted vbe):
This script changes the user account on the Com+ applications to use the proper account.  When browsing the properties of the components of the SFSQL object I saw it referenced the dll to it:

 

 

Fortunately, it seems these files are captured by the MSI export of the Com+ Application.
To test everything now, I must now create my AppV package.  Before doing the install I installed my prerequisites (the Com+ Applications) into the Sequencer so they are not captured by the package.
I ran this script to do my prereq install:
Now I startup the sequencer and do my sequencing first steps.
I selected a batch file I created for the install (per the vendor’s initial instructions):
Once the install completed I saved my package and took it to my XenApp Test server and published the application and ran it.
No, this is not correct
For some reason my application was failing.  I double checked the COM+ Applications are installed and appear to be working, so I opened Procmon.exe to see if I can find where it’s going wrong:
I’ve used procmon enough now to practically pull the old’ Matrix, “blonde, brunette, red head“.  Working from bottom to top, the message box is generated from the ‘Text’ (LanguagePack) back to the KernelBase.dll.  So we can exclude that has the cause, this is telling us *of* the issue.  So if we look at the lines preceding it, one of them must be causing our message.
Looking at the first keys, the OLE keys, I can compare them to my sequencer and they match exactly, so the likelihood of them causing our issues is none.
So we go back a little further to the AppID key and jumping to the *SUCCESS* location to look at it’s value…
I am on a server called WSCTXAPP301T but the registry key for this value is pointing to the system I did my sequencing on.  I changed the value to the actual server name and did a registry search for “WSAPVSEQ07” to see if it appeared anywhere else.  It did in one other location so I modified the value there as well to point to the local server:
And tested my application again:
Success!  The login screen.  This is as far as I can take this app until I get credentials or a user to login and try testing for me.  So, for now, I can create a preflight-script to setup some prerequisites when the application is published.  I modified my preflight script to include the two registry values being updated:

Then I updated the Deployment_Config.xml file for this application:

Lastly, I logged onto a new, fresh, system and manually tested my XML file by publishing from the command line:

Using Procmon to monitor to ensure the prerequisites get installed:

“Exit Status: 0” –> Usually a good thing 🙂

Testing the application:

Looks good

Then testing removal and monitoring with Procmon:

Com+ Application are removed cleanly 🙂

So everything looks good and I can now add this application to our AppV Management Server and deploy it to the machines that need it.

Read More

AppV5 – When Application Compatibility Toolkit doesn’t work

2015-11-27
/ / /

I have an application that embeds an IE window and generates a small HTML file for some content.  This application has some…..   interesting….  configuration settings that require a folder with “EVERYONE:F” permissions.  This folder cannot be in a path that contains spaces, cannot be a environment variable and the vendor recommends to put the folder on the root of the C: drive as C:TMP.  To make it more crazy, when launching the application it will generate a folder under the C:TMP that is locked to the %CLIENTNAME% (e.g., C:TMPComputerName) and every login to the application generates unique files to that session, so technically, each folder underneath TMP is unique to the user.  The vendor actually recommends completely deleting the contents of the TMP folder DAILY.

So why not just store these files in %TEMP% instead?  Great question.  If the vendor made that happen I wouldn’t have this lovely blog post.

(Just for a note this application utilized multiple folders on the root of C: as well, TMP being one, since we move our PackageInstallationRoot to a different drive, TMP isn’t available on C:)

With that all said, what am I looking to solve?

Well, I don’t want to be creating folders on the root of the C: drive as this application will be on our ‘generic’ PVS Citrix servers and minimizing the potential for pollution on the master image is a goal, having to create a folder and configure permissions is possible and I’ve done it before but it’s not very clean or elegant; really if we’re going to do that then we may as well pollute the C: drive.  We want to maintain portability so creating a C:TMP folder on the master image prevents us from publishing this package on desktops or other systems in the future unless this is well documented requirement.

We’ve just learned a new trick though, we can try using Microsoft ACT to create a file path shim that redirects the path to a different one.  So let’s do that…

The problem.

I created the shim using the steps from this post.  (C:TMP;C:ProgramDataMicrosoftAppVClientIntegrationD8E3DB68-4E48-4409-8E95-4354CC6E664BRootVFSProgramFilesX64dlc11.2TMP) I then launched the application and…

Huh.  That doesn’t look right.

And it didn’t work.  I then used procmon.exe to examine to see if it was reading the file correctly:

I see SUCCESS…

Procmon.exe is reporting that it IS reading that HTM file correctly.  So why isn’t it being displayed?

I ran across a similar issue on another application and what I found was that the embedded component appears to be running *outside* the bubble.  Since the shim targets applications (in this case prowin32.exe) all reads from prowin32.exe are being redirected by other processes are not.  I suspect (though I have no proof), in this case, that somehow the IE component is breaking outside the bubble and so it’s NOT getting redirected.

Can we force all programs to get redirected to the proper path?  Yes, we can.

Using symbolic links we can force any access to the directory to be redirected to the AppV package path.  AppV will then see this is a path within the ‘bubble’ and redirect *again* to your local profile where the AppV5 writes will take place.

I removed the shim and then executed:

Symbolic Link
mklink /d C:TMP C:ProgramDataMicrosoftAppVClientIntegrationD8E3DB68-4E48-4409-8E95-4354CC6E664BRootVFSProgramFilesX64dlc11.2TMP

Tracing with procmon.exe now showed us this:

And the application now displayed this:

So the application is now working without any issues, all of these ‘temp’ files are being redirected to a location where they will not be saved between sessions so no cleanup is ever needed.  We need to add the mklink.exe to the DeploymentConfig.xml.

AHS-BDMPHARMACY-PREFLIGHT.CMD:

And we are now done.

Read More

AppV5 – Integrating Certificates into your AppV package

2015-10-20
/ / /

These are the steps I’ve found to sequence root certificates into your AppV5 application.

Where do you get certmgr.exe from?

The visual studio downloads apparently contains this tool.

Once you’ve started your sequencer and run the command it will add the certificate to these two places:
HKLM\SOFTWARE\Microsoft\SystemCertificates\ROOT\Certificates\
HKLM\SOFTWARE\Wow6432Node\Microsoft\SystemCertificates\ROOT\Certificates

And that’s how you add certificates to a sequenced package.

Read More