Registry

AppV5 – Applications being published to “C:\App-V Packages” instead of %PACKAGEINSTALLATIONROOT%

2016-09-21
/ /
in Blog
/

The last little while we’ve had an issue with some applications seemingly being published to “C:\App-V Packages” instead of our defined path “D:\AppVData\PackageInstallationRoot”.

AppV-Packages

Notice the timestamps? Packages published during the same ‘publishing refresh’

 

Video of the issue in action that I was able to setup a repro:

So what’s going on here that would cause this to happen?

I turned to the event viewer to find the time when this package “8FE4C99E…” was published and just looked to see if there was anything surrounding it.

Highlighted is the package publish event

This is the event showing the package being published to “C:\App-V Packages”.  There is an odd thing different about this publish event compared to the rest (that you can see above).  Between ‘Configure Package’ and when the ‘Streaming’ starts there is a ‘Publishing Refresh’ event.

Publishing refresh

Why is there a Publishing Refresh event?  It turns out, it can be triggered by a group policy refresh.

GP Refresh

Notice the time stamps?

So, is there something that occurs during Group Policy refresh that could cause this?

To test this I used procmon to monitor for PackageInstallationRoot registry key and ran a Group Policy Refresh:

screen-shot-2016-09-21-at-8-27-01-am

Procmon_PackageInstallationRoot

The key is deleted then recreated and inbetween the AppV service is trying to read it.

So what is happening here?

At our organization we prefer to set Group Policies using Group Policies Preferences – Registry because of the power and flexibility of Item Level Targetting.  We prefer to only apply registry keys/policies to specific systems in certain groups rather than having a complicated OU structure.  Since I know we have a policy that configures our AppV5 values I went and looked into how this value was being configured:

GroupPolicyManagement

The ‘Action’ is ‘Replace’

screen-shot-2016-09-21-at-8-44-18-am

We have this value set to ‘Replace‘.  What does replace do?

Replace Delete and recreate a registry value or key for computers or users. If the target is a registry value, the net result of the Replace action is to overwrite all existing settings associated with the registry value. If the target is a registry key, the net result is to delete all values and subkeys in the key, leaving only a default value name with no data. If the registry value or key does not exist, then the Replace action creates a new registry value or key.

Well, process monitor is showing EXACTLY that scenario.  Replace is deleting the key and recreating it.  In between that time ‘AppVClient.exe’ is trying to read that value.  If PackageInstallationRoot doesn’t exist, then AppV defaults to ‘C:\App-V Packages” as the PackageInstallationRoot.

screen-shot-2016-09-21-at-10-01-23-am

No “PackageInstallationRoot” key -> “C:\App-V Packages” folder is where packages go.

 

It appears we have a race condition between group policy being applied and when our AppVClient is refreshing.  It is less than 600ms but that’s enough time for a package or two to start their refresh and get caught in ‘default folder’ purgatory.

How can we fix this?

The easiest solution and the one we are going to implement:

We’re going to change the action to “Update”.  When I initially tried to change the action was greyed out.  When “Remove this item when it is no longer applied” is checked, it forces the action to “Replace”.
screen-shot-2016-09-21-at-8-44-27-am

I had to ‘uncheck’ that value

screen-shot-2016-09-21-at-8-44-34-am

And then I could select ‘Update’.

screen-shot-2016-09-21-at-8-44-43-am

After updating the GPO and refreshing policies on the affected system, I ran procmon to verify that it does not delete the value anymore:

GPP_Update

It does not delete the value, it now ‘Sets’ the value to the same string we had before.  I don’t know if the individual ‘Set’ action will cause an issue; I suspect not since the value will always exist and is technically the same value throughout the entire set of actions that this issue is now ‘resolved’ for us.

Read More

Citrix XenApp – Graphical Artifacts

2016-07-22
/ /
in Blog
/

In our Citrix XenApp 6.5 environment we started having a couple applications encounter an issue where they would experience some serious graphical artifacts.  What was supposed to look like this:

1

Would look like this:

2

 

Here’s a short video demonstrating this issue:

Or sometimes it would show the windows *behind* the artifacted image.  That is, instead of the ‘White’ you see in my image, the application behind it shows through.

When investigating this we found there was a couple symptoms that we were going to experience these artifacts.

  1. The window would become ‘frosted’ or ‘ghosted’ (as seen in Spy++ or AutoIt Window Info)
    4
  2. The application would switch to ‘Not Responding’
    3
  3. If you completed the task ‘Edit’ quickly there would be no artifacting (time was important)
  4. When ‘timing’ the switch from ‘normal’ to frosted or ghosted window it would be around 5-7 seconds.

So what’s going on here?

For this particular instance, the application is launching MSPaint with some modified properties.  It sets Paint to ‘Always On Top’, which in itself isn’t an issue, but then it purposefully locks the UI so you must complete the drawing and close paint before continuing.  This is how the vendor designed this application to operate with this workflow.

And what’s Windows doing?  It turns out, Windows is trying to alert you that your program is non-responsive!  Windows has a built in feature to ‘Frost/Ghost’ the window of a non-responsive UI to prevent you from entering input that won’t be received.  The ghosting effect is time sensitive!  So that explains why if we opened and closed our document quickly their would be no artifacting but if we manipulated it for some time the artifacts would appear.  The time limit for monitoring unresponsiveness is 5 seconds.  DWM.exe is the process responsible for creating the ‘Frost’ window and when responsive returns, it appears it does a poor job telling the application to repaint all affected Windows.

Microsoft recommends a couple ‘fixes’ which is really a programmatic way to ‘disable’ the ghosting feature.  The two methods Microsoft suggests is to create a NoGhost application compatibility fix or have the programmer use ‘DisableProcessWindowsGhosting’.  But there is a 3rd method.

The 5 second time limit is programmable.  If we extend the timeout we don’t need to configure ‘NoGhost’ compatibility fixes for each app or go back to the vendor.  The timer is global and affects every application and window.  Unfortunately, I know of no way to permanently disable it, but we can set a high enough value to prevent it from appearing.

So what do we have to do to ‘resolve’ this?

My preferred choice was to use Group Policy Preferences (Registry) and set a new value for HKCU\Control Panel\Desktop /v HungAppTimeout /t REG_SZ /d 120000

This sets the timeout to 2 minutes as opposed to 5 seconds.  Now when our program is used we get this result:

No More Artifacts.

When I was investigating this I found I could get the artifacting to occur in both ICA and RDP but not when on the console.  The frustrating thing about this issue is that it was not consistent.  Because of the 5 second default time limit, the program(s) we had that would ‘artifact’ would sometimes complete their UI locking job faster than 5 seconds, but sometimes not.  This lead to reports of artifacting occurring more often ‘during the peak work hours’ when the application/server/user load was the most.  This makes sense as the higher load undoubtedly lead to everything being slower, the database, server, etc. leading to the application waiting longer and thus exceeding the timeout.  I did find through the course of troubleshooting this issue that it seemed to ‘go away’ when I was trying to replicate after hours, which is frustrating to only have a slice of time to try and resolve this during peak hours.

Fortunately, after implementing the HungAppTimeout registry key the artifacts for several application ‘went away’.

Lastly, contrary to this article you do NOT need to restart for this value to take effect.  WinLogon.exe reads the HungAppTimeout value and then configures DWM accordingly when your profile loads.  So for this value to take effect you only need to log on with this value already residing in your user’s registry hive.

5

Read More

Citrix Client Side Extensions 1.7.6+ breaks policies

2016-07-14
/ /
in Blog
/

We apply our several group policy settings via Active Directory group policy objects, within them we set ‘Citrix’ policies.  This includes things like ‘Licensing’, platform, etc.

AD_Policies

 

When we upgraded the Citrix Group Policy Client Side Extensions (x86) to version 1.7.6 or 1.7.7 we found that these values were no longer being applied.  FYI!

 

 

Read More

Citrix XenApp 6.5 – IMA errors galore, mfcom won’t start

2016-07-04
/ /
in Blog
/

I’ve seen this happen a few times now where the “Citrix Independent Management Architecture” (aka IMAService) won’t start, erroring with various errors:

All of these errors appear to be a registry with incorrect permissions configured on the Citrix keys.  Why did these keys get their permissions reset?  I’m unsure.  I DID just install Citrix UPM 5.4 which may reset the keys?

Here is how you fix the permissions (at least, everything I could possibly find):
1) Download SetACL.exe
2) Save this file to ‘CitrixRegPerms.txt’:

You may need to identify the local SID for ‘NETWORKSERVICE’.  In my example the value is:

 

You may need to replace your SID for NetworkService with the one from my file above.

Lastly this script will ‘fix’ the incorrect permissions:

Done.

 

Read More

Microsoft Office – Your mailbox is currently being optimized as part of upgrading to Outlook 2010

2016-06-15
/ /
in Blog
/

I was using our Office 2010 through Citrix and started noticing I was getting this message:

1

“Your mailbox is currently being optimized as part of upgrading to Outlook 2010.  This one time process may take over 15 minutes and performance may be affected while optimization is in progress.”

“Upgrade Outlook Connector”

“You must upgrade to the latest version of Outlook Hotmail Connector to continue using this e-mail account.”

In addition, all of the options under ‘Account Settings’ do nothing.  This is in addition to my Home/Inbox being empty and my calendar blank.  Attempting to create a calendar event results in:
2

I then tried to open the ‘Mail’ control panel and found it was missing.  I manually opened it by launching it from:
“C:\Program Files (x86)\Microsoft Office\Office14\MLCFG32.CPL”

3

From here, clicking any of the buttons resulted in nothing happening.  Email Accounts… Data Files… did nothing.  Clicking ‘Show Profiles’ did work, though.  So what the heck is going on?  I opened up Process Monitor to try and find out.  I opened up the ‘Mail’ control panel via the command line, since this is a ‘Rundll32.exe’ I set the procmon filter to ‘Command Line’ ‘contains’ ‘MLCFG’ and clicked one of the non-functioning buttons.  My result:

4 5

PATH NOT FOUND.  Why isn’t the path found?  I opened a command prompt and followed the path doing a dir /x to show shortnames:

6

Well, there is the issue.  It’s showing as ‘MICROS~4’ when it’s searching for ‘MICROS~2’.  Doing a repair on office will cause re-registration to occur and this does resolve the issue, but I wanted to change my shortname instead of doing a repair.  Since these are PVS servers I created a new version, mounted the vDisk and changed the shortnames via fsutil:

7

In the screenshot, you can clearly see ‘Microsoft Office’ is now ‘MICROS~2’.  I unmounted the vDisk, set a target device and booted it up.  I opened the command prompt and…

8

What the heck?  For some reason mounting my vDisk offline and I can’t change the shortname.  I get access denied trying to change the shortname when the server is online so I was hoping offline would work.  Unfortunately, it does not appear to be so.

So what is the solution?  I guess repair office to force registration and it will point to the new paths because it will re-register all components. 🙁

Read More

Multi-homed server, lock ICA and RDP to a specific NIC

2016-04-21
/ /
in Blog
/

We implement a multi-homed setup with Citrix Provisioning Services.  We have all of our production traffic on one NIC and all PVS traffic on a second nic.  This helps us in troubleshooting when doing packet captures but does introduce other sets of challenges.  One of these challenges is when I uninstalled and then installed updated VMWare tools on one of our vDisks it caused the NIC’s to renumber and reorder themselves.  You’ve probably read some articles saying to ‘show hidden devices’ and uninstall any ‘ghost’ devices; with a multi-homed setup this may not resolve your issue.  My specific issue is my NIC’s now went from “0x1 and 0x2” to “0x3 and 0x4” in the LANATABLE.  We apply a GPO to the ICA-TCP and RDP-TCP to force them to only utilize our ‘Production’ NIC which we decided was going to be the second NIC.

Provision = 1st Nic, Production = 2nd Nic

I uninstalled the ghost devices but because this change is all ‘in the registry’ it wasn’t immediately noticeable by myself that the LANATABLE and NIC ordering had changed.  I promoted my vDisk and then tried to RDP into it:

Well…  I knew this wasn’t right.  So I logged onto the server and checked the LANATABLE values:

Provision NIC
Production NIC

 

Provision NIC LanaID = 0x3 after the VMWare Tools upgrade

 

Production NIC LanaID = 0x4 after the VMWare Tools upgrade

 

RDP targets LanAdapter 0x1 (!?)

 

ICA targets LanAdapter 0x2

A couple issues popped out.   The first was that the LanaID’s were wrong.  I *thought* Provision should be #1 and Production should be #2, but we apply our LanAdapter ID’s via GPP so these values are correct for our other systems.  I know both RDP and ICA need to be locked to the Production NIC and the order is *correct* but I’m a bit confused to why the numbers are different between RDP-TCP and ICA-TCP.

So I started on getting the issues resolved.  First, I was going to resolve RDP.  If it is targeting 0x1 that means that I need the Production NIC (VMWare Network Adapter #2) needs to be set as 0x1.  So I edit the LanaId of the Production NIC to 0x1 and the Provision NIC to 0x2.  I rebooted the box and I could RDP into it without issue.  I then checked the Remote Desktop Session Host Configuration:

This is the correct value.

OK, so RDP is set correctly and works.  I then tried to launch a Citrix application.

It failed.

It generated an event on the server:

Unable to connect to the CGP tunnel destination

Looking at the ICA Listener configuration showed me the following:

The value is wrong.  It should be #2

So the ICA-TCP listener was set to the ‘Provision’ NIC.  So our production traffic was not getting to it. This is the wrong value.  My first thought was the LANATABLE would make sense here…  We have the LANADAPTER key is set to 0x2 which would equal the ‘Provision’ NIC under this configuraiton…   So I changed the LANATABLE to be the reverse.  0x1 = Provision NIC and 0x2 = Production NIC.

The results:

Now both are wrong!

Jeez.
I reverted the LANATABLE to 0x1 = Production and 0x2 = Provision.  Again, only the RDP-TCP connection changed.  At this point the ICA Listener *must* be looking at another place…  I used Procmon to trace the registry when I opened the ICA Listener Configuration and noticed it did NOT query LANATABLE but did go through and look and query this key:
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesTcpipLinkage
The BIND order is set as Production (#2 NIC) first and Provision second in this order…  To change this order you need to go to Advanced Settings and modify the connection order:
Provision *should* be the top NIC here…
So I used the arrows on the side and moved Production to be second in this order:
Rebooted and I checked the ICA Listener:
Hurray!
I then tried both RDP and ICA traffic and both now worked correctly.
So, the lesson learned here is that this registry key:
Targets the BINDING order of the NIC’s.
And this registry key:

Targets the LANATABLE and the values specified within.

Read More

AppV 5 – Unable to launch applications but other users can launch without issue

2014-09-16
/ / /

We have AppV 5 SP2 HF4 installed on a Citrix PVS server with our AppV PackageInstallationRoot redirected to a static drive.  I recently updated the PVS server than found my account could not launch AppV applications but other accounts could launch them without issue.  I deleted my user account from the server, tried launching applications with /appvve (which appeared to work but didn’t actually initialize the AppV environment) and procmon’ed the launch of the application.  Procmon showed me the application actually launching, but immediately terminating.

Bad Launch

 

Good Launch

We can see that the application stops launching at the moment it initializes the virtual environment.  Next we’ll dive into the event logs and see if it gives us any more relevant information.  I chose to enable a few event logs that deal with the early process of AppV5.  I can’t really tell you why I chose those ones, they are just a few that I’ve learned to monitor when I have problems launching apps.

I enabled:
Client-Integration
Client-Orchestration
Client-Vemgr
Client-VirtualizationManager
Subsystem-Venv

Launching the application gave me some error messages.  From Client-VirtualizationManager:

 

If we do the backwards HRESULT dance to try and get a useful error code we get this for the two errors:

According to the link, the object is 04 (

04
Virtualization Manager Client-VirtualizationManager Log or Client-Vemgr Log

)

with error code 0x28 and 0x2c.  Googling those HRESULT codes reveals nothing.  I then looked into Client-Vemgr and found some error messages there:

 

Well, this is interesting.  So it appears that AppV can’t do mappings; which I assume are the CoW (Copy on Write) for the filesystem…?  Procmon does not reveal anything with a access denied or failed to create directories, as a matter of fact it actually creates the %userprofile%\local\Microsoft\AppV\VFS directories.  But it is specifically targeting my user account, as referenced by the SID.

At this point I scanned Procmon for my SID and found it existed in the AppV registry path here:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AppV\Client\Virtualization\LocalVFSSecuredUsers

Procmon did not notice anything wrong (no access denied or such error messages).  I then launched another application with a runas and saw it that a new registry key SID combination was dynamically generated.  I then suspected that maybe the existence of this key maybe causing my issue…  To verify I rebooted the PVS server and without using my account, checked and saw the registry key was there through reboots.  My next test was to delete the key and try launching the application again.

Lo and behold it worked!  AppV 5 regenerated the key and launched the application.  It appears if the key exists then it will fail to create the CoW mappings in the registry (which you can see in the screenshot when it’s creating the MAV keys) and my issue is resolved.

Read More

IMA Service Fails with the Following Events: 3989, 3634, 3614

2013-09-25
/ / /

We have run into an issue where we have Provisioning Service 6.1 and XenApp 6.5 working together. After we update the vDisk (say, for Windows Update) we run through a script that does things like the “XenApp Prep” to allow the XenApp 6.5 ready for imaging. It appears that there is a bug in the XenApp Prep that sometimes causes it to not fully get XenApp 6.5 ready for rejoining the farm. The initial symptoms I found were:

Event ID 4003
“The Citrix Independent Management Architecture service is exiting. The XenApp Server Configuration tool has not been run on this server.”

I found this CTX article about it, but nothing of it was applicable.

I did procmon traces and I found the following registry keys were missing on the bad system:


A broken system missing the Status Registry key


A working system with the Status key. Note Joined is “0”

After adding the Status Registry key:

I tried restarting the service and the progress bar got further but then still quit. Procmon showed me this:

That is ACCESS DENIED when trying to see that registry key. It turns out that the IMAService does not have appropriate permissions to read this key. The Magic Permissions on a working box and what you need to set here looks like this:

Notice that none of the permissions are inherited and “NETWORK SERVICE” is added with full control to this key. Now when we try and start the Citrix Independent Management Architecture service we get the following errors:

To correct these errors the local host cache needs to be rebuilt. To fix that we need to run:
Dsmaint recreatelhc
Dsmaint recreaterade

After doing that, we can start the IMA Service and MFCOM. If instead of IMA Service starting you get the following error message:

Ensure the following registry is populated:
Read More

Registry keys needed to set a default server Exchange server for Outlook

2012-02-16
/ / /

We recently upgraded our Exchange infrastructure from 2007 to 2010. During this we changed the host name of the server from an internal name to a nice, easy to remember one (outlook.company.corp). But, we did not update our Outlook’s defaults to this server, so when you open Outlook for the first time you are presented with this message:

Microsoft Exchange is unavailable.

With the options:
Retry, Work Offline, and Cancel. If you choose Work Offline you are given this error message:

“—————————
Microsoft Office Outlook
—————————
Outlook cannot log on. Verify you are connected to the network and are using the proper server and mailbox name. The connection to Microsoft Exchange is unavailable. Outlook must be online or connected to complete this action.
—————————
OK
—————————

then the opportunity to enter the server name to connect to:

If you correct the name here you will get an error where you can close Outlook then reopen it for it to operate. Another option is to change the default of that server to the correct one. I figured out the registry keys to do those so you can place in a GPO object.

Here are the keys:

The last key is a REG_BINARY of the server name (outlook.company.corp). If we make this into a GPO object then these keys can be placed in and users can connect to Exchange without the messages above.

The GPO object needs to look like so:

Notice the %LogonUser% variable.

Each “Key Path” requires your username substituted for the %LogonUser% variable as well:

Set this GPO with a loopback processing setting and you’re rolling. The negative that I’ve seen with this approach is that it will set the registry keys on a new login, but launching Outlook for the first time will overwrite them with the defaults set in the PRF. If you cancel out and relaunch the registry keys will apply again and the server you specified in them will work.

Or you can setup a DNS Alias, but this was an interesting exercise anyways 🙂

Read More

Cool tool!

2011-10-25
/ / /

Mariano Sergio Cosentino created a script that will convert registry keys into ADMX template files. This is awesome as the alternative to deploying large number of registry keys and values is typically a startup script with regedit.exe /s %regfile%.

http://mscosentino-en.blogspot.com/2010/02/convert-registry-file-to-admx-policy.html

Tool is available here:
http://www.mscosentino.com/desarrollos/reg2admxl/reg_2_admx.vbs

Usage is: CSCRIPT REG_2_ADMXL.vbs registry-file language [name]

I used this tool to create a ADMX template of the following registry key:
KEY_CURRENT_USERSoftwareMicrosoftWindows NTCurrentVersionWindows Messaging Subsystem

We use Microsoft fRX and it utilizes this key to determine your mail prefences if you’re using exchange. If you have the old Office 2000/2003 (IIRC) you should have this key. 2007 and greater now use a different method of storing email account information (apparently). This content is generated by using the “Mail” control panel icon. We used this tool to prestage the server name and a “Windows Messaging Profile” so that when you try to email from fRX you don’t go through a complicated wizard asking for things like “server name”. If you’re organization is like ours, your internal email server name is something users won’t know and won’t be able to guess (eg, 3-digit-company-abbr,3-digit-code-for-prod-or-dev,3-digit-code-for-virtual-or-physical,3-digit-code-for-server-role(eg EXC-exchange),3-digit-code-for-number).

Read More