Group Policy

Group Policy Preferences Registry Extension vs Group Policy Registry Extension

2018-04-11
/ /
in Blog
/

In various discussions I’ve read about the drawbacks of Group Policy Preferences but is it really that bad?

 

…Or is it how you are using it?

 

There are two methods of applying registry keys/values with Group Policy.  The Group Policy Registry Extension is the “traditional” form of applying policies.  Also known as ADM or ADMX policies, when creating GPO’s with this method a binary file, “.pol”, is created.  When policy application occurs this file is read and applied to your registry.  As a binary file, this file is kept small and fast.  Reading and applying the settings should be nearly instant.

The second method of applying registry keys is with Group Policy Preferences (GPP).  This was a “new” method introduced in Windows Server 2008 with the purchase of PolicyMaker by Microsoft.  Group Policy Preferences are much, much more flexible than the traditional form.  There are different ways of applying registry values, including the CRUD model (Create, Replace, Update, Delete), filtering by the way of “Item Level Targeting“, either on an individual value or on a collection.

I’ve seen an organization heavily leverage GPP to great success.  I started to wonder though, what are the performance impacts of using GPP over the traditional method.  This post will explore the differences in the CRUD model and how it compares to the traditional method..

I intend to look the following scenarios:

  1. Creating a registry value
  2. Updating a previous registry value
  3. Removing a registry value

However, GPP has a fourth method, “Replace” and I’ll explore what it does in addition to these 3 methods.

Creating a Registry Value

In this scenario, the registry will be clean and a new value will be created.  I’m going to refer to the Group Policy Registry Extension (AKA, Administrative Templates, ADM/ADX) as the “traditional” method and use the abbreviation GPP for the Group Policy Preferences Registry Extension.

Traditional:

After reading the Registry.Pol from the sysvol, the application of the registry key takes just 3 operations.  RegCreateKey, RegSetValue, and RegCloseKey.

Each one of these operations took around 1-1.1ms, with the caveat that Process Monitor (procmon) consumes some resources capturing this information, slowing it down slightly.

 

GPP:

We can see a new operation “RegQueryValue”.  As described by William Stanek, “The Create action creates a preference if it doesn’t already exist. For example, you can use the Create action to create and set the value of a user environment variable called CurrentOrg on computers where it does not yet exist. If the variable already exists, the value of the variable will not be changed.”

The RegQueryValue is executing the check to see if a variable already exists.  So what does GPP look like if the value is already present?

3 operations with the process exiting on a success on the value being present.

The end result, is 3 operations for our traditional method, and 4 operations for the Group Policy Preferences method for creating a registry entry.

Updating a registry value

In this scenario, the registry will contain a value, and the policy will be updated with a new value.  For the traditional method this will involve changing the Microsoft “User Profiles” policy.  I set the “HomeDir” location to “TrententTest”, applied the value, then updated it to “TrententTye”.  This will ensure a new, changed key is applied.  For GPP I’m going to change the value on the policy to 0x0 from 0x1 and use the “Update” operation.

Traditional:

Traditional maintains a very simple “3 operation” action with updating a value having the same effect as if the value was never present to begin with.

GPP:

With the “Update” action, GPP now executes just 3 operations, same as the traditional.

The end result, is 3 operations for our traditional method, and 3 operations for the Group Policy Preferences method for updating a registry entry.

Removing a registry value

In this scenario, I am going to remove a registry value.  Using the traditional method this means modifying my group policy to “Not Configured”, and for GPP this means setting “Delete” for our operation.

Traditional:

Again, Traditional performs it’s work in just 3 operations.

 

GPP:

GPP also performs this work in just 3 operations.

 

GPP – The Replace Method

Group Policy Preferences has another operation to explore.  “Replace”.

This operation …”creates preferences that don’t yet exist, or deletes and then creates preferences that already exist.”

This sounds like it performs a few operations.  Lets see what it looks like:

Replace executes “6” operations.  RegOpenKey, RegDeleteValue, RegCloseKey, RegCreateKey, RegSetValue, RegCloseKey.  I’m not entirely sure why you’d want a DeleteValue before SetValue but that’s what this selection does.

 

Revisiting GPP: “Creating a Registry Value”

During the process of creating this post, I wondered if the 3 operation “Update” would work better for creating a key.  The GPP “Create” selection has 4 operations, but the “Update” selection only has 3 operations.  I deleted my “TrententTestPreferences” key and refreshed group policy:

 

3 operations!  So Group Policy Preferences has the potential to operate at the same speed as the traditional group policy IF YOU STICK TO USING “UPDATE”.  At the very least, these operations should take the same amount of time.  Of course, implementation might be a different story.

The final tally:

Stay tuned for part 2 — The Performance Comparison

Read More

Group Policy – Monolithic vs Functional Design and Performance Evaluation

2018-04-09
/ /
in Blog
/

Group Policy Design is a hotly discussed topic, with lots of different ideas and discussions.  However, there is not a whole lot of actual metrics.  ADM and ADMX templates apply registry keys in an ‘enforced’ manner.  That is, if you or the machine has access to read the policies, the registry keys within are applied.  If you stuck purely to ADM/ADMX policies but wanted to do dynamic filtering or application of the keys/values based on a set of criteria you’d probably design multiple policies and nested organizational units (OUs).  From here, you could filter certain policies based on the machine or user location in the OU structure or by filtering on the policies themselves and denying access to certain groups or doing explicit allows to certain groups.  This design style, back in the day, was called “functional” design.

 

However, the alternative style, “monolithic” design, simplifies the group policy object (GPO) design into much fewer GPO’s.

 

Setup

My test setup is very simple; a organizational unit (OU) with inheritance blocked to control the application of the GPO’s.  I created 100 individual GPO’s with a single registry value, and 1 GPO with 100 values.  I chose to do a simple registry addition as it should be the best performance option for group policy.  I created a custom ADMX file for this purpose:

Monolithic simulation:

 

Functional simulation:

 

Testing

In testing these two designs I elected to focus on the one factor that would have the most impact: latency.  I setup my client machine in the OU, put a WAN emulator that can manipulate latency and measured the performance between the functional and monolithic designs at varying latencies.  I looked for the following event ID’s: 4257, 5257, 4016, 5016.  The x257 events correspond to when group policy downloads the group policy objects off the SYSVOL file share.  The x016 event’s determine how long it took the policy to be processed.

 

The results:

 

Raw Data:

Functional GPO – applying 100 registry values
Event ID 4016 to 5016 (ms)
Latency Time (ms)
0 271
10 4089
25 8078
50 15315
75 22904
100 29820
Event ID 4257 to 5257 – Starting to download policies
Latency Time (s)
0 0
10 3
25 6
50 12
75 17
100 22

 

Monolithic GPO – applying 100 registry values
Event ID 4016 to 5016 (ms)
Latency Time (ms)
0 117
10 156
25 198
50 284
75 336
100 435
Event ID 4257 to 5257 – Starting to download policies
Latency Time (s)
0 0
10 0
25 1
50 1
75 1
100 1

 

Analysis:

There is a huge edge to the monolithic design.  Both in terms of how long it takes to process a single policy vs multiple policies and the resiliency to the effects of latency.  Even ‘light’ latency can have a profound impact on performance.  Going from 0ms to 10ms increased the length of time to process a functional design by 15 times!  The monolithic design, on the other hand, was barely impacted.  Even with a latency of 100ms, it only added ~300ms of time to process the policy.  I would consider this imperceptible in the real world, where as the functional design going from ~271ms to ~4000ms would be an extremely noticeable impact!  Let alone about 30 seconds at 100ms!

Another factor is how much additional time is required to download the policies.  This is time in addition to the processing time.  This probably shouldn’t be a huge surprise, it appears that group policies are downloaded and processed sequentially, in an order.  I’m sure this is necessary to maintain some semblance of prediction if you have conflicting policies settings, the one last in the list (whether sorted alphabetically or what have you) can be relied on to be the winner.

Adding latency, even just a little latency, has a noticeable impact.  And the more policies, the more traffic, the more the impact of latency.  Again, a loss for a functional design and a win for a more monolithic design.

Conclusion:

Group Policy Objects can have a large impact on user experience.  The goal should be to minimize them to as few as possible.  As with everything, there are exceptions to the rules, but for Group Policy it’s important to try and maintain this rule.  Even just a little latency between the domain controller and the client can have a massive impact in group policy performance.  This can impact the length of time it takes a machine to boot, to delaying a user logging into a system.

 

Read More

Citrix Provisioning Server – PXE requests stop working

2017-08-10
/ /
in Blog
/

We use a bootable ISO in our environment to boot our VM’s to a specific set of PVS servers.  This ISO will vary by region enforcing that each target device that boots will be directed to their closest PVS server.

However, we have 1 region that does not leverage this capability and this region was designed to utilize the PXE services of the Citrix PVS server’s.  Occasionally, we encountered VM’s that will not boot and instead the console shows “PXE-E53: no boot filename received”

 

When I logged onto the Citrix PVS servers, I checked their services.  Both services were reported as “Running”:

When I checked the event logs I did not see any errors in either the application log or the system log.  Administrative events showed nothing out the usual either.

In order to confirm that the PVS service was actually listening, I executed

this showed me all the open ports the server was listening for and the processes tied to those ports.  Since PXE is a UDP operation, I examined the UDP portion of the netstat output.  

Port 69 is used by TFTP to transfer files, and port 67 is used by PXE.  However, I only saw port 69, port 67 was no where to be found.  I restarted the “Citrix PVS PXE Service”, reran netstat and confirmed that the PXE port was not listening and matched up the process ID to the proper services.

Restarting the failed target devices and they began to boot properly.

However, why did this fail in the first place?  I read on the Citrix forums that the Citrix services can become unbound if the network is not available when the services are started.  To test this I rebooted one the affected Citrix PVS servers.  Sure enough, it came back up with port 67 not being monitored but the service in a ‘Running’ state.  I wanted to see if I could capture the flow of communication from the network and when the service started so I used procmon and enabled “Boot Logging”.

Lo and behold, the procmon monitoring on startup added enough of a delay that the PXE service was bound consistently.  Stopping the boot logging and the PXE service would start but fail to bind to the port.

So now this leads to a bit of a quandary.  The delay seems to be in the milliseconds.  I’ve considered a couple solutions for this issue.

  1. A startup script that checks to ensure both ports and restarts the proper service if one of the ports is not found.
  2. Change the service startup type to be “Automatic (Delayed Start)”.  This delays the service by up to 2 minutes.  This does mean that the PVS server will NOT be able to service target device boot requests during this window.

I think we’re going to go with option 2.  The reason is we can apply this setting change via Group Policy Preferences.  This ensures that if we any removal/upgrade of the PVS software this setting will get reapplied.  And then we don’t have to worry about upgrading the OS and losing the startup script either or maintaining a script.

We’ve been affected by this a few times in the past, the fix has always been to restart the PVS server, but I managed to hit a window where the failure was happening consistently and managed to get this information.  🙂

 

Read More

Citrix Workspace Environment Manager – Limitations (part 1?)

2017-04-03
/ /
in Blog
/

In a brief evaluation of Citrix Workspace Environment Manager, I looked at the utility of the product to replace Group Policy Preferences (GPP) in a XenApp environment context.  For this I focused on replacing a set of registry keys we apply via GPP for our XenApp environment.  My results were not favorable for using WEM in this context for the Registry portion as WEM pushes processing of it’s entries into the ‘shell’ session.  For XenApp, the Shell session is typically applied quickly and so the application may launch without those keys present (which is bad — the application needs those keys present first).  So although logon times maybe reduced, this scenario does not work for the Registry portion.  We are still exploring the effects of WEM and whether some other components that operate synchronously within GPP are needed.  Can these components be moved to WEM?  One of the big ‘wins’ for this approach maybe Drive Mappings, which apply synchronously and requires the Drive Mappings to be processed before allowing a user to logon.  Moving this to WEM may be a win worth exploring… IF the application doesn’t require drive mappings before being launched.  But that’s for another article..

However, for the registry portion of WEM we did encounter a few ‘gotcha’s worth mentioning if you are going to use WEM.

WEM does not do ‘Registry Binary’ keys.

Well…  it says it does.  And it kind of does.  But odds are you are not going to get the results you expect.

Looking at a simple REG_BINARY key it contains data is displayed as ‘hexadecimal’ data.

If I want to use WEM to apply this key, I would create an entry within WEM:

HINT: it's not.

Does this look correct to you? I thought it looked correct to me.

 

However, when I apply this key I get a completely wrong result.  Why?  Because WEM is applying the hexadecimal code in the ASCII field of the REG_BINARY

In order to get WEM to apply the code properly we would need to copy/convert the ASCII from the REG_BINARY.

 

However, I have bad results doing so:

WEM with ASCII converted binary values applied.

 

Result:

This is closer but grossly wrong.

 

Why is this wrong?  WEM stores everything as XML.  And XML files do not like storing binary or non-ascii data.  WEM stores these values as their ASCII representations and not REG_BINARY representations so if your REG_BINARY (which, there’s a 99% chance) contains a non-ASCII character it will fail to apply the key properly.

 

Even worse, during my time fiddling with this, I BROKE WEM.

If the ASCII representation of “&#x1” or whatever was set it caused WEM to crash when applying the values.  So REG_BINARY’s are completely out.  In my limited testing, we only had 1 REG_BINARY to apply, but in our environment we use GPP to apply 5 different REG_BINARY keys.  So using WEM for these applications is right out.  I filed and asked Citrix for a ‘feature enhancement’ to support applying REG_BINARY’s properly, but I was told this was operating as expected so I’m not holding my breathe but this does limit the utility of WEM.

 

 

Read More

Citrix Workspace Environment Manager – First Impressions

2017-03-02
/ /
in Blog
/

Citrix Workspace Environment Manager can be used as a replacement for Active Directory (AD) Group Policy Preferences (GPP).  It does not deal with machine policies, however.  Because of this AD Group Policy Objects (GPO) are still required to apply policies to machines.  However, WEM’s goal isn’t to manipulate machine policies but to improve user logon times by replacing the user policy of an AD GPO.  A GPO has two different engines to apply settings.  A Registry Policy engine and the engine that drives “Client Side Extensions” (CSE).  The biggest time consumer of a GPO is processing the logic of a CSE or the action of the CSE.  I’ll look at each engine and what they mean for WEM.

Registry Extension

The first is the ‘Registry’ policy engine.  This engine is confusingly called the “Registry Extension” as opposed to the CSE “Group Policy Registry”.  The “Registry Extension” is engine applies registry settings set in via ‘Administrative Templates’.

These settings are ‘dumb’ in that there is no logic processing required.  When set to Enabled or Disabled whatever key needs to be set with that value is applied immediately.  Processing of this engine occurs very, very fast so migrating these policy settings would have minimal or no improvement to logon times (unless you have a ton of GPO’s apply and network latency becomes your primary blocker).

If you use ControlUp to Analyze GPO Extension Load Times it will display the Registry Extension and the Group Policy Registry CSE:

Client Side Extensions

However, CSE’s allow you to put complex logic and actions within that require processing to determine if a setting should be applied or how a settings should be applied.  One of the most powerful of these is the Registry CSE.  This CSE allows you to apply registry settings with Boolean logic and can be filtered on a huge number of variables.

All of this logic is stored in a XML document that is pulled when the group policy object is processed.  This file is located in “C:\ProgramData\Microsoft\Group Policy\History\GUID OF GPO\SID OF USER\Preferences\Registry”.

Parsing and executing the Boolean logic takes time.  This is where we would be hopeful that WEM can make this faster.  The processing of all this, in our existing environment consumes the majority of our logon time:

Migrating Group Policy Preferences to WEM

Looking at some of our Registry Preferences we’ll look at what is required to migrate it into WEM.

Basic settings “eg, ‘Always applied’”.

“Visual Effects”

These settings have no filters and are applied to all users.  To migrate them to WEM I’ve exported these values and set them into a registry file:

Switching to WEM I select ‘Actions’ then ‘Registry Entries’ and then I imported the registry file.

An interesting side note, it appears the import excluded the REG_BINARY.  However you can create the REG_BINARY via the GUI:

To set the Registry Entries I created a filter condition called “Always True”

And then created a rule “Always True”

We have a user group that encompasses all of our Citrix users upon which I added in ‘Configure Users’.  Then, during the assignment of the registry keys I selected the ‘Always True’ filter:

And now these registry keys have been migrated to WEM.  It would be nice to ‘Group’ these keys like you can do for a collection in Group Policy Preferences.  Without the ability to do so the name of the action becomes really important as it’s the only way you can filter:

Next I’ll look at replacing Group Policy Preferences that contain some boolean logic.

Read More

AppV5 – Applications being published to “C:\App-V Packages” instead of %PACKAGEINSTALLATIONROOT%

2016-09-21
/ /
in Blog
/

The last little while we’ve had an issue with some applications seemingly being published to “C:\App-V Packages” instead of our defined path “D:\AppVData\PackageInstallationRoot”.

AppV-Packages

Notice the timestamps? Packages published during the same ‘publishing refresh’

 

Video of the issue in action that I was able to setup a repro:

So what’s going on here that would cause this to happen?

I turned to the event viewer to find the time when this package “8FE4C99E…” was published and just looked to see if there was anything surrounding it.

Highlighted is the package publish event

This is the event showing the package being published to “C:\App-V Packages”.  There is an odd thing different about this publish event compared to the rest (that you can see above).  Between ‘Configure Package’ and when the ‘Streaming’ starts there is a ‘Publishing Refresh’ event.

Publishing refresh

Why is there a Publishing Refresh event?  It turns out, it can be triggered by a group policy refresh.

GP Refresh

Notice the time stamps?

So, is there something that occurs during Group Policy refresh that could cause this?

To test this I used procmon to monitor for PackageInstallationRoot registry key and ran a Group Policy Refresh:

screen-shot-2016-09-21-at-8-27-01-am

Procmon_PackageInstallationRoot

The key is deleted then recreated and inbetween the AppV service is trying to read it.

So what is happening here?

At our organization we prefer to set Group Policies using Group Policies Preferences – Registry because of the power and flexibility of Item Level Targetting.  We prefer to only apply registry keys/policies to specific systems in certain groups rather than having a complicated OU structure.  Since I know we have a policy that configures our AppV5 values I went and looked into how this value was being configured:

GroupPolicyManagement

The ‘Action’ is ‘Replace’

screen-shot-2016-09-21-at-8-44-18-am

We have this value set to ‘Replace‘.  What does replace do?

Replace Delete and recreate a registry value or key for computers or users. If the target is a registry value, the net result of the Replace action is to overwrite all existing settings associated with the registry value. If the target is a registry key, the net result is to delete all values and subkeys in the key, leaving only a default value name with no data. If the registry value or key does not exist, then the Replace action creates a new registry value or key.

Well, process monitor is showing EXACTLY that scenario.  Replace is deleting the key and recreating it.  In between that time ‘AppVClient.exe’ is trying to read that value.  If PackageInstallationRoot doesn’t exist, then AppV defaults to ‘C:\App-V Packages” as the PackageInstallationRoot.

screen-shot-2016-09-21-at-10-01-23-am

No “PackageInstallationRoot” key -> “C:\App-V Packages” folder is where packages go.

 

It appears we have a race condition between group policy being applied and when our AppVClient is refreshing.  It is less than 600ms but that’s enough time for a package or two to start their refresh and get caught in ‘default folder’ purgatory.

How can we fix this?

The easiest solution and the one we are going to implement:

We’re going to change the action to “Update”.  When I initially tried to change the action was greyed out.  When “Remove this item when it is no longer applied” is checked, it forces the action to “Replace”.
screen-shot-2016-09-21-at-8-44-27-am

I had to ‘uncheck’ that value

screen-shot-2016-09-21-at-8-44-34-am

And then I could select ‘Update’.

screen-shot-2016-09-21-at-8-44-43-am

After updating the GPO and refreshing policies on the affected system, I ran procmon to verify that it does not delete the value anymore:

GPP_Update

It does not delete the value, it now ‘Sets’ the value to the same string we had before.  I don’t know if the individual ‘Set’ action will cause an issue; I suspect not since the value will always exist and is technically the same value throughout the entire set of actions that this issue is now ‘resolved’ for us.

Read More

About the XenApp 6.5 Group Policy Client Side Extensions (CSE)

2016-08-02
/ /
in Blog
/

TLDR; using a newer Citrix Group Policy Management (GPM) than 1.7.X on XenApp 6.5 will cause your policies to disappear if you upgrade your Client Side Extensions to a version higher than 1.7.6.

The Citrix Client Side Extension (CSE) are the ‘Citrix Group Policy Engine’.  The CSE takes whatever policies you set through Active Directory, locally or the AppCenter console and apply them to your server or Citrix session (if a user policy).  There is some oddity with the CSE and ‘Citrix Group Policy Management’ portions of the Citrix products.  You see, they are interoperable, but in certain scenario’s they are not.

The split appears to be for the XenApp 6.5 product for CSE 1.7.6+.  My Citrix TRM informed me that the Active Directory Group Policy Schema changed for CSE 1.7.6+.  If you intend to use CSE 1.7.6+ you will need to upgrade your Group Policy Mangement (GPM) to 1.7.11.  To upgrade your AD policy seems simple enough.  Citrix says open the policy on a computer with GPM 1.7.11 and then close it and it will become updated.

But here’s a bit of the rub.  Citrix supports and encourages “some” mixing and matching of some components.  Specifically, the Citrix Universal Print Server (UPS) and Client (UPC).

And here’s my story:

We wanted to use the 7.6 version of UPS/UPC as it had some improvements we deemed critical.  We had not upgraded the version of the Citrix GPM/CSE from what came with the Citrix XenApp 6.5 (1.5.0).  When we downloaded UPS/UPC 7.6 we found we could not configure the Group Policy  settings for the Universal Print Client… Until we upgraded the GPM that came with XenApp 7.6.  Then the UPM policies appeared, ready to configure.  The version of GPM included with XA7.6 was 2.2.0.

Only on reboot, with the policies set, we found they were still not applying.  At this time, I found you need version 1.7.0 of the CSE to recognize the new policies.  We installed CSE 1.7.0 and it recognized all the policies and we were flying.

Fast forward a year or so later and we decided to ‘get up to date’ with our operational software.  Essentially, we wanted to ensure we had all the bug fixes enhancements of all of the latest and greatest for XA6.5 so we can survive for the next couple years while we transition to whatever Citrix will have out by then.  So the latest and greatest CSE is 1.7.6+ and I installed it, and all my policies went poof.  This prompted my earlier post.

During the course of troubleshooting my issue I installed various versions of the CSE’s and GPM’s that came with the various versions of Citrix XenApp.  Since we had GPM 2.2.0 installed, nothing from the 1.7.6 CSE branch recognized any of the policies.  BUT, installing any of the CSE’s from XA7.5+ recognized and applied the 6.5 policies and everything on top of that.  So I started asking our Citrix TRM if it was supported to have the CSE from the newer XenApp 7+ on 6.5 and if they included all the policies.  The answer was ‘Maybe it works, probably not supported’.  So I asked why the policies of 2.2.0+ don’t work with CSE 1.7.6 and the answer I got was the schema changed for the GPO’s.  This is implied in CTX202233:

Note: This fix addresses the issue for AD policies you create after installing this update. It also addresses it for existing policies where Citrix settings were configured before Microsoft settings. It does not address it for existing AD policies where Microsoft settings were configured before Citrix settings. For those AD policies, you must open the affected policies and save the Citrix settings.

Opening and saving the policies updates the schema.

 

Read More

Citrix XenApp – Graphical Artifacts

2016-07-22
/ /
in Blog
/

In our Citrix XenApp 6.5 environment we started having a couple applications encounter an issue where they would experience some serious graphical artifacts.  What was supposed to look like this:

1

Would look like this:

2

 

Here’s a short video demonstrating this issue:

Or sometimes it would show the windows *behind* the artifacted image.  That is, instead of the ‘White’ you see in my image, the application behind it shows through.

When investigating this we found there was a couple symptoms that we were going to experience these artifacts.

  1. The window would become ‘frosted’ or ‘ghosted’ (as seen in Spy++ or AutoIt Window Info)
    4
  2. The application would switch to ‘Not Responding’
    3
  3. If you completed the task ‘Edit’ quickly there would be no artifacting (time was important)
  4. When ‘timing’ the switch from ‘normal’ to frosted or ghosted window it would be around 5-7 seconds.

So what’s going on here?

For this particular instance, the application is launching MSPaint with some modified properties.  It sets Paint to ‘Always On Top’, which in itself isn’t an issue, but then it purposefully locks the UI so you must complete the drawing and close paint before continuing.  This is how the vendor designed this application to operate with this workflow.

And what’s Windows doing?  It turns out, Windows is trying to alert you that your program is non-responsive!  Windows has a built in feature to ‘Frost/Ghost’ the window of a non-responsive UI to prevent you from entering input that won’t be received.  The ghosting effect is time sensitive!  So that explains why if we opened and closed our document quickly their would be no artifacting but if we manipulated it for some time the artifacts would appear.  The time limit for monitoring unresponsiveness is 5 seconds.  DWM.exe is the process responsible for creating the ‘Frost’ window and when responsive returns, it appears it does a poor job telling the application to repaint all affected Windows.

Microsoft recommends a couple ‘fixes’ which is really a programmatic way to ‘disable’ the ghosting feature.  The two methods Microsoft suggests is to create a NoGhost application compatibility fix or have the programmer use ‘DisableProcessWindowsGhosting’.  But there is a 3rd method.

The 5 second time limit is programmable.  If we extend the timeout we don’t need to configure ‘NoGhost’ compatibility fixes for each app or go back to the vendor.  The timer is global and affects every application and window.  Unfortunately, I know of no way to permanently disable it, but we can set a high enough value to prevent it from appearing.

So what do we have to do to ‘resolve’ this?

My preferred choice was to use Group Policy Preferences (Registry) and set a new value for HKCU\Control Panel\Desktop /v HungAppTimeout /t REG_SZ /d 120000

This sets the timeout to 2 minutes as opposed to 5 seconds.  Now when our program is used we get this result:

No More Artifacts.

When I was investigating this I found I could get the artifacting to occur in both ICA and RDP but not when on the console.  The frustrating thing about this issue is that it was not consistent.  Because of the 5 second default time limit, the program(s) we had that would ‘artifact’ would sometimes complete their UI locking job faster than 5 seconds, but sometimes not.  This lead to reports of artifacting occurring more often ‘during the peak work hours’ when the application/server/user load was the most.  This makes sense as the higher load undoubtedly lead to everything being slower, the database, server, etc. leading to the application waiting longer and thus exceeding the timeout.  I did find through the course of troubleshooting this issue that it seemed to ‘go away’ when I was trying to replicate after hours, which is frustrating to only have a slice of time to try and resolve this during peak hours.

Fortunately, after implementing the HungAppTimeout registry key the artifacts for several application ‘went away’.

Lastly, contrary to this article you do NOT need to restart for this value to take effect.  WinLogon.exe reads the HungAppTimeout value and then configures DWM accordingly when your profile loads.  So for this value to take effect you only need to log on with this value already residing in your user’s registry hive.

5

Read More

Citrix Client Side Extensions 1.7.6+ breaks policies

2016-07-14
/ /
in Blog
/

We apply our several group policy settings via Active Directory group policy objects, within them we set ‘Citrix’ policies.  This includes things like ‘Licensing’, platform, etc.

AD_Policies

 

When we upgraded the Citrix Group Policy Client Side Extensions (x86) to version 1.7.6 or 1.7.7 we found that these values were no longer being applied.  FYI!

 

 

Read More

Multi-homed server, lock ICA and RDP to a specific NIC

2016-04-21
/ /
in Blog
/

We implement a multi-homed setup with Citrix Provisioning Services.  We have all of our production traffic on one NIC and all PVS traffic on a second nic.  This helps us in troubleshooting when doing packet captures but does introduce other sets of challenges.  One of these challenges is when I uninstalled and then installed updated VMWare tools on one of our vDisks it caused the NIC’s to renumber and reorder themselves.  You’ve probably read some articles saying to ‘show hidden devices’ and uninstall any ‘ghost’ devices; with a multi-homed setup this may not resolve your issue.  My specific issue is my NIC’s now went from “0x1 and 0x2” to “0x3 and 0x4” in the LANATABLE.  We apply a GPO to the ICA-TCP and RDP-TCP to force them to only utilize our ‘Production’ NIC which we decided was going to be the second NIC.

Provision = 1st Nic, Production = 2nd Nic

I uninstalled the ghost devices but because this change is all ‘in the registry’ it wasn’t immediately noticeable by myself that the LANATABLE and NIC ordering had changed.  I promoted my vDisk and then tried to RDP into it:

Well…  I knew this wasn’t right.  So I logged onto the server and checked the LANATABLE values:

Provision NIC
Production NIC

 

Provision NIC LanaID = 0x3 after the VMWare Tools upgrade

 

Production NIC LanaID = 0x4 after the VMWare Tools upgrade

 

RDP targets LanAdapter 0x1 (!?)

 

ICA targets LanAdapter 0x2

A couple issues popped out.   The first was that the LanaID’s were wrong.  I *thought* Provision should be #1 and Production should be #2, but we apply our LanAdapter ID’s via GPP so these values are correct for our other systems.  I know both RDP and ICA need to be locked to the Production NIC and the order is *correct* but I’m a bit confused to why the numbers are different between RDP-TCP and ICA-TCP.

So I started on getting the issues resolved.  First, I was going to resolve RDP.  If it is targeting 0x1 that means that I need the Production NIC (VMWare Network Adapter #2) needs to be set as 0x1.  So I edit the LanaId of the Production NIC to 0x1 and the Provision NIC to 0x2.  I rebooted the box and I could RDP into it without issue.  I then checked the Remote Desktop Session Host Configuration:

This is the correct value.

OK, so RDP is set correctly and works.  I then tried to launch a Citrix application.

It failed.

It generated an event on the server:

Unable to connect to the CGP tunnel destination

Looking at the ICA Listener configuration showed me the following:

The value is wrong.  It should be #2

So the ICA-TCP listener was set to the ‘Provision’ NIC.  So our production traffic was not getting to it. This is the wrong value.  My first thought was the LANATABLE would make sense here…  We have the LANADAPTER key is set to 0x2 which would equal the ‘Provision’ NIC under this configuraiton…   So I changed the LANATABLE to be the reverse.  0x1 = Provision NIC and 0x2 = Production NIC.

The results:

Now both are wrong!

Jeez.
I reverted the LANATABLE to 0x1 = Production and 0x2 = Provision.  Again, only the RDP-TCP connection changed.  At this point the ICA Listener *must* be looking at another place…  I used Procmon to trace the registry when I opened the ICA Listener Configuration and noticed it did NOT query LANATABLE but did go through and look and query this key:
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesTcpipLinkage
The BIND order is set as Production (#2 NIC) first and Provision second in this order…  To change this order you need to go to Advanced Settings and modify the connection order:
Provision *should* be the top NIC here…
So I used the arrows on the side and moved Production to be second in this order:
Rebooted and I checked the ICA Listener:
Hurray!
I then tried both RDP and ICA traffic and both now worked correctly.
So, the lesson learned here is that this registry key:
Targets the BINDING order of the NIC’s.
And this registry key:

Targets the LANATABLE and the values specified within.

Read More
Page 1 of 3123