Using ControlUp to launch a Citrix application published on a server

/ /
in Blog

Occasionally, we have Citrix servers that ‘die’ in a peculiar way.  What happens may vary when they die but the usual symptoms are something like:

  1. The server is still somewhat responsive.  It responds to pings, RPC requests (tasklist /s %servername%)
  2. The server is not responsive.  You cannot RDP to it, console CTRL-ALT-DEL fails, etc.

This is frustrating because the services appear to be operating so the Citrix server will say, “hey, I’m working!  I can take sessions!”  And usually these servers won’t have any sessions because logons actually fail so their “XenApp Server Load” is low, so its priority for sessions to be directed to it is higher!  So how do we detect these servers with these issues?  Unfortunately, I haven’t seen any events in the Event Viewer or anything that stands out to search and find these troublesome servers.  Using ControlUp, sometimes it’s obvious because that troublesome server will have a much lower session count than other servers or something else is at fault and triggers the ‘Stress Level’ to go critical.  But these flags don’t usually exist if the problem has just occurred, they usually are more visible after time has passed.

Our helpdesk asked if there was a way they could test servers to help pinpoint a troublesome one.  I came up with a “Script-based Action” that targets a specific Citrix server and lists all published applications on that server.  You then select the application and it generates a ICA file and tries to launch it.  You need to have permission to the application on Citrix and Powershell remoting enabled on the XenApp servers/ZDC’s .  So if your a Citrix admin and PS Remoting is enabled this script will work out of the box.

However, I tried to make the script dynamic so you could query the XenApp servers from a standalone server without installing Citrix Powershell SDK locally.  To do this I use PowerShell remoting so you need to have PowerShell remoting enabled on your Citrix servers in your environment.  Secondly, if you have ‘lower’ privilege users you need to grant them the ability to connect to the servers via PowerShell remoting (by default only Administrators have access).  To grant them access you need to do the following:


And in the ‘Set-PSSessionConfiguration’ command you need to enable the ‘Invoke’ permissions on your support group:
As well, you need to grant view properties on your Citrix farm since the group needs to query application properties, and workergroups (if you publish your applications to workergroups):


Now that we have our permissions configured we can create our ControlUp Script-Based action:


So what does this look like?

And the script:


Read More

Using VMWare Remote Console with ControlUp

/ /
in Blog

I wanted to connect to the console session of some of our VM’s but ControlUp doesn’t have a native way of doing so.  Enter Script-Based-Actions and the ability to create those features!  Here is a video of it in action:

VMWare Remote Console on ControlUp

We use multiple individual vCenter servers so I have a list of them I need to connect to in order to find the VM and get the required data.  This takes a bit longer but is still faster than running 6 different vCenter consoles.  You will need to modify the vCenter list in my script and add your own:


Read More

About the XenApp 6.5 Group Policy Client Side Extensions (CSE)

/ /
in Blog

TLDR; using a newer Citrix Group Policy Management (GPM) than 1.7.X on XenApp 6.5 will cause your policies to disappear if you upgrade your Client Side Extensions to a version higher than 1.7.6.

The Citrix Client Side Extension (CSE) are the ‘Citrix Group Policy Engine’.  The CSE takes whatever policies you set through Active Directory, locally or the AppCenter console and apply them to your server or Citrix session (if a user policy).  There is some oddity with the CSE and ‘Citrix Group Policy Management’ portions of the Citrix products.  You see, they are interoperable, but in certain scenario’s they are not.

The split appears to be for the XenApp 6.5 product for CSE 1.7.6+.  My Citrix TRM informed me that the Active Directory Group Policy Schema changed for CSE 1.7.6+.  If you intend to use CSE 1.7.6+ you will need to upgrade your Group Policy Mangement (GPM) to 1.7.11.  To upgrade your AD policy seems simple enough.  Citrix says open the policy on a computer with GPM 1.7.11 and then close it and it will become updated.

But here’s a bit of the rub.  Citrix supports and encourages “some” mixing and matching of some components.  Specifically, the Citrix Universal Print Server (UPS) and Client (UPC).

And here’s my story:

We wanted to use the 7.6 version of UPS/UPC as it had some improvements we deemed critical.  We had not upgraded the version of the Citrix GPM/CSE from what came with the Citrix XenApp 6.5 (1.5.0).  When we downloaded UPS/UPC 7.6 we found we could not configure the Group Policy  settings for the Universal Print Client… Until we upgraded the GPM that came with XenApp 7.6.  Then the UPM policies appeared, ready to configure.  The version of GPM included with XA7.6 was 2.2.0.

Only on reboot, with the policies set, we found they were still not applying.  At this time, I found you need version 1.7.0 of the CSE to recognize the new policies.  We installed CSE 1.7.0 and it recognized all the policies and we were flying.

Fast forward a year or so later and we decided to ‘get up to date’ with our operational software.  Essentially, we wanted to ensure we had all the bug fixes enhancements of all of the latest and greatest for XA6.5 so we can survive for the next couple years while we transition to whatever Citrix will have out by then.  So the latest and greatest CSE is 1.7.6+ and I installed it, and all my policies went poof.  This prompted my earlier post.

During the course of troubleshooting my issue I installed various versions of the CSE’s and GPM’s that came with the various versions of Citrix XenApp.  Since we had GPM 2.2.0 installed, nothing from the 1.7.6 CSE branch recognized any of the policies.  BUT, installing any of the CSE’s from XA7.5+ recognized and applied the 6.5 policies and everything on top of that.  So I started asking our Citrix TRM if it was supported to have the CSE from the newer XenApp 7+ on 6.5 and if they included all the policies.  The answer was ‘Maybe it works, probably not supported’.  So I asked why the policies of 2.2.0+ don’t work with CSE 1.7.6 and the answer I got was the schema changed for the GPO’s.  This is implied in CTX202233:

Note: This fix addresses the issue for AD policies you create after installing this update. It also addresses it for existing policies where Citrix settings were configured before Microsoft settings. It does not address it for existing AD policies where Microsoft settings were configured before Citrix settings. For those AD policies, you must open the affected policies and save the Citrix settings.

Opening and saving the policies updates the schema.


Read More

Microsoft Patch (KB3170455) breaks PrintBRM importing printers with drivers

/ /
in Blog

We have an application (ARIA MO) that has some special requirements.  The application requires all printers used by it in the different locations to be manually loaded on the Citrix server with all drivers.  This totals around 220 or some odd printers installed with around 15 different drivers loaded.  The printers must have a local port and cannot be mapped via TCPIP or network mapping.

Our design goals for our Citrix environment are to minimize the various PVS images we use so we use various ‘layers’ to allow a single master image to be able to host various unique and difficult configurations.  For this application we use AppV as our layering technology to put the application on the server, but for the printers we use a script to load them onto the server.  What we have is a print server that hosts all the printers needed by this application and we can export the printers into a file using Print Management.  Then we save that file on a network share somewhere.  When the Citrix server boots, I can take that file and manipulate its contents to change the queues to ‘local’ queues then import that modified file to the Citrix server.  I configured the script to take two parameters, a print file name and a server name.  I call the script with a command line like this:

The powershell command it calls is here:

So, what does this have to do with KB3170455?  Well, since installing KB3170455 it prevents importing printer files with print drivers embedded.  This is what it looks like with KB3170455 installed:

Screen Shot 2016-07-29 at 9.04.48 AM

Screen Shot 2016-07-29 at 9.07.35 AM

And the failure import with 3170455 installed:

Remove KB3170455 and the import works without issue.

Read More

Citrix XenApp – Graphical Artifacts

/ /
in Blog

In our Citrix XenApp 6.5 environment we started having a couple applications encounter an issue where they would experience some serious graphical artifacts.  What was supposed to look like this:


Would look like this:



Here’s a short video demonstrating this issue:

Or sometimes it would show the windows *behind* the artifacted image.  That is, instead of the ‘White’ you see in my image, the application behind it shows through.

When investigating this we found there was a couple symptoms that we were going to experience these artifacts.

  1. The window would become ‘frosted’ or ‘ghosted’ (as seen in Spy++ or AutoIt Window Info)
  2. The application would switch to ‘Not Responding’
  3. If you completed the task ‘Edit’ quickly there would be no artifacting (time was important)
  4. When ‘timing’ the switch from ‘normal’ to frosted or ghosted window it would be around 5-7 seconds.

So what’s going on here?

For this particular instance, the application is launching MSPaint with some modified properties.  It sets Paint to ‘Always On Top’, which in itself isn’t an issue, but then it purposefully locks the UI so you must complete the drawing and close paint before continuing.  This is how the vendor designed this application to operate with this workflow.

And what’s Windows doing?  It turns out, Windows is trying to alert you that your program is non-responsive!  Windows has a built in feature to ‘Frost/Ghost’ the window of a non-responsive UI to prevent you from entering input that won’t be received.  The ghosting effect is time sensitive!  So that explains why if we opened and closed our document quickly their would be no artifacting but if we manipulated it for some time the artifacts would appear.  The time limit for monitoring unresponsiveness is 5 seconds.  DWM.exe is the process responsible for creating the ‘Frost’ window and when responsive returns, it appears it does a poor job telling the application to repaint all affected Windows.

Microsoft recommends a couple ‘fixes’ which is really a programmatic way to ‘disable’ the ghosting feature.  The two methods Microsoft suggests is to create a NoGhost application compatibility fix or have the programmer use ‘DisableProcessWindowsGhosting’.  But there is a 3rd method.

The 5 second time limit is programmable.  If we extend the timeout we don’t need to configure ‘NoGhost’ compatibility fixes for each app or go back to the vendor.  The timer is global and affects every application and window.  Unfortunately, I know of no way to permanently disable it, but we can set a high enough value to prevent it from appearing.

So what do we have to do to ‘resolve’ this?

My preferred choice was to use Group Policy Preferences (Registry) and set a new value for HKCU\Control Panel\Desktop /v HungAppTimeout /t REG_SZ /d 120000

This sets the timeout to 2 minutes as opposed to 5 seconds.  Now when our program is used we get this result:

No More Artifacts.

When I was investigating this I found I could get the artifacting to occur in both ICA and RDP but not when on the console.  The frustrating thing about this issue is that it was not consistent.  Because of the 5 second default time limit, the program(s) we had that would ‘artifact’ would sometimes complete their UI locking job faster than 5 seconds, but sometimes not.  This lead to reports of artifacting occurring more often ‘during the peak work hours’ when the application/server/user load was the most.  This makes sense as the higher load undoubtedly lead to everything being slower, the database, server, etc. leading to the application waiting longer and thus exceeding the timeout.  I did find through the course of troubleshooting this issue that it seemed to ‘go away’ when I was trying to replicate after hours, which is frustrating to only have a slice of time to try and resolve this during peak hours.

Fortunately, after implementing the HungAppTimeout registry key the artifacts for several application ‘went away’.

Lastly, contrary to this article you do NOT need to restart for this value to take effect.  WinLogon.exe reads the HungAppTimeout value and then configures DWM accordingly when your profile loads.  So for this value to take effect you only need to log on with this value already residing in your user’s registry hive.


Read More

Citrix Client Side Extensions 1.7.6+ breaks policies

/ /
in Blog

We apply our several group policy settings via Active Directory group policy objects, within them we set ‘Citrix’ policies.  This includes things like ‘Licensing’, platform, etc.



When we upgraded the Citrix Group Policy Client Side Extensions (x86) to version 1.7.6 or 1.7.7 we found that these values were no longer being applied.  FYI!



Read More

Citrix XenApp 6.5 – IMA errors galore, mfcom won’t start

/ /
in Blog

I’ve seen this happen a few times now where the “Citrix Independent Management Architecture” (aka IMAService) won’t start, erroring with various errors:

All of these errors appear to be a registry with incorrect permissions configured on the Citrix keys.  Why did these keys get their permissions reset?  I’m unsure.  I DID just install Citrix UPM 5.4 which may reset the keys?

Here is how you fix the permissions (at least, everything I could possibly find):
1) Download SetACL.exe
2) Save this file to ‘CitrixRegPerms.txt’:

You may need to identify the local SID for ‘NETWORKSERVICE’.  In my example the value is:


You may need to replace your SID for NetworkService with the one from my file above.

Lastly this script will ‘fix’ the incorrect permissions:



Read More

Citrix Netscaler – Load balancing options with XML brokers

/ /
in Blog

We were encountering ‘slowness’ with our Citrix Web Interface that logging into it and displaying the applications would take a significant amount of time ( > 10 seconds).  I have spent considerable amount of time investigating what our performance should be in our environment.  With this information we were still seeing these long logins.  It was consistently at 200 or greater concurrent connections on our web interface servers that we saw our login times spike.  I know at ~200 or so connections our response time is around 5,000ms for our XML brokers.  I then used my PowerShell script to measure the response times of our XML broker’s we are using in load balancing.  We had two servers WI01 and WI02 that I monitored and as our web interface servers were becoming loaded I noticed something odd.

Only one of our XML brokers was being hit, the other was sitting idle.  Since we use a VIP that is supposed to have load balancing I brought our network team who support the Netscalers to help me take a look.
I configured two loads:
1) Direct connections to the XML broker/VIP. This request simulates what the web interface sends to the XML broker on behalf of the user. The traffic POST’ed is an XML request for a list of all applications available to the user. This is the hardest request the web interface can send an XML broker because the XML broker needs to filter out applications the user does not have access to before it can send a response.
2) Through the web interface. This request simulates going through the HTML pages to get your list of applications available on the web AND a PNA query for all applications available to the user.

To establish a baseline I ‘loaded’ a XML server directly (WSCTXAPP301T)


The burgundy line is 200 connections to the server posting a XML request for the list of all applications. When loading a single server the response time goes up to 5000ms.

I then loaded the VIP with 200 connections with the XML request.


From this we can see the ‘load’ switch servers between the WSCTXZDC301T and WSCTXAPP301T. This is NOT load balancing, but the Netscaler failing over when *it* does not receive a reply from the XML monitor within 5000ms. The performance with PERSISTENCE set to SOURCEIP is no better than a single server for servicing requests.

Loading the Webinterface with requests:


We see the exact same situation when going through the web interface.


After setting the PERSISTENCE to NONE I directed traffic directly to the VIP:


What we are seeing is for the exact same load (200 connections) our response time is 50% better with PERSISTENCE set to NONE. We are getting responses back in ~2300-3000ms.

Load testing the Web Interface with PERSISTENCE set to NONE sees the same results:


What’s the impact on the end user by setting PERSISTENCE to NONE? There is no impact. During load testing I made several ‘real’ connections using Citrix Receiver via PNA and through the Web Interface and both continued to operate without issue enumerating applications. With this testing and information we need to change our PERSISTENCE value to NONE.


Read More

Microsoft Office – Your mailbox is currently being optimized as part of upgrading to Outlook 2010

/ /
in Blog

I was using our Office 2010 through Citrix and started noticing I was getting this message:


“Your mailbox is currently being optimized as part of upgrading to Outlook 2010.  This one time process may take over 15 minutes and performance may be affected while optimization is in progress.”

“Upgrade Outlook Connector”

“You must upgrade to the latest version of Outlook Hotmail Connector to continue using this e-mail account.”

In addition, all of the options under ‘Account Settings’ do nothing.  This is in addition to my Home/Inbox being empty and my calendar blank.  Attempting to create a calendar event results in:

I then tried to open the ‘Mail’ control panel and found it was missing.  I manually opened it by launching it from:
“C:\Program Files (x86)\Microsoft Office\Office14\MLCFG32.CPL”


From here, clicking any of the buttons resulted in nothing happening.  Email Accounts… Data Files… did nothing.  Clicking ‘Show Profiles’ did work, though.  So what the heck is going on?  I opened up Process Monitor to try and find out.  I opened up the ‘Mail’ control panel via the command line, since this is a ‘Rundll32.exe’ I set the procmon filter to ‘Command Line’ ‘contains’ ‘MLCFG’ and clicked one of the non-functioning buttons.  My result:

4 5

PATH NOT FOUND.  Why isn’t the path found?  I opened a command prompt and followed the path doing a dir /x to show shortnames:


Well, there is the issue.  It’s showing as ‘MICROS~4’ when it’s searching for ‘MICROS~2’.  Doing a repair on office will cause re-registration to occur and this does resolve the issue, but I wanted to change my shortname instead of doing a repair.  Since these are PVS servers I created a new version, mounted the vDisk and changed the shortnames via fsutil:


In the screenshot, you can clearly see ‘Microsoft Office’ is now ‘MICROS~2’.  I unmounted the vDisk, set a target device and booted it up.  I opened the command prompt and…


What the heck?  For some reason mounting my vDisk offline and I can’t change the shortname.  I get access denied trying to change the shortname when the server is online so I was hoping offline would work.  Unfortunately, it does not appear to be so.

So what is the solution?  I guess repair office to force registration and it will point to the new paths because it will re-register all components. 🙁

Read More