Citrix Receiver

Citrix Storefront – Performance Testing and Tuning – Part 5 – Establishing a Peak Load

2017-06-06
/ /
in Blog
/

In part 1 I setup a script to load test our Citrix Storefront server, as a user logging in through the website.  Part 2 was users connecting via PNA.  In this part I’m going to examine our existing load.  In order to do so, I only want to count the following users:

  1. Users must have logged into Web Interface.  This ensures that we count users who have actually logged in to launch an application.  Some of our users may have the Web Interface set as their homepage so this way we can avoid idle user counts.
  2. Users who have launched an application.  I’m interested in the users who have actually launched an application via Web Interface.
  3. PNA connections
  4. PNA launches

I want to get these user counts sorted via time in order to determine our peak load.

To get these counts I decided to key into some unique web URL’s or resources.  For #1, I decided to search the IIS logs for the “Logoff.png”.

This icon will only be pulled down for a user who has logged into the Web Interface.

For #2, I decided to look at the IIS logs and search for “launch.ica”.  This will get us the application launches via the sites.

For #3, I decided to look at the IIS logs and search for “config.xml”.  This file is pulled each time PNA is launched or a user logs onto the machine.

For #4, I decided to look at the IIS logs and search for “launch.aspx”.

Users must have logged into Web Interface.

For #1 I created a powershell script to parse the IIS log file and export my data:

I then opened it in Excel and created time ranges per-hour and then counted how many of each item was found in that range.  For the Logoff.PNG it was accessed like this:

The peak count was 2775 logons over a course of an hour.  This equals ~45 logons per minute or 1 logon every 1.29 seconds.

In order to get even more precise, I broke down the hour that had the 2775 logons and looked at how many logons occurred over 15 minute time spans.

The peak rate over a 15 min span was:

1 logon every 0.974 second. (1.02 logons per second)


Users who have launched an application

 

For users who launch an application we are at 7593 launches as a peak over an hour.

The peak access to Web Interface is 2.10 launches per second.

If we do a failover strategy to ensure , say in the future, we unified our architecture to a single location (or single cloud?) what would our combined rate be?

1 launch every 0.561 second. (1.78 launches per second)

Citrix has Storefront guidance that says it can handle connections/users in the ten’s of thousands.  I thought we were a large organization, but apparently Storefront has been designed for something even bigger! (Or web-based front ends are rarely the bottle neck…)

So these numbers are purely for logons through the web site.  But previous testing for PNA functionality showed that it’s a *much* heavier traffic to Web Interface.  Why is the traffic heavier for PNA?  Because each time a user logs on to a workstation it kicks in.  Whereas accessing the web interface requires some work the user must do, PNA is automatic.  It’s also more convenient for users as they prefer to launch desktop/start menu icons.


PNA connections

How am I going to determine our concurrent connection rate for PNA?  This is what a PNA connection looks like:

What I’m going to do is query the IIS logs for ‘config.xml‘, as that file gets called only once when Receiver is launched or upon login.  Once I get that I can search for ‘enum.aspx’ command that has been called and divide by the number of config.xml’s.  This should get me the number of icon requests per user.

After doing the math I found Receiver makes 2.63 enum.aspx request for every config.xml requests.

Lastly,

My thought process here is once we get our peak ‘Receiver’ connection rate to the Storefront server we can create another script to simulate it.  I know WCAT can do this as no funny cookie business occurs for Receiver.

So what does the PNA Receiver numbers look like?

We have a peak PNA connection rate of 7.531 config.xml connections per second.  Multiplying by the additional 2.63 enum.aspx requests give us a total of 19.806 connections per second for a peak rate.

For PNA application launches, our peak rate is 30,075 launches in a single hour.  This was measured by parsing for “launch.aspx”.  Breaking this hour down to 15 minute intervals and a single 15min window contained 53% of the launches.  For our peak 15 minute window this is 16148 launches.  This gives us a total of 17.94 application launches per second.

Peak Rates (in per second)
PNA Logons PNA App Launches Web Logons Web LaunchICA Connections per second
7.531 17.94 1.02 1.78 28.271

 

 

Read More

Citrix Storefront – Adventures in customization – Dynamically configure workspace control based on group membership

2017-05-07
/ /
in Blog
/

I’ve been exploring the customization capabilities of Citrix Storefront and have some exciting ideas on simplifying our deployment.  What I’d really like is to reduce our store count down to a as few stores as possible.  In our Web Interface we have multiple stores based on the non-configurable settings.  They are:

  • Workspace Control Enabled with Explicit Logon
  • Workspace Control Disabled with Explicit Logon
  • Workspace Control Enabled with Domain Passthrough authentication
  • Workspace Control Disabled with Domain Passthrough authentication
  • Anonymous site

We can’t mix and match authenticated sites and anonymous sites (right?… ?) but Citrix does offer the ability to configure Authentication methods AND Workspace Control options via their ‘Receiver Extension API’s’.

These are the API’s in question:

There isn’t really a whole lot of documentation on them and how to use them.  Richard Hayton has created the Citrix Customization Cookbook which details some examples of some of the API’s.  He has several blog articles on the Citrix website that have varying degrees of applicability.  Unfortunately, he hasn’t blogged in over a year on this topic as it feels like the situation has changed a bit with the Citrix Store API’s available as well (note: these are different!).

My target is to make it so these options can be set dynamically based on the users group membership.  If you’re a member of the group ‘workspaceControlEnabled’ you get all the settings set to true, if you’re a member of ‘workspaceControlDisabled’ you get all the settings set to false.

Seems like a pretty straightforward goal?

So I thought I’d start with something pretty simple.  I have a store with Workspace Control Enabled, with show ‘Connect and Disconnect’ buttons selected:

If I log into the site:

I see everything (as I should).

So let’s start with a simple customization.  Let’s try using the API to disable these options.  I created a totally blank script.js file and added the following lines:

Now what does our menu look like?

Awesome!  Workspace Control was disabled by script!

So I said I want to disable Workspace Control if you are a member of a specific group.  Richard Hayton actually wrote a pretty good article on creating a service to facilitate grabbing your group membership with Storefront.  Unfortunately, the download link is dead.  So I wrote another PowerShell HTTP listener that takes an input and queries AD for that user and their membership, and returns a positive value if workspace control should be enabled, or a negative value if it should be disabled.  To get it to query though, it needs a value.  The best value for this, I thought, would be the user name.

Citrix provides a function to get the username and I can then pass it to my webservice, test for group membership and return whether Workspace Control (WSC) should be enabled or disabled.

I wrote my first bit of code and it crashed immediately.  I simply entered it straight into my script.js.

In order to trace the error, you simply enter “#-tr” to the end of your store URL:

and allow pop-ups.  A new tab will open allowing you to follow the ‘flow’ of Storefront as it executes its commands.  Mine crashed at:

“get username data:”

And it makes sense why it crashed there.  I haven’t even logged in yet so it has no idea who or how to get the username so it looks like Storefront just returns the login page.  We need to call our functions after logging in.

Citrix provides the following event-based functions we can hook into:

Notifications of progress

preInitialize(callback)
postInitialize(callback)
postConfigurationLoaded()
postAppListLoaded()

Note that during these calls, the UI may be hidden in native Receivers, so it is not safe to show UI
For APIs passing a callback, you MUST call the callback function, though you may delay calling it until code of your own has run.

beforeLogon(callback)

Web browsers only. Called prior to displaying any logon dialogs. You may call ‘showMessage’ here, or add your own UI.

beforeDisplayHomeScreen(callback)

All clients, called prior to displaying the main UI. This is the ideal place to add custom startup UI.
Note that for native clients, the user may not have logged in at this stage, as some clients allow offline access to the UI.

afterDisplayHomeScreen()

All clients, called once the UI is loaded and displayed. The ideal place to call APIs to adjust the initial UI, for example to start in a different tab.

So the question becomes, when does each of these get called?  We are only interested in the ones after you login.  To determine this, I hooked into each one and just did a simple trace command and then refreshed my browser to the login screen.

This is where I stopped:

And trace tab results:

These 4 stages are called before the user logs on so they are of no use to me:

preInitialize
postInitialize
postConfigurationLoaded
beforeLogon

The order of the other 3 after logon:

beforeDisplayHomeScreen
postAppListLoaded
afterDisplayHomeScreenStage

Before adding my code to the post-logon event functions I just added it back -plain jane- and rerun with a trace:

What I found is these extensions appear to have a fixed entry order point.  The workspace control extensions cannot be called after “beforeDisplayHomeScreen” stage.  If you do not call the workspace control extensions before the callback on the ‘beforeDisplayHomeScreen’ function you will be unable to control the setting.  The trace log in my screenshot for these extensions will always occur at this point in time regardless if you actually set it in ‘preInitialize, postInitialize, postConfigurationLoaded, or beforeLogon’.  And if you attempt to set it in either of the two later functions it will not log anything and your code has no effect.  So the only point in time where I can take the username and set these values are in the event function beforeDisplayHomeScreen.

<Digress>

During the course of my testing this feature I had thought about adding a button that would allow you to toggle this feature ‘enabled or disabled’ on your own whim.  But it appears once you call the Extension it’s a one and done.  I also discovered that it appears you must set the workspace control feature early in the process.  If I set it in postAppListLoaded or afterDisplayHomeScreen nothing happened.  To be fair, I do not know how to reinitialize the menu, maybe that would allow it to kick in dynamically…?  I guess that’s for further exploration on another day…

</Digress>

Ok, so we’ve found the one and only functional place we can execute our code.  So I added it.

 

The result?  Nothing.  Nothing happened.

Well, that’s not entirely true.

I’ve highlighted in yellow/orange my “getUsername” function.  We can see on line 35 we get into the function.  And on line 67 it is successfully finding and returning my name.  But the problem is that it’s getting that information after the point in time that we can set the WSC features (highlighted in blue — line 60-62).

I found that using ajax for this command and attempting to use async was causing my failure.  I understand it’s bad practice to do synchronous commands, especially in javascript as they will lock the UI while executing, but, thus far it’s the only way I know to ensure it gets completed in the proper order.  I am really not a web developer so I don’t know what’s the proper technique here to send a couple ajax requests that only blocks at the specific point in time that WSC kicks in…  Or find a way to redraw the menu?  But for the purposes of getting this working, this is the solution I’ve chosen to go with.  I’m wide open to better suggestions.  The real big extension that would be an issue is the ‘webReconnectAtStartup’.  This feature will reconnect any existing sessions you have and the way that Citrix currently implements it, they want it run as soon as the UI is displayed.  This makes some sense as that’s the whole point.  You don’t want to wait around after logging in some indeterminate or random amount of seconds for your session to reconnect…  But, this issue can be alleviated.  Citrix actually offers a way to implement this feature yourself via the Store API so we could implement our own custom version of this function that would get all your sessions and reconnect them…

Which could just leave building the menu as something that could be moved back to async if I can figure out how to rebuild it or build it dynamically…

Anyway, that maybe for another day.  For today, the following works for my purpose.

This is my custom/script.js file that I finished from this blog post:

 

Here is the LDAP_HttpListener.psm1

Lastly, the scheduled task to call the listener:

 

 

 

 

Read More

Citrix Storefront – Pass URI parameters to an application

2017-05-01
/ /
in Blog
/

In my previous post, I was exploring taking URI parameters and passing them to an application.

The main issue we are facing is that Storefront launches the ica file via an iframe src.  When launching the ica via this method the iframe does a simple ‘GET’ without passing any HEADER parameters — which is the only (documented) way to pass data to Storefront.

What can I do? I think what I need to do is create my own *custom* launchica command.  Because this will be against an unauthenticated store we should be able remove the authentication portions AND any unique identifiers (eg, csrf data).  Really, we just need the two options — the application to launch and the parameter to pass into it.  I am NOT a web developer, I do not know what would be the best solution to this problem, but here is something I came up with.

My first thought is this needs to be a URL that must be queried and that URL must return a specific content-type.  I know Powershell has lots of control over specifying things like this and I have some familiarity with Powershell so I’ve chosen that as my tool of choice to solve this problem.

In order to start I need to create or find something that will get data from a URL to powershell.  Fortunately, a brilliant person by the name of Steve Lee solved this first problem for me.

What he created is a Powershell module that creates a HTTP listener than waits for a request. We can take this listener and modify it so it listens for our two variables (CTX_Application and NFuseAppCommandLine) and then returns a ICA file.  Since this is an unauthenticated URL I had to remove the authentication feature of the script and I added a function to query the real Storefront services to generate the ICA file.

So what I’m envisioning is replacing the “LaunchIca” command with my custom one.

 

This is my modification of Steve’s script:

And the command to start the HTTP listener:

Eventually, this will need to be converted to a scheduled task or a service.  When running the listener manually, it looks like this:

 

I originally planned to use the ‘WebAPI’ and create a custom StoreFront, but I really, really want to use the new Storefront UI.  In addition, I do NOT want to have to copy a file around to each Storefront server to enable this feature.  So I started to wonder if it would be possible to modify Storefront via the extensible customization API’s it provides.  This involves adding javascript to the “C:\inetpub\wwwroot\Citrix\StoreWeb\custom\script.js” and modifying the “C:\inetpub\wwwroot\Citrix\StoreWeb\custom\style.css” files.  To start, my goal is to mimic our existing functionality and UI to an extent that makes sense.

The Web Interface 5.4 version of this web launcher looked like this:

When you browse to the URL in Web Interface 5.4 the application is automatically launched.  If it doesn’t launch, “click to connect” will launch it for you manually.  This is the function and features I want.

Storefront, without any modifications, looks like this with an authenticated store:

So, I need to make a few modifications.

  1. I need to hide all applications that are NOT my target application
  2. I need to add the additional messaging “If your application does not appear within a few seconds, click to connect.” with the underlined as a URL to our launcher.
  3. I want to minimize the interface by hiding the toolbar.  Since only one application will be displayed we do not need to see “All | Categories   Search All Apps”
  4. I want to hide the ‘All Apps’ text
  5. I want to hide “Details”, we’re going to keep this UI minimal.

The beauty of Storefront, in its current incarnation, is most of this is CSS modifications.  I made the following modifications to the CSS to get my UI minimalized:

This resulted in a UI that looked like this:

So now I want to remove all apps except my targeted application that should come in a query string.

I was curious if the ‘script.js’ would recognize the URI parameter passed to Storefront.  I modified my ‘script.js’ with the following:

Going to my URL and checking the ‘Console’ in Chrome revealed:

Yes, indeed, we are getting the URI parameters!

Great!  So can we filter our application list to only display the app in the URI?

Citrix offers a bunch of ‘extensions’.  Can one of them work for our purpose?  This one sounds interesting:

Can we do a simple check that if the application does not equal “CTX_Application” to exclude it?

The function looks like this:

Did it work?

Yes!  Perfectly!  Can we append a message to the application?  Looking at Citrix’s extensions this one looks promising:

Ok.  That warning sucks.  My whole post is based on StoreFront 3.9 so I cannot guarantee these modifications will work in future versions or previous versions.  Your Mileage May Vary.

So what elements could we manipulate to add our text?

Could we add another “p class=storeapp-name” (this is just text) for our messaging?  The onAppHTMLGeneration function says it is returned when the HTML is generated for an app, so what does this look like?

I added the following to script.js:

And this was the result in the Chrome Console:

So this is returning an DomHTMLElement.  DOMHTMLElements have numerous methods to add/create/append/modify data to them.  Perfect.  Doing some research (I’m not a web developer) I found that you can modify the content of an element by this style command:

This results in the following:

We have text!

Great.

My preference is to have the text match the application name’s format.  I also wanted to test if I could add a link to the text.  So I modified my css line:

The result?

Oh man.  This is looking good.  My ‘click to connect’ link isn’t working at this point, it just goes to google, but at least I know I can add a URL and have it work!  Now I just need to generate a URL and set that to replace ‘click to connect’.

When I made my HTTPListener I purposefully made it with the following:

The reason why I had it set to share the url of the Citrix Store is the launchurl generated by Storefront is:

The full path for the URL is actually:

So the request actually starts at the storename.  And if I want this to work with a URL re-write service like Netscaler I suspect I need to keep to relative paths.  So to reach my custom ica_launcher I can just put this in my script.js file:

and then I can replace my ‘click to connect’ link with:

The result?

The url on the “click to connect” goes to my launcher!  And it works!  Excellent!

Now I have one last thing I need to get working.  If I click the ‘Notepad 2016 – PLB’ icon I get the regular Storefront ica file, so I don’t get my LongCommandLine added into it.  Can I change where it’s trying to launch from?

Citrix appears to offer one extension that may allow this:

Huh.  Well.  That’s not much documentation at all.

Fortunately, a Citrix blog post came to rescue with some more information:

Hook APIs That Allow Delays / Cancellations

doLaunch
doSubscribe
doRemove
doInstall

On each of these the customization might show a dialog, perform some checks (etc) but ultimately should call ‘action’ if (and only if) they want the operation to proceed.

This extension is taking an object (app).  What properties does this object have?  I did a simple console.log and examined the object:

Well, look that that.  There is a property called ‘launchurl’.  Can we modify this property and have it point to our custom launcher?

I modified my function as such:

The result?

A modified launchurl!!!!

Excellent!

And launching it does return the ica file from my custom ica_launcher!

Lastly, I want to autolaunch my program.  It turns out, this is pretty simple.  Just add the following the script.js file:

Beautiful.  My full script.js file looks like so:

And that’s it.  We are able to accomplish this with a Powershell script, and two customization files.  I think this has a better chance of ‘working’ across Storefront versions then the SDK attempt I did earlier, or creating my own custom Storefront front end.

Read More

Citrix Netscaler 11.1 Unified Gateway and a non-working Citrix HTML5 Receiver

2016-12-06
/ /
in Blog
/

We setup a Citrix Unified Gateway for a proof of concept and were having an issue getting the HTML5 Receiver to connect for external connections.  It was presenting an error message: “Citrix Receiver cannot connect to the server”.  We followed this documentation.  It states, for our use case:

What would probably help is having a proxy that can parse out all websocket traffic and convert to ICA/CGP traffic without any need of changes to XA/XD. Netscaler Gateway does exactly this… …NetScaler Gateway also doesn’t need any special configuration for listening over websockets…

Connections via NetScaler Gateway:

…When a gateway is involved, the connections are similar in all Receivers. Here, Gateway acts as WebSocket proxy and in-turn opens ICA/CGP/SSL native socket connections to backend XenApp and XenDesktop. …

…So using a NetScaler Gateway would help here for ease of deployment. Connection to gateway is SSL/TLS and gateway to XenApp/XenDesktop is ICA/CGP….

And additional documentation here.

WebSocket connections are also disabled by default on NetScaler Gateway. For remote users accessing their desktops and applications through NetScaler Gateway, you must create an HTTP profile with WebSocket connections enabled and either bind this to the NetScaler Gateway virtual server or apply the profile globally. For more information about creating HTTP profiles, see HTTP Configurations.

Ok.  So we did the following:

  1. We enabled WebSocket connections on Netscaler via the HTTP Profiles
  2. We configured Storefront with HTML5 Receiver and configured it for talking to the Netscaler.

And then we tried launching our application:

We started our investigation.  The first thing we did was test to see if HTML5 Receiver works at all.  We configured and enabled websockets on our XenApp servers and then logged into the Storefront server directly, and internally.  We were able to launch applications without issue.

The second thing we did was enable logging for HTML5 receiver:

To view Citrix Receiver for HTML5 logs

To assist with troubleshooting issues, you can view Citrix Receiver for HTML5 logs generated during a session.

  1. Log on to the Citrix Receiver for Web site.
  2. In another browser tab or window, navigate to siteurl/Clients/HTML5Client/src/ViewLog.html, where siteurlis the URL of the Citrix Receiver for Web site, typically http://server.domain/Citrix/StoreWeb.
  3. On the logging page, click Start Logging.
  4. On the Citrix Receiver for Web site, access a desktop or application using Citrix Receiver for HTML5.

    The log file generated for the Citrix Receiver for HTML5 session is shown on the logging page. You can also download the log file for further analysis.

This was the log file it generated:

The “Close with code=1006” seemed to imply it was a “websocket” issue from google searches.

 

The last few events prior to the error are “websocket” doing…  something.

I proceeded to spin up a home lab with XenApp and a Netscaler configured for HTML5 Receiver and tried connecting.  It worked flawlessly via the Netscaler.  I enabled logging and took another look:

So there is a lot of differences but we focus on the point of failure in our enterprise netscaler we see it seems to retry or try different indexes (3 in total, 0, 1 and 2).

So there is a lot of evidence that websockets seem to be culprit.  We have tried removing Netscaler from the connection picture by connecting directly to Storefront and HTML5 receiver works.  We have configured both Netscaler and Storefront (with what we think) is a correct configuration.  And still we are getting a failure.

I opened up a call to Citrix.

It was a fairly frustrating experience.  I had tech’s ask me to go to “Program Files\Citrix\Reciever” and get the receiver version (hint, hint, this does not exist with HTML5).  I captured packets of the failure “in motion” and they told me, “it’s not connecting to your XenApp server”.  — Yup.  That’s the Problem.

It seems that HTML5 is either so new (it’s not now), so simple (it’s not really), or tech’s are just poorly trained.  I reiterated to them “why does it make 3 websocket connections on the ‘bad’ netscaler? Why does the ‘good’ netscaler appear to connect the first time without issue?”  I felt the tech’s ignore and beat around the bush regarding websockets and more focus put on the “Storefront console”.  Storefront itself was NOT logging ANYTHING to the event logs.  Apparently this is weird for a storefront failure.  I suspected Storefront was operating correctly and I was getting frustrated we weren’t focusing on what I suspected was the problem (websockets).  So I put the case on hold so I could focus on doing the troubleshooting myself instead of going around in circles on setting HTML5 to “always use” or “use only when native reciever is not detected”.

Reviewing the documentation for the umpteenth time this “troubleshooting connections” tidbit came out:

Troubleshooting Connections:

In cases where you are not able to connect some of the following points might help in finding out the problem. They also can be used while opening support case or seeking help in forums:

1) Logging: Basic connection related logs are logged by Receiver for HTML5 and Receiver for Chrome.

2) Browser console logs: Browsers would show errors related to certificates or network related failures here.

  • Receiver for HTML5: Open developer tools for HDX session browser tab. Tip: In Windows, use F12 shortcut in address bar of session page.

  • Receiver for Chrome: Go to chrome://inspect, click on Apps on left side. Click on inspect for Citrix Receiver pages (Main.html, SessionWindow.html etc)

The browser may show a log?  I wish I would have thought of that earlier.  And I wish Citrix would have put that in the actual “Receiver for HTML5” documentation as opposed to buried in a blog article.

So I opened the Console in Chrome, launched my application and reviewed the results:

We finally have some human readable information.

Websocket connections are failing during the handshake “Unexpected response code: 302”

What heck does 302 mean?  I installed Fiddler and did another launch withe Fiddler tracing:

 

I highlighted the area where it tells us it’s attempting to connect with websockets.  We can see in the response packet we are getting redirected, that’s what ‘302’ means.  I then found a website that lets you test your server to ensure websockets are working.  I tried it on our ‘bad’ netscaler:

 

Hitting ‘Connect’ left nothing in the log.  However, when I tried it with my ‘good’ netscaler…

 

It works!  So we can test websockets without having to launch and close the application over and over…

 

So we started to investigate the Netscaler.  We found numerous policies that did URL or content redirection that would be taking place with the packet formulated like so.  We then compared our Netscaler to the one in my homelab and did find one, subtle difference:

The one on the left is showing a rule for HTTP.REQ.URL.CONTAINS_ANY(“aaa_path”) where the one on the right just shows “is_vpn_url”.  Investigating further it was stated that our team was trying to get AAA authentication working and this was an option set during a troubleshooting stage.  Apparently, it was forgotten or overlooked when the issue was resolved (it was not applicable and can be removed).  So we set it back to having the “is_vpn_url” and retried…

It worked!  I tried the ‘websockets.org’ test and it connected now!  Looking in the Chrome console showed:

 

Success!  It doesn’t pause on the websocket connection and the console logging shows some interesting information.  Fiddler, with the working connection, now displayed the following:

Look!  A handshark response!

 

So, to review what we learned:

  1. Connections via Netscaler to HTML5 reciever do NOT require  (but is possible) a SSL connection on each target XenApp device
  2. Connection via Netscaler work over standard port (2598/1494) and do not require any special configuration on your XenApp server.
  3. You can use ‘http://www.websocket.org/echo.html’ to test your Netscaler to ensure websockets are open and working.
  4. Fiddler can tell you verbose information on your websocket connection and their contents.
  5. The web browser’s Javascript console is perfect to look at verbose messages in HTML5.

 

And with that, we are working and happy, Good Night!

Read More

Citrix Receiver 4+ rants and “Your apps are not available at this time. Please try again in a few minutes or contact your help desk with this information”

2015-10-28
/ / /

The environment I’m working in is a mix of XenApp 6.5, 5.0 and Presentation Server 4.5.  The 6.5 and 5.0 farm also have a nearly identical test farm.  We’ve been migrating the applications off the Presentation Server farms and are moving them to XenApp 6.5.  At this point in the migration we have around 5-10 applications left on 4.5 to move, with around 400-450 on the 6.5 farms and probably around 20-30 or so on the 5.0 farms.  These farms utilize a Citrix Webinterface 5.4.2 frontend for web interface and PNA.

We have standardized the environment on mostly Citrix Receiver 3.3 and some 3.4.  Time has marched and we’ve been tasked with getting Receiver 4+ working the Windows 7/8/10 rollout.  We were not able to do so with the earlier versions of Receiver 4 because things like sort icons into custom folders on the desktop and Start Menu.  This feature came in around 4.2.  In our environment memberships to applications are granted through group membership and Citrix PNA allowed the user to ‘roam’ from computer to computer only displaying the applications they have access to, as opposed to a bunch of applications they do not have access with the onus on them to pick and choose the correct applications.

So we started work on planning this migration and have started with the latest greatest (as of today) Citrix Receiver 4.3.  We have been able to come close to simulating all the features of the Enterprise editions of 3.3 and 3.4.  Namely:

Citrix Receiver automatically connects and populates a folder (MyApps) in the Start Menu and on the desktop.
No self-service.  Applications are automatically presented to you and defined by Group Membership.
Single sign-on.  Receiver will take your Windows logon credentials to use for authentication.

We do have some outstanding items.  We’ve set the client to have all applications as ‘MANDATORY’ which does populate the applications in the MyApps folders; but applications marked as ‘Create shortcut on the desktop’ in the applications properties in AppCenter are not created.

Anyways, onto the problem.  Now that we have Receiver 4.3 setup, SSON working, PNA working, we logged onto our system and watched the applications populate.

Slowly.

Really Slowly.

Eventually a dialog popped up.

Then another, and another, and another.

And these dialogs are completely custom!  They are NOT native Windows dialogs!

So if you have multiple ones of them, sometimes clicking the X (close button) or OK doesn’t work because the dialog appears ‘modal’ and you need to click the button on the ‘active’ window.  But you generally don’t know which one that is so you have to go through and select each dialog from the task bar and try clicking ok until you magically get lucky and select the one that has priority.  Then you do it all over again as the next primary window *may not be the one on top*.

1 minute 24 seconds to populate 470 applications

Just for giggles, how fast does Receiver 3.3 populate the same list?

8 seconds.  And all the icons show up.

Alright, so you’ve passed that point and are now looking at your applications.  But they are missing icons!

But not all applications are missing their icons…  Only some.

So let’s find out what’s consuming all this time and maybe, just maybe, we’ll solve our “Your apps are not available” error message.

First thing we need to do is enable Citrix Receiver Logging:

Next is to exit and restart receiver and logs will start to generate.  They are located here:
%USERPROFILE%AppDataLocalCitrixReceiver
%USERPROFILE%AppDataLocalCitrixAuthManager
%USERPROFILE%AppDataLocalCitrixSelfService

The most important log tends to be the ‘SelfService.txt’ log.  If you search that log for the “Your apps are not available” error message it pops up in locations like this:

So this dialog popped up for an application called ‘BMTServe’.  And what does BMTServe look like?

Generic icon!

But BMTServe was not the only application that encountered this dialog.  From my video it popped up numerous times.  Searching the SelfService.txt file for ‘Your apps’ and looking for the application it references points to an application with a blank icon 100% of the time.  Not every application that produces a blank icon causes this prompt, as we literally have ~250 applications with blank icons and the dialog pops up anywhere from 0 to 10 times.  Sometimes it pops up 2 times, sometimes none, sometimes 10 times.

So, why are these icons blank?

Citrix Receiver 4.3 seems to only prefer 32bit icons or icons of a particular size.  I haven’t confirmed what exactly yet, but I do know that 8bit 32×32 icons don’t seems to get ‘translated’.  The Citrix logs all but confirm this as well.

I confirmed with Citrix that icons are required to be 32bit and the order they are checked is 48×48, 32×32 then 16×16.

This is how Receiver processes icons that are formatted correctly:

The icons were processed instantly. 00:00:00. But if they are formatted in a way that Receiver decides it needs to ‘reformat’ them:

This call to get an icon took 11.6 seconds!!! If it doesn’t get the icon formatted in the way it wants, it appears the SelfService.exe setups a queue of icons that it needs and ‘re-requests’ them from the server. Could it be that Receiver is submitting too many queries? The error mentions to check the authmansvr.txt log file. This log file shows the following:

The error appears to start at “CWindowsReceiver::CallARGetConnectedVpnGateway” When this call is successful it returns:

So, I guess it’s possible that trying to re-pull the icon data is causing authmansvr.exe to crash…?  Another crazy thing is I was attempting to automate this process of terminating Receiver and relaunching it to see if I could get a gauge on the frequency of this occurrence and this is what I saw: