Blog

Citrix Storefront – Pass URI parameters to an application

2017-05-01
/ /
in Blog
/

In my previous post, I was exploring taking URI parameters and passing them to an application.

The main issue we are facing is that Storefront launches the ica file via an iframe src.  When launching the ica via this method the iframe does a simple ‘GET’ without passing any HEADER parameters — which is the only (documented) way to pass data to Storefront.

What can I do? I think what I need to do is create my own *custom* launchica command.  Because this will be against an unauthenticated store we should be able remove the authentication portions AND any unique identifiers (eg, csrf data).  Really, we just need the two options — the application to launch and the parameter to pass into it.  I am NOT a web developer, I do not know what would be the best solution to this problem, but here is something I came up with.

My first thought is this needs to be a URL that must be queried and that URL must return a specific content-type.  I know Powershell has lots of control over specifying things like this and I have some familiarity with Powershell so I’ve chosen that as my tool of choice to solve this problem.

In order to start I need to create or find something that will get data from a URL to powershell.  Fortunately, a brilliant person by the name of Steve Lee solved this first problem for me.

What he created is a Powershell module that creates a HTTP listener than waits for a request. We can take this listener and modify it so it listens for our two variables (CTX_Application and NFuseAppCommandLine) and then returns a ICA file.  Since this is an unauthenticated URL I had to remove the authentication feature of the script and I added a function to query the real Storefront services to generate the ICA file.

So what I’m envisioning is replacing the “LaunchIca” command with my custom one.

 

This is my modification of Steve’s script:

And the command to start the HTTP listener:

Eventually, this will need to be converted to a scheduled task or a service.  When running the listener manually, it looks like this:

 

I originally planned to use the ‘WebAPI’ and create a custom StoreFront, but I really, really want to use the new Storefront UI.  In addition, I do NOT want to have to copy a file around to each Storefront server to enable this feature.  So I started to wonder if it would be possible to modify Storefront via the extensible customization API’s it provides.  This involves adding javascript to the “C:\inetpub\wwwroot\Citrix\StoreWeb\custom\script.js” and modifying the “C:\inetpub\wwwroot\Citrix\StoreWeb\custom\style.css” files.  To start, my goal is to mimic our existing functionality and UI to an extent that makes sense.

The Web Interface 5.4 version of this web launcher looked like this:

When you browse to the URL in Web Interface 5.4 the application is automatically launched.  If it doesn’t launch, “click to connect” will launch it for you manually.  This is the function and features I want.

Storefront, without any modifications, looks like this with an authenticated store:

So, I need to make a few modifications.

  1. I need to hide all applications that are NOT my target application
  2. I need to add the additional messaging “If your application does not appear within a few seconds, click to connect.” with the underlined as a URL to our launcher.
  3. I want to minimize the interface by hiding the toolbar.  Since only one application will be displayed we do not need to see “All | Categories   Search All Apps”
  4. I want to hide the ‘All Apps’ text
  5. I want to hide “Details”, we’re going to keep this UI minimal.

The beauty of Storefront, in its current incarnation, is most of this is CSS modifications.  I made the following modifications to the CSS to get my UI minimalized:

This resulted in a UI that looked like this:

So now I want to remove all apps except my targeted application that should come in a query string.

I was curious if the ‘script.js’ would recognize the URI parameter passed to Storefront.  I modified my ‘script.js’ with the following:

Going to my URL and checking the ‘Console’ in Chrome revealed:

Yes, indeed, we are getting the URI parameters!

Great!  So can we filter our application list to only display the app in the URI?

Citrix offers a bunch of ‘extensions’.  Can one of them work for our purpose?  This one sounds interesting:

Can we do a simple check that if the application does not equal “CTX_Application” to exclude it?

The function looks like this:

Did it work?

Yes!  Perfectly!  Can we append a message to the application?  Looking at Citrix’s extensions this one looks promising:

Ok.  That warning sucks.  My whole post is based on StoreFront 3.9 so I cannot guarantee these modifications will work in future versions or previous versions.  Your Mileage May Vary.

So what elements could we manipulate to add our text?

Could we add another “p class=storeapp-name” (this is just text) for our messaging?  The onAppHTMLGeneration function says it is returned when the HTML is generated for an app, so what does this look like?

I added the following to script.js:

And this was the result in the Chrome Console:

So this is returning an DomHTMLElement.  DOMHTMLElements have numerous methods to add/create/append/modify data to them.  Perfect.  Doing some research (I’m not a web developer) I found that you can modify the content of an element by this style command:

This results in the following:

We have text!

Great.

My preference is to have the text match the application name’s format.  I also wanted to test if I could add a link to the text.  So I modified my css line:

The result?

Oh man.  This is looking good.  My ‘click to connect’ link isn’t working at this point, it just goes to google, but at least I know I can add a URL and have it work!  Now I just need to generate a URL and set that to replace ‘click to connect’.

When I made my HTTPListener I purposefully made it with the following:

The reason why I had it set to share the url of the Citrix Store is the launchurl generated by Storefront is:

The full path for the URL is actually:

So the request actually starts at the storename.  And if I want this to work with a URL re-write service like Netscaler I suspect I need to keep to relative paths.  So to reach my custom ica_launcher I can just put this in my script.js file:

and then I can replace my ‘click to connect’ link with:

The result?

The url on the “click to connect” goes to my launcher!  And it works!  Excellent!

Now I have one last thing I need to get working.  If I click the ‘Notepad 2016 – PLB’ icon I get the regular Storefront ica file, so I don’t get my LongCommandLine added into it.  Can I change where it’s trying to launch from?

Citrix appears to offer one extension that may allow this:

Huh.  Well.  That’s not much documentation at all.

Fortunately, a Citrix blog post came to rescue with some more information:

Hook APIs That Allow Delays / Cancellations

doLaunch
doSubscribe
doRemove
doInstall

On each of these the customization might show a dialog, perform some checks (etc) but ultimately should call ‘action’ if (and only if) they want the operation to proceed.

This extension is taking an object (app).  What properties does this object have?  I did a simple console.log and examined the object:

Well, look that that.  There is a property called ‘launchurl’.  Can we modify this property and have it point to our custom launcher?

I modified my function as such:

The result?

A modified launchurl!!!!

Excellent!

And launching it does return the ica file from my custom ica_launcher!

Lastly, I want to autolaunch my program.  It turns out, this is pretty simple.  Just add the following the script.js file:

Beautiful.  My full script.js file looks like so:

And that’s it.  We are able to accomplish this with a Powershell script, and two customization files.  I think this has a better chance of ‘working’ across Storefront versions then the SDK attempt I did earlier, or creating my own custom Storefront front end.

Read More

Citrix StoreFront – Experiences with Storefront Customization SDK and Web API

2017-04-24
/ /
in Blog
/

Our organization has been exploring upgrading our Citrix environment from 6.5 to 7.X.  The biggest road blocks we’ve been experiencing?  Various nuanced features in Citrix XenApp 6.5 don’t work or are not supported in 7.X.

This brings me to this post and an example of the difficulties we are facing, my exploration of solutions to this problem, and our potential solution.

In Citrix Web Interface 5.1+ you can create a URI with a set of application launch parameters and those parameters would be passed to the application.  This is detailed in this Citrix article here (CTX123998).  For us, these launches occur with a Site (WI5.4 terminology) or Store (Storefront terminology) that allows anonymous (or unauthenticated) users.

By following that guide and modifying your Web Interface you can change one of your sites so that it accepts launch parameters.  You simply enter a specific URI in your browser, and the application would launch with said parameters.  An example:

http://bottheory.local/Citrix/XenApp/site/launcher.aspx?CTX_Application=Citrix.MPS.App.XenApp.Notepad&LaunchId=1263894298505&NFuse_AppCommandLine=C:\Windows\WindowsUpdate.log

This allows you to send links around to other people and they can click to automatically launch an application with the parameters specified.  If you are really unlucky, your org might document this as an official way to launch a hosted application and this actually gets coded into certain applications.  So now you may have tens of local apps utilizing this as an acceptable method to launch a hosted application.  For an organization, this may be an acceptable way to launch certain hosted applications since around ~2008 so this “feature”, unfortunately, has built up quite a bit of inertia.

We can’t let this feature break when we move to StoreFront.  We track the number of launches from these hosted applications and it’s in the hundreds/thousands per day.  This is a critical and well used feature.

So how does this work in Web Interface and what actually happens?

URI substitution and launch process

The modifications that you apply add a new ‘query’ to the URI that is picked up.  This ‘query’ is “NFuse_AppCommandLine” and the value it is equal to (“C:\Windows\WindowsUpdate.log” in my example) is passed into the ICA file.

The ICA file, when launched, will pass the parameter to a special string “%**” the is set on the command line of the published application.  This token “%**” gets replaced by the parameter specified in the ICA which then generates the launch string.  This string is executed and the result is the program launches.

Can we do this with Storefront?  Well, first things first, is this even possible with XenApp/XenDesktop 7.X?  In order to test this I created an application and specified the special token.

I then generated an ICA file and manually modified the LongCommandLine to add my parameter.  I then launched it:

 

Did it work?

Yes, it worked.

Success!  So XenApp/XenDesktop 7.X will substitute the tokens with the LongCommandLine in the ICA file.  Excellent.  So now we just need Storefront to take the URI and add it to the LongCommandLine.  Storefront does not do this out of the box.

However, Citrix offers a couple of possible solutions to this problem (that I explored).

StoreFront WebAPI
StoreFront Store Customization SDK

What are these and how do they work?

StoreFront Web API

This API is billed as:

“Write a new Web UI or integrate StoreFront into your own Web portal”

We are going to need to either modify Storefront or write our own in order to take and parameters in the URI.  I decided to write our own.  Using the ApiExample.html in the WebAPI I added the following:

This looks into the URI and allows you to get the value of either query string.  In my example I am grabbing the values for “CTX_Application” and “NFuse_AppCommandLine”.

I then removed a bunch of the authentication portions from the ApiExample.html (I’ll be using an unauthenticated store).  In order to automatically select the specified application I added some javascript to check the resource list and get the launch URL:

There is a function within the ApiExample.html file that pulls the ICA file.  Could we take this and modify it before returning it with the LongCommandLine addition?  We have the NFuse_AppCommandLine now.

It turns out, you can. You can capture the ICA file as a variable and then using javascripts ‘.replace’ method you can modify the line so that it contains your string.  But how do you pass the ICA file to the system to launch?

This is how Citrix launches the ICA file in the ApiExample.html:

It generates a URL then sets it as the src in the iframe.  The actual result looks like this:

The ICA file is returned in this line:

When we capture the ICA file as a variable the only way that I’ve found you can reference it is via a blob.  What does the src path look like when do that?

Ok, this looks great!  I can create an ICA file than modify it all through WebAPI and return the ICA file to the browser for execution.  Does it work?

Yes and no. 🙁

It works in Chrome and Firefox, but IE doesn’t auto-launch.  It prompts to ‘save’ a file.  Why?  IE doesn’t support opening ‘non-standard’ blobs.  MS offers a method called “msSaveOrOpenBlob” which you can use instead, and this method then prompts for opening the blob.  This will work for opening the ICA file but now the end user requires an extra step.  So this won’t work.  It needs to be automatic like its supposed to be for a good experience.

So WebAPI appears to offer part of the solution.  We can capture the nFuse_AppCommandLine but we need to get it to LongCommandLine.

At this point I decided to look at the StoreFront Store Customization SDK.  It states it has this ability:

Post-Launch ICA file—use this to modify the generated ICA file. For example, use this to change ICA virtual channel parameters and prevent users from accessing their clipboard.

That sounds perfect!

StoreFront Store Customization SDK

The StoreFront store customization SDK bills one of its features as:

The Store Customization SDK allows you to apply custom logic to the process of displaying resources to users and to adjust launch parameters. For example, you can use the SDK to control which apps and desktops are displayed to users, to change ICA virtual channel parameters, or to modify access conditions through XenApp and XenDesktop policy selection.

I underlined the part that is important to me.  That’s what I want, exactly!  I want to adjust launch parameters.

To start, one of the tasks that I need is to take my nFuse_AppCommandLine and get it passed to the SDK.  The only way I’ve found to make this happen is to enable ‘forwardedHeaders‘ in your web.config file.

With this, you need to set your POST/GET Header on your request to Storefront to get this parameter passed to the SDK.  Here is how I setup my SDK:

Install the “Microsoft Visual Studio Community Edition” on a test StoreFront server.

 

Download the StoreCustomizationSDK and open the ‘Customization_Launch’ project file:

 

Right-click ‘Customization_Launch’ and select ‘Properties’.

 

Modify the Outpath in Build to the “%site%\bin” location.  This is NOT the %site%Web, but just the %site%.

Open the LaunchResultModifer.cs

 

Set a breakpoint somewhere in the file.

Build > Build ‘Customization_Launch’

Ensure the build was successful:

If you get an error message:

Just ignore it.  We’re compiling a library and not an executable and that’s why you get this message.

Now we connect to the debugger to the ‘Citrix Delivery Services Resources’.  Select ‘Attach to Process…’

Select the w3wp.exe process whose user name is ‘Citrix Delivery Services Resources’.  You may need to select ‘Show processes from all users’ and click ‘ Attach.

Click ‘Attach’.

Browse to your website and click to launch an icon:

Your debugger should pause at the breakpoint:

And you can inspect the values.

At this point I wasn’t interested in trying to re-write the ApiExample.html to get this testing underway, I instead used PowerShell to submit my POST’s and GET’s.  Remember, I’m using an unauthenticated store so I could cut down on the requests sent to StoreFront to get my apps.  I found a script from Ryan Butler and made some modifications to it.  I modified it to remove the parameters since I’m doing testing via hardcoding 🙂

Executing the powershell script and examining a custom variable with breakpoint shows us that our header was successfully passed into the file:

Awesome!  So can we do something with this?  Citrix has provided a function that does a find/replace of the value you want to modify.  To enable it, we need to add it the ICAFile.cs:

Browse to ICAFile.cs

 

At this point we can use the helper methods within the IcaFile.cs.  To do so, just add “using Examples.Helpers;” to the top of the file:

I cleaned out the HDXRouting out of the file, grabbed the nFuseAppCMDLine header, and then returned the result:

The result?

Success!  We’ve used the WebAPI and the StoreFront Customization SDK to supply a header, modify the ICA file, and return it with the value needed!  The ICA file returned works perfectly!  Ok, so I’m thinking this looks pretty good right?!  We just need to get the iframe to supply a custom header when the src gets updated.

Except I don’t think it’s possible to tag a custom header when you update a src in an iframe.   So then I started thinking maybe I can use a cookie!  If I can pass a cookie into the Storefront Customization SDK I could use that instead of the header.  Unfortunately, I do not see anyway to query a custom cookie or pass a custom cookie into the SDK.  Citrix appears to have (rightfully!) locked down Storefront that this doesn’t seem possible.  No custom cookies are passed into the httpcontext (that I can see).

So all this work appears to be for naught (outside of an exercise on how one could setup and customize Storefront).

 

But what about the Storefront feature “add shortcuts to websites“?

Although doing that will allow to launch an app via a URL, it doesn’t offer any method to pass values into the URL.  So it’s a non-starter as well.

Next will be a workaround/solution that appears to work…

Read More

ControlUp – Realtime Troubleshooting Example

2017-04-13
/ /
in Blog
/

I’ve been using a troubleshooting tool for some time and I can’t speak highly enough about.  It has saved myself so much time in finding/resolving and remediating issues.  I decided to capture myself utilizing the tool *in realtime* to remediate a reported issue that came in during a weekend call.  I was able to find the servers that had the issues within 5 mins and prevent them from hosting users (effectively remediating the issue).

The root of the issue is we had some servers reboot during the daylight savings time change and this messed up some of our scripts that were relying on some timing to load our AppV packages.  After receiving the call I started my recording.

Youtube link.  Please ensure to select the highest quality.

Without ControlUp, trying to search all the servers was a manual task or a scripted task that was executed serially.  These tasks, in the past, would take tons of time.  Generally, an issue like this would have taken an hour or so to resolve.  With ControlUp this was resolved in less than 5 minutes.

Higher Quality Version here.

Read More

Citrix XenApp Enumeration Performance – IMA vs FMA – oddities

2017-04-13
/ /
in Blog
/

During the course of load testing FMA to see how it compares with IMA in terms of performance I encountered some oddities with FMA.  It definitely is more efficient at enumerating XenApp applications compared to IMA.  But…  The differences in my testing may have been overstated.

During the load testing I captured some performance counters.  One of them, “Citrix XML Service – enumerate resources – Concurrent Transactions” seemed to line up almost perfectly with the load wcat was applying to the broker.  BUT…  It capped out.  It seemed to cap at 200-220 concurrent connections.

 

The slope of the increase in the concurrent transactions should have continued but it stops.   I had set a maximum of 1000 concurrent connections via WCAT but it stops then does this weird ‘stepping’.  When this limit appears to be reached, that appears to be when the “performance” of FMA succeeds IMA significantly.

(ORANGE is FMA, BLUE is IMA).

What I’m considering is that FMA is taking all requests up to that limit and then ‘queuing’ them.  I have no proof of that outside of the Perf Counter just hitting that ceiling.  And when the concurrent connections hit ~800 and the response time soared, was the broker actually working?

No.  It was not working.  The responses I got when the connections hit 800+ looked like this:

ErrorID ‘unspecified’.  Undoubtedly, it appears there is a timer somewhere that if the broker cannot respond to requests within a reasonable amount of time (20 seconds?) that it returns this error code.

So is the amount of processing the broker can handle a hardcoded limit?  I scanned the startup of the service with Procmon to see what registry keys/values the BrokerService was looking for.

I found a document from Citrix that details *some* of these registry keys.

There is a registry value that determines the maximum number of concurrent requests the FMA broker will process:

I added that key and restarted the service.  What was the impact?

16vCPU

Very positive.  The 800 concurrent limit was easily bypassed. Why is this limited to 500 by default?  Changing this value to 1000 had a massive improvement in terms of how many concurrent connections it could handle.  It pushed itself to 1350 concurrent connections before it broke.  If I had to push for a single tweak to FMA, this one might be it.  Mind you, the Performance Counter was still capping itself around 220 concurrent connections, so I’m not sure if that’s a limit of the counter or something else but it is VERY accurate until that point.  If I start my WCAT test and go to 50 the perf counter goes to 50.  If I choose 80, it goes to 80.  If I choose 173 it goes to exactly 173.  So it is odd.

During the course of my investigation I found a registry key that shows Citrix has decided any request over 20 seconds to the broker will fail.  This key is:

I suspect this is my “unspecified” error.  20 seconds is a really long time for a broker to respond to a request though, if you were a user this would suck.  However, is it better to increase this value?  I have it on my todo list to try and increase it for my testing to see what happens, but values like the Storefront or Netscaler XML requests to see if the brokers are responding are handicapped to 20 seconds with this value.  So it’s probably important to have them match to reduce any further timeout you may encounter.  For instance, it would not make sense to set a 60 second timeout in a Netscaler XML broker test if the broker is going to timeout at 20 seconds anyways.

Next will be testing this with Local Host Cache to see if this causes any impact.

Read More

Citrix XenApp Enumeration Performance – IMA vs FMA load testing

2017-04-12
/ /
in Blog
/

In the previous post we found IMA outperformed FMA in terms of an individual request for resources.  However, the 30ms difference would be imperceptible for a end user.  This post will focus on maximum load for an individual broker.  In the past we discovered that the IMA service is heavily dependent on the number of CPU’s.  The more CPU’s allocated to the machine hosting the IMA service, the more requests could be handled (supposedly to a max of 16 — the maxmium number for threads for IMA).

What I have configured for my testing is a XenApp 6.5 machine with 2 sockets and 1 vCPU’s for a total of 2 cores.  I have an identical machine configured for XenApp 7.13.

Doing my query directly to the 7.13 broker, I’ve discovered my request appears to be tied to the performance counter “Citrix XML Service – Transactions/sec – enumerate resources”.  I can visibly see how many requests are hitting the broker through my testing, although the numbers vs what I’m sending don’t necessarily line up.

For XenApp 6.5 the performance counter that appears to tie directly to my requests is the “Citrix MetaFrame Presentation Server – Filtered Application Enumerations/sec”.

Using WCAT to apply a load to XenApp 6.5 and 7.13 and measuring the per second results I should be able to see which is more efficient over a larger load.

My WCAT options for the FMA server will be:

and for IMA:

 

My “scenario” file (XML-Direct.ubr) looks like so:

The new FMA service has an additional counter called “Citrix XML Service – enumerate resources – Avg. Transaction Time” which actually reports back how long it took to execute the enumeration.

Just like my previous testing I’m going to test with 2, 4, 8 and now with 16 CPU.  Both machines are VM’s on the same cluster with the same specs for their host (Intel Xeon E5-2680 @ 2.70GHz).  A difference between the two VM’s is the XenDesktop 7.13 is a Server 2016 Standard platform.

2vCPU

IMA is in BLUE and FMA is in ORANGE.

Interesting results.  It appears XenDesktop FMA broker can handle higher loads with much greater efficiency compared to IMA.  And it does not appear to take much for FMA to flex some muscle and show that it can handle the same load faster than IMA.  FMA kept under our 5000ms target up to 180 concurrent connections where as IMA breaks that around 140 connections.  This initial data shows that FMA could handle approximately 23% more load.  Fairly impressive.

4vCPU

Again, FMA continues its dominance in enumeration performance.  The gap between them grows slightly to 26% at 400 concurrent connections.  At our target of under 5000ms, IMA breaks at 250 connections, FMA actually held until around 380 connections.

8vCPU

Again, FMA handles the additional load with the additional CPU’s without breaking a sweat.  It seemed to be unstoppable, keeping under 5000ms until about 655 connections.  IMA, on the other hand, exceeded 5000ms response time at 400 connections.  But I had something strange happen that I assumed to be my fault.  The FMA broker service suddenly spiked at 800 concurrent connections (this graph only shows up to 780) and its response time zoomed to 30-40 seconds.  I assumed this was a fault of mine and continued on, trying 16 CPU’s.

16vCPU

But this graph shows it pretty readily.  As soon as you cross 800 concurrent connections FMA pukes.  I didn’t scale the graph to show that spike, but it goes up to 40 seconds.  So there appears to be a pretty hard limit of <800 concurrent connections (600 would probably be a pretty safe buffer…).  If you exceed that limit your performance is going to tank, HARD.  IMA, however, pushes on.  With 16vCPU IMA didn’t break the 5000ms target until 570 concurrent connections.  FMA appeared to be handling it just fine until it exploded.

For this testing the FMA broker is set in ‘connection leasing’ mode.  Perhaps this is related to that?  My next test will be to set the broker to Local Host Cache mode, retest, and then simulate a DB failure and test the LHC and see how quickly it can respond.

Summary:

The FMA broker, when acting in a purely XenApp fashion works pretty well.  It handles loads faster than IMA, but apparently this is only true to an extreme rate.  You should not have more than 800 concurrent connections per broker.  <period>.   You should probably keep a target maximum of 600 to be safe.  And assign at least 4, but preferably 8 CPU’s if you can to your broker servers.  This appears to be the sweet spot.

The highest rate our organization sees is 600 concurrent connections but we spread this lead across 9 IMA brokers, and divide that geographically.  The highest consecutive load we’ve measured with this on any single broker is ~150 concurrent connections during a peak load period.  We target a response time of less than 5 seconds for any broker enumeration and it does appear we could handle this quite easily with FMA.  However, this does not take into account XenDesktop traffic which is ‘heavier’ than XenApp for the FMA.  When we get to a point that I can test XenDesktop load I will do so.  Until then, I am impressed with FMA’s XenApp enumeration performance (until breakage anyways).

Next up…  Oddities encountered in testing

Read More

Citrix XenApp Enumeration Performance – IMA vs FMA

2017-04-11
/ /
in Blog
/

We’re exploring upgrading our Citrix XenApp 6.5 environment to 7.X (currently 7.13) and we have some architecture decisions that are driven by the performance of the infrastructure components of XenApp.  In 6.5 these components are the “Citrix Independent Management Architecture” and in 7.13 this is the “Citrix Broker Service”.  The performance I’ll be measuring is how long it takes to enumerate applications for a user.  In XenApp 6.5 this is the most intensive task put on the broker.  I’ve taken our existing XenApp 6.5 TEST environment and “migrated” it to 7.X.  The details of the environment are 189 enabled applications with various security groups applied to each application.  The user I will be testing with has access to 55 of them.  What the broker/IMA service has to do when it receives the XML request is evaluate each application to see if the user has permissions and return the results.  The ‘request’ is slightly different to the broker vs the IMA.  This is what the FMA requests will look like:

 

And the IMA requests:

In our environment, we have measured a ‘peak’ load of 600 concurrent connections per second to our XenApp 6.5 IMA service.  We split this load over 7 servers and the load is load-balanced via Netscaler VIP’s.  This lessens the peak load to 85 concurrent connections per server per second.  What’s a “connection”?  A connection is a request to the IMA service and a response from it.  This would be considered a single connection in my definition:

 

This is a single request (in RED) and response (in BLUE).  No further follow up is required by the client.

I’m going to profile a single response and request to better understand the individual performance of each product.

This is what my network traffic will look like (on the 7.X broker service):

The total time between when the FMA broker receives a single request to beginning the response is:

Initial receipt of traffic at 37.567664-37.567827.
Response starts at 37.633986

Response ends at 37.634432.

Total time for FMA request for list of applications and the response for that list:

For IMA the total time between when the IMA service receives a single request to beginning response is:

Initial receipt of traffic at 38.359944-38.360198.
Response starts at 38.440197

Response ends at 38.450032.

Total time for IMA request for list of applications and the response for that list:

Why the size difference (18KB vs 24KB)?

Looking at the data returned from the FMA via the IMA shows there is a new field passed by the FMA broker as apart of ‘AppData’

These two lines add 61 bytes per application.  A standard application response is (with title) ~331 bytes per IMA application and ~400 bytes per FMA application.

However, these single request are exactly that.  Single.

In order to get a better feel I ran the requests continuously in a loop, sending a request the FMA than the IMA, delay 1 second, and resend.  This should get me a more accurate feel for the performance differences.  I ran this over a period of 10 minutes.  My results were:

 

IMA is faster by approx 30ms per request.

Next up is load testing IMA vs FMA.

Read More

Citrix Workspace Environment Manager – Limitations (part 1?)

2017-04-03
/ /
in Blog
/

In a brief evaluation of Citrix Workspace Environment Manager, I looked at the utility of the product to replace Group Policy Preferences (GPP) in a XenApp environment context.  For this I focused on replacing a set of registry keys we apply via GPP for our XenApp environment.  My results were not favorable for using WEM in this context for the Registry portion as WEM pushes processing of it’s entries into the ‘shell’ session.  For XenApp, the Shell session is typically applied quickly and so the application may launch without those keys present (which is bad — the application needs those keys present first).  So although logon times maybe reduced, this scenario does not work for the Registry portion.  We are still exploring the effects of WEM and whether some other components that operate synchronously within GPP are needed.  Can these components be moved to WEM?  One of the big ‘wins’ for this approach maybe Drive Mappings, which apply synchronously and requires the Drive Mappings to be processed before allowing a user to logon.  Moving this to WEM may be a win worth exploring… IF the application doesn’t require drive mappings before being launched.  But that’s for another article..

However, for the registry portion of WEM we did encounter a few ‘gotcha’s worth mentioning if you are going to use WEM.

WEM does not do ‘Registry Binary’ keys.

Well…  it says it does.  And it kind of does.  But odds are you are not going to get the results you expect.

Looking at a simple REG_BINARY key it contains data is displayed as ‘hexadecimal’ data.

If I want to use WEM to apply this key, I would create an entry within WEM:

HINT: it's not.

Does this look correct to you? I thought it looked correct to me.

 

However, when I apply this key I get a completely wrong result.  Why?  Because WEM is applying the hexadecimal code in the ASCII field of the REG_BINARY

In order to get WEM to apply the code properly we would need to copy/convert the ASCII from the REG_BINARY.

 

However, I have bad results doing so:

WEM with ASCII converted binary values applied.

 

Result:

This is closer but grossly wrong.

 

Why is this wrong?  WEM stores everything as XML.  And XML files do not like storing binary or non-ascii data.  WEM stores these values as their ASCII representations and not REG_BINARY representations so if your REG_BINARY (which, there’s a 99% chance) contains a non-ASCII character it will fail to apply the key properly.

 

Even worse, during my time fiddling with this, I BROKE WEM.

If the ASCII representation of “&amp;#x1” or whatever was set it caused WEM to crash when applying the values.  So REG_BINARY’s are completely out.  In my limited testing, we only had 1 REG_BINARY to apply, but in our environment we use GPP to apply 5 different REG_BINARY keys.  So using WEM for these applications is right out.  I filed and asked Citrix for a ‘feature enhancement’ to support applying REG_BINARY’s properly, but I was told this was operating as expected so I’m not holding my breathe but this does limit the utility of WEM.

 

 

Read More

Citrix Workspace Environment Manager vs. Group Policy Preferences – Performance Story

2017-03-10
/ /
in Blog
/

Part one: Citrix Workspace Environment Manager – First Impressions
Part two: Citrix Workspace Environment Manager – Second Impressions

If you’ve been following me, I’ve been exploring using WEM to apply some registry values instead of using Group Policy Preference.  WEM does things differently which requires some thinking on how to implement it.  The lack of a Boolean OR value is the biggest draw back, but it is possible to get around it, although our environment hasn’t required multiple AND OR AND OR AND OR statements, so all the settings migrations I have done were possible.

But the meat of this post is HOW quickly can WEM process registry entries vs. GPP.

In order to compare these two I’ve subscribed to an old standby — Procmon.  I logged into one of my Citrix servers with another account, and started procmon.  I then launched an application published on this server (notepad).  I used ControlUp to measure the performance of the actual logon and group policy extension processing.  The one we are particularly interested in is the Group Policy Registry entry.  This measures the performance of the Group Policy Registry portion:

 

Group Policy Registry executed in 3494ms.  However, I have two GPO objects with values that were evaluated by this client side extension.  For WEM I only migrated a single GPO so I’ll have to focus on that single one for the CSE.  To find the Group Policy engine, I used ControlUp to discover it via selecting svchost.exe’s processes and discovering them.  The PID was 1872:

One of the cool things about procmon is being able to suss out time stamps exactly.

For the Group Policy Registry CSE I can see it was activated at exactly 2:33:12.2056862.  From there it checks the group policy history for the XML file then compares it to the one in your sysvol:

With this particular policy, we actually check to see if it’s a 32bit or 64bit system, we check for various pieces of software to see if they’re installed or not and we then apply registry keys based on those results.  For instance:

We have a GPP Registry values that are set via some item-level-targetting that are dependent on whether PDF architect is installed or not.  You can literally see it check for PDF Architect and then set whatever values we determined need to be set by that result (ShowAssociationDialog,ShowPresentation, etc).

However cool this is, this GPO is not the one I want 🙂

I want the next GPO ({E6775312-…}).  This GPO is the one that I have converted to WEM as it only dealt with group membership.  WEM can filter on conditions like a file/folder exist but since I didn’t want to do another thousand or so registry entries I focused on the smaller GPO.

 

This is the real meat.  We can see the GPO I’m interested in started processing at 2:33:14:5252890.

And then completed at 2:33:15.2480580.

The CSE didn’t actually finish though, until 2:33:15.706579 :

 

It looks like it was finishing some RSOP stuff  (RSOPLogging is disabled, BTW) storing things in the user registry hive like GPOLink, GPOName, etc.  Either way, these actions add time to the CSE to complete.  The total time spent in the Group Policy Registry CSE was:

~3501ms.

The total time reported by ControlUp was 3494ms.  So I’m probably a bit off by the start/finish of the CSE but pretty goddamn close.  The real meat of the GPO Registry processing (eg, the GPO I was actually concerned about) was:

1181ms.

 

So how does WEM do?

One of the ways that WEM ‘helps’ counting stats is by pushing the processing into the user session where it can be processed asynchronously.  WEM creates a log in your %userprofile% folder that you can examine to see when it starts:

 

Unfortunately, WEM’s log isn’t very granular.  Procmon will fix that again 🙂

The entry I’m looking for in WEM is “MainController.ProcessRegistryValues()”.  This tells us when it starts doing the registry work:

The processing started after 3:34:39.  Procmon will help us zero in on that time:

We can see pretty clearly that it starts applying registry values at 3:34:39.4995477.

It completes at…

3:34:46.5753350.

Total time:

7076ms.

Hmmm.  About twice as slow.  It is certainly possible that the WMI queries are what is killing my performance, but without a way to check the group membership of the server the user is on, I am hobbled.  It’s possible that if we were implementing WEM from the get go we could think of a naming scheme that would work better for us, with the limitations of the wildcard (although I think it’s a bug in WEM as opposed to a poor naming scheme — just my opinion), but to rework our entire environment is not feasible.

In order to determine if WEM will perform better without the WQL queries, I manually edited the condition to be focused on ‘Computer Name Match’ and specified all the relevant servers.

The results?

Started processing: 12:37:56.029

Finished at 12:37:59.634

 

Total time: 3605ms.

So there is big savings without doing any WQL processing.

 

Final Results

 

 

But it still doesn’t compare to the Group Policy Preferences – Registry CSE.  The speed of the CSE is still faster.  Pretty significantly, actually.  And there are other considerations you need to consider for WEM as well.  It applies the registry values *after* you’ve logged in, whereas GPP does it before.  This allows WEM to operate asynchronously and should reduce logon times but there is a drawback.  And for us it’s a big drawback.  When it applies the registry values, for most of our apps they need to be in place *before* you launch the application.  So setting the values after or while an application is launching may lead to some inconsistent experiences.  For us, this caveat only applies to a XenApp environment.

When talking about a XenDesktop environment a premium is generally placed on getting to the desktop where a user has to navigate to an application to launch it, which would probably require enough time that WEM will be able to apply its required values before the user is able to launch their application.  In this scenario, saving 3-4 seconds of the user waiting for their desktop to appear are valuable and WEM (mostly) solves this issue by pushing it asynchronously to the shell.

For XenApp, we are still considering WEM to see how it performs with roaming users and having printers change for session reconnections; depending on their client name/ip space or some other variable.  That investigation will come in the future.  For now, it looks like we’re going to keep using Group Policy Preferences for our registry application for XenApp.

Stay tuned for some WEM gotcha’s I’ve learned doing this exercise.

 

Read More

Citrix Workspace Environment Manager – Second Impressions

2017-03-08
/ /
in Blog
/

Workspace Environment Manager does not do Boolean logic.  It only does AND statements.

Note: These conditions are AND statements, not OR statements. Adding multiple conditions will require all of them to trigger for the filter to be considered triggered.

So we have to reconsider how we will apply some settings.  An example:

How would you apply this setting without WEM?  The setting is targeted to a specific set of servers that we’ve put into a group, AND the user must be a member of at least 1 out of about 10 groups for it to apply.  And this is what we’ve set.  It seems fairly logical and the the way the targeting editor is setup it’s actually fairly easy to follow.

How do we configure this in WEM?

WEM works by applying settings based on your individual user account or a group containing users.  So each group needs to be added to WEM and then configured individually.  Alternatively, this can change your thinking and you can create groups that you want settings to apply towards.  For instance, in this example I only have two statements I need to be true.

The server must belong to a particular group.
The user must belong to at least one of several groups.

So I started by attempting to get to the result we wanted.

I exported the value as we wanted it from the registry and imported it into WEM:

Now to create our conditions. I want this registry action to only apply if you are on a specific server.  We add our servers into logical groups.  I attempted to set the filter condition to target the server with this specific AD group.  It turns out this was not possible with the builtin filter conditions.

The Active Directory filter conditions are only evaluated against the user and not the machine.  However, there is an option for ‘ComputerName Match’:

Our server naming convention is CTX (for Citrix) APP or silo name (eg, EPIC) ### (3 digit numerical designation) and the letter “T” at the end for a test server or no letter for a production server.  So our naming scheme would look like so:

CTXAPP301 – for a prod server
CTXAPP301T – for a test server

The guide for WEM does specify you can use a wild card “*” to allow arbitrary matches.  So I tested it and found it has limitations.

The wild card appears to only be valid if it’s at the beginning or end of the name.  For instance, setting the match string to:

CTXAPP3*

Will match both the test and prod servers.  However, setting CTXAPP3*T and the wildcard is ‘dropped’ from the search (according to the WEM logs).  So the search result is actually CTXAPP3T.  Which is utterly useless.  So a proper naming scheme is very important.  Unfortunately, this will not work for us unless we manually specify all of our computer names.  For example, we would have to do the following for this to work:

CTXAPP301T, CTXAPP302T, CTXAPP303T, etc.

Although this works, this is not workable.  Adding/removing servers would require us to keep this filter updated fairly constantly.  Is there another option?

Apparently, you can execute a WMI Query.  We can use this to see if the computer is a member of a group.  The command to do so is:

WEM supports variable replacement through a list of HASHTAGS

I modified my command as such:

Executing this command does result in checking to see if the computer is a member of a specific group.  The only caveat is the user account that logs on must have permissions within AD to check group memberships.  Which is usually granted.  And in my case, it is so this command works.

Next on the Group Policy Preference is the ‘AND – OR’.  We have the filter condition now that ensures these values will only be applied if the computer is in a certain group.  Now we need the value to apply if the user is a member of a certain group.

An easy solution might be to create a super-group “HungAppTimeout” or some such and add all the groups I want that value to apply to that group.  Another alternative is we can ‘configure’ each user group with the ‘server must belong to group’ filter and that should satisfy the same requirements.  I chose this route for our evaluation of WEM to avoid creating large swaths of groups simply for the purpose of applying a single registry entry.

Instead of doing the ‘OR’, we select the individual groups that this check would be against and actually specify that we apply this settings to that group.

To do that we add each group to the ‘Configured Users’:

And then for each group, under ‘Assignment’ we apply our setting with the filter:

And now each group has this value applied with the appropriate conditions.

Continuing, we have the following policy:

So this filtering is applied to a collection, not the individual settings.  The filtering checks to see if the computer is a member of a specific group of servers, and whether the user is a member of a specific group.

In order to accomplish this same result I have no choice but to create a parent group for the machines.  Instead of an ‘OR’ we create a new group and add both server groups within it.  This should result in the same effective ‘OR’ statement for the machine check, at least.  Then we apply all the settings to the specific groups so only they get the values applied.

In total we have 154 total individual registry entries we apply.

So how does it compare to Group Policy Preferences?

Stay Tuned 🙂

Read More

Citrix Workspace Environment Manager – First Impressions

2017-03-02
/ /
in Blog
/

Citrix Workspace Environment Manager can be used as a replacement for Active Directory (AD) Group Policy Preferences (GPP).  It does not deal with machine policies, however.  Because of this AD Group Policy Objects (GPO) are still required to apply policies to machines.  However, WEM’s goal isn’t to manipulate machine policies but to improve user logon times by replacing the user policy of an AD GPO.  A GPO has two different engines to apply settings.  A Registry Policy engine and the engine that drives “Client Side Extensions” (CSE).  The biggest time consumer of a GPO is processing the logic of a CSE or the action of the CSE.  I’ll look at each engine and what they mean for WEM.

Registry Extension

The first is the ‘Registry’ policy engine.  This engine is confusingly called the “Registry Extension” as opposed to the CSE “Group Policy Registry”.  The “Registry Extension” is engine applies registry settings set in via ‘Administrative Templates’.

These settings are ‘dumb’ in that there is no logic processing required.  When set to Enabled or Disabled whatever key needs to be set with that value is applied immediately.  Processing of this engine occurs very, very fast so migrating these policy settings would have minimal or no improvement to logon times (unless you have a ton of GPO’s apply and network latency becomes your primary blocker).

If you use ControlUp to Analyze GPO Extension Load Times it will display the Registry Extension and the Group Policy Registry CSE:

Client Side Extensions

However, CSE’s allow you to put complex logic and actions within that require processing to determine if a setting should be applied or how a settings should be applied.  One of the most powerful of these is the Registry CSE.  This CSE allows you to apply registry settings with Boolean logic and can be filtered on a huge number of variables.

All of this logic is stored in a XML document that is pulled when the group policy object is processed.  This file is located in “C:\ProgramData\Microsoft\Group Policy\History\GUID OF GPO\SID OF USER\Preferences\Registry”.

Parsing and executing the Boolean logic takes time.  This is where we would be hopeful that WEM can make this faster.  The processing of all this, in our existing environment consumes the majority of our logon time:

Migrating Group Policy Preferences to WEM

Looking at some of our Registry Preferences we’ll look at what is required to migrate it into WEM.

Basic settings “eg, ‘Always applied’”.

“Visual Effects”

These settings have no filters and are applied to all users.  To migrate them to WEM I’ve exported these values and set them into a registry file:

Switching to WEM I select ‘Actions’ then ‘Registry Entries’ and then I imported the registry file.

An interesting side note, it appears the import excluded the REG_BINARY.  However you can create the REG_BINARY via the GUI:

To set the Registry Entries I created a filter condition called “Always True”

And then created a rule “Always True”

We have a user group that encompasses all of our Citrix users upon which I added in ‘Configure Users’.  Then, during the assignment of the registry keys I selected the ‘Always True’ filter:

And now these registry keys have been migrated to WEM.  It would be nice to ‘Group’ these keys like you can do for a collection in Group Policy Preferences.  Without the ability to do so the name of the action becomes really important as it’s the only way you can filter:

Next I’ll look at replacing Group Policy Preferences that contain some boolean logic.

Read More