Blog

Citrix Storefront – Performance Testing and Tuning – Part 6 – Testing Peak Load

2017-06-13
/ /
in Blog
/

In part 5 I figured out our peak load against our existing infrastructure.  In this part I’m going to test against a Storefront server to see if it can handle the baseline load.

This is the counters used for PNA launches:

The loads to test are:

Peak Rates (in per second)
PNA Logon PNA App Launches Web Logon Web LaunchICA Connections per second
7.531 17.94 1.02 1.78 28.271

I am going to use WCAT to test PNA as WCAT is fairly easy to configure.  From the counters we can easily target “ICA Launches Calls / second” and “Find all resources Calls / second” (and divide by ~2.5) to get to our PNA load count.

Now, I can test the PNA portion with WCAT, which is good because it can maintain a nice, steady rate.  Web Logon’s and launches though, require a PowerShell script because of some cookie manipulation.

Testing a single Web Interface server, I tested just PNA app launches (20 per second) and our Web Interface server was unable to maintain that rate.  The maxmium number a single Web Interface server (4vCPU 12GB RAM) was ~15 application launches per second.  At that point my Powershell script was getting errors during its requests.

Since we load balance we could sustain ~30 PNA app launches per second with Web Interface.  How does Storefront do?  Can a single server maintain our peak load?

 


Ok, first off, I have to say I love it when the math adds up.

I was hoping for an average of 8 PNA app launches (ICA Launch Calls / second) and I got 8.6, and I was hoping for ~19 “Find All resources Calls / second” and I got 17.7.  So I believe I’m pretty close to simulating my real peak load traffic.

So we can see the load testing definitely put an impact on Storefront server.  We had some peaks, which varied between requesting icons and requesting the ICA file.  Getting the Config.xml file didn’t move the needle neither did find PRELAUNCH applications.  I’m not entirely sure why we have some spikes but overall I think I’m pretty satisfied with this performance.

When does Storefront break (in my environment)?

I set WCAT to do 1000 connections. At 32 connections per second my response time was starting to increase dramatically.  This was with a 4vCPU system.

Monitoring my XML brokers showed they were not breaking a sweat.  So the bottle neck has to be Storefront.  Looking at my CPU graphs showed this to be a CPU bottle neck.

So what happens if I up the vCPU to 8?  When does the bottleneck push to the brokers or no longer become the CPU?

 

At 8vCPU StoreFront scaled nearly linearly.  At 32 connections per second It was handling requests at around 7 seconds.  At 38 connections per second though, it was starting to choke, going at about 10 seconds per request.

 

And this is the last up I can do.  I moved the StoreFront up to 16 vCPU with a whole host for itself to prevent CPU oversubscription.

This configuration enabled us to handle 64 connections per second before consistently breaking the 10 second limit.

 

 

If we set a target of a “fail” when the response time exceeds 10 seconds this is what my maximum number of requests can be per vCPU:

So Storefront seems to scale pretty well with CPU, with the biggest gain going from 4vCPU to 8vCPU.  At 4vCPU a single StoreFront server should be able to handle our entire peak load, albeit barely.  At 8vCPU a single server should have more than enough capacity to handle almost double our peak loading.

Read More

Citrix Storefront – Performance Testing and Tuning – Part 5 – Establishing a Peak Load

2017-06-06
/ /
in Blog
/

In part 1 I setup a script to load test our Citrix Storefront server, as a user logging in through the website.  Part 2 was users connecting via PNA.  In this part I’m going to examine our existing load.  In order to do so, I only want to count the following users:

  1. Users must have logged into Web Interface.  This ensures that we count users who have actually logged in to launch an application.  Some of our users may have the Web Interface set as their homepage so this way we can avoid idle user counts.
  2. Users who have launched an application.  I’m interested in the users who have actually launched an application via Web Interface.
  3. PNA connections
  4. PNA launches

I want to get these user counts sorted via time in order to determine our peak load.

To get these counts I decided to key into some unique web URL’s or resources.  For #1, I decided to search the IIS logs for the “Logoff.png”.

This icon will only be pulled down for a user who has logged into the Web Interface.

For #2, I decided to look at the IIS logs and search for “launch.ica”.  This will get us the application launches via the sites.

For #3, I decided to look at the IIS logs and search for “config.xml”.  This file is pulled each time PNA is launched or a user logs onto the machine.

For #4, I decided to look at the IIS logs and search for “launch.aspx”.

Users must have logged into Web Interface.

For #1 I created a powershell script to parse the IIS log file and export my data:

I then opened it in Excel and created time ranges per-hour and then counted how many of each item was found in that range.  For the Logoff.PNG it was accessed like this:

The peak count was 2775 logons over a course of an hour.  This equals ~45 logons per minute or 1 logon every 1.29 seconds.

In order to get even more precise, I broke down the hour that had the 2775 logons and looked at how many logons occurred over 15 minute time spans.

The peak rate over a 15 min span was:

1 logon every 0.974 second. (1.02 logons per second)


Users who have launched an application

 

For users who launch an application we are at 7593 launches as a peak over an hour.

The peak access to Web Interface is 2.10 launches per second.

If we do a failover strategy to ensure , say in the future, we unified our architecture to a single location (or single cloud?) what would our combined rate be?

1 launch every 0.561 second. (1.78 launches per second)

Citrix has Storefront guidance that says it can handle connections/users in the ten’s of thousands.  I thought we were a large organization, but apparently Storefront has been designed for something even bigger! (Or web-based front ends are rarely the bottle neck…)

So these numbers are purely for logons through the web site.  But previous testing for PNA functionality showed that it’s a *much* heavier traffic to Web Interface.  Why is the traffic heavier for PNA?  Because each time a user logs on to a workstation it kicks in.  Whereas accessing the web interface requires some work the user must do, PNA is automatic.  It’s also more convenient for users as they prefer to launch desktop/start menu icons.


PNA connections

How am I going to determine our concurrent connection rate for PNA?  This is what a PNA connection looks like:

What I’m going to do is query the IIS logs for ‘config.xml‘, as that file gets called only once when Receiver is launched or upon login.  Once I get that I can search for ‘enum.aspx’ command that has been called and divide by the number of config.xml’s.  This should get me the number of icon requests per user.

After doing the math I found Receiver makes 2.63 enum.aspx request for every config.xml requests.

Lastly,

My thought process here is once we get our peak ‘Receiver’ connection rate to the Storefront server we can create another script to simulate it.  I know WCAT can do this as no funny cookie business occurs for Receiver.

So what does the PNA Receiver numbers look like?

We have a peak PNA connection rate of 7.531 config.xml connections per second.  Multiplying by the additional 2.63 enum.aspx requests give us a total of 19.806 connections per second for a peak rate.

For PNA application launches, our peak rate is 30,075 launches in a single hour.  This was measured by parsing for “launch.aspx”.  Breaking this hour down to 15 minute intervals and a single 15min window contained 53% of the launches.  For our peak 15 minute window this is 16148 launches.  This gives us a total of 17.94 application launches per second.

Peak Rates (in per second)
PNA Logons PNA App Launches Web Logons Web LaunchICA Connections per second
7.531 17.94 1.02 1.78 28.271

 

 

Read More

AppV5 – System Environment Variables Gotcha’s

2017-06-05
/ /
in Blog
/

We had an application that has its environment configured via ‘system environment variables’.  System environment variables differ from ‘user’ environment variables in that they are global so all users see them, and in a native Windows environment, they are stored in a different location; but still in the registry.  So it seemed, like everything else App-V, these values would be captured (they are just registry values).  Imagine our surprise when they were *not* captured.

The tech who was working on this thought this must be a mistake, exported the registry keys he needed into a file and imported them into the package:

 

 

 

He created a startup script that would modify the environment variables to whatever environment the user wanted to launch (Test, Prod, Dev, etc.).

BUT, no matter what was set the application always launched with the parameters it was sequenced with.  In this case it was always PROD.  We would put a ‘SET’ or ECHO ‘%TT_SYS1%’ or some such into the batch file, after the ‘SET/SETX’ commands we could see the variables were different.  They were what we expected them to be.  But launching a new process with these changed variables resulted with the new process inheriting the original environment variables.  So it was always PROD.  Now, these are SYSTEM environment variables and Microsoft released a utility, setx, which you can use to manipulate them.  This utility still did not work as expected, the variables were not being retained by a child process.

“By default, a child process inherits the environment variables of its parent process.”

When this was brought to my attention I suggested we explore ‘Exporting the manifest and seeing what was stored there’. Changing the registry entries in the package did NOTHING.  So I suspected they weren’t being used at all…

What we found is when we extracted the manifest file:

We saw where the environment variables where being stored.  By deleting this section and reimporting the manifest file we found that we could set environment variables and they would correctly inherit from the parent process from then on.  But if you do NOT delete this section of the Manifest file, then whatever is set for that AppV session will be reset for any new processes created in the bubble.


 

Final Analysis:

AppV5 treats Environment Variables differently then one would expect from a native system.  My expectation when I enter the bubble is an environment that I could manipulate and then have those changes persist for the duration of that session.  Instead, AppV environment variables are reset for EVERY process in the bubble.  Although you can manipulate them in the current process, creating a new process (cmd.exe/powershell.exe/whatever.exe) results in those changes being reset for variables specified in the manifest file.

Read More

Citrix Storefront – Performance Testing and Tuning – Part 4 – PerfMon Counters, PNA Logon

2017-06-02
/ /
in Blog
/

In looking at the performance of the Citrix Storefront Server, one of the thing I want is to understand what Storefront is doing at each stage of a session life cycle.  There are two session-types that I’m curious about, a web browser based connection and a PNA connection.  I’m going to examine a PNA based connection in this post.  See this post for a browser based examination.

Using my powershell script to simulate a user connection I put ‘pauses’ between each stage.  I then setup Perfmon to capture counters from the following objects:

Read More

Citrix Storefront – Performance Testing and Tuning – Part 3 – PerfMon Counters, browser logon

2017-06-01
/ /
in Blog
/

In looking at the performance of the Citrix Storefront Server, one of the thing I want is to understand what Storefront is doing at each stage of a session life cycle.  There are two session-types that I’m curious about, a web browser based connection and a PNA connection.  I’m going to examine a web browser based connection first.

Using my powershell script to simulate a user connection I put ‘pauses’ between each stage.  I then setup Perfmon to capture counters from the following objects:

I’ve added 6 farms to the storefront server to examine the load “in a real world” environment as Storefront does do some magic with concurrent enumeration.  What I love that Citrix has done, is actually time the transactions AND gives you the ‘rate’ the transactions are occurring at.  This will make it much easier to baseline what your load is vs how I had to do it to measure the ‘rates’ for Web Interface.


$stage = “Initial Connection”
$store = “http://storefront.bottheory.local/Citrix/StoreWeb/”

So for a web browser based connection, I ‘connected’ to the site and saw nothing.  Storefront does very little work for the initial connection.


$stage = “Client Configuration”
$store + “Home/Configuration”

Result:

This request is to get the client configuration.  Essentially, it pulls down an xml file from the store containing important links to various locations in Storefront. Again, no perf counter seems to occur and it doesn’t seem to generate any load.


$stage = “Get Authentication Methods”
$store + “Resources/List”

 

So this command starts to get a little exciting.  It’s a bit of a misnomer to call it “Resources/List” as it doesn’t actually return resources, yet.  It returns the challenge to logon.  We get our first Peformance Counter hit.

The four counters that register are

“Citrix Receiver for Web\List resources Calls / second”
“Citrix Receiver for Web\List resources Average Time (microseconds)”
“Citrix Delivery Services Web Application(dazzleresources:list:get:_citrix_store)\Controller Action Calls / second”
“Citrix Delivery Services Web Application(dazzleresources:list:get:_citrix_store)\Controller Action Average Time (Microseconds)”
Results:
List resources Calls / second: 1
List resources Average Time (microseconds): 3,714
(dazzleresources:list:get:_citrix_store)\Controller Action Average Time (Microseconds): 607
(dazzleresources:list:get:_citrix_store)\Controller Action Calls / second: 1


$stage = “Get Auth Methods”
$store + “Authentication/GetAuthMethods”

Response:

Another exciting call.  This call hits these four performance counters:

“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Calls / second”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds)”
“Citrix Delivery Services Web Application(protocols:choices:post:_citrix_authentication)\Controller Action Calls / second”
“Citrix Delivery Services Web Application(protocols:choices:post:_citrix_authentication)\Controller Action Average Time (Microseconds)”

Results:
(protocols:choices:post:_citrix_authentication)\Controller Action Calls / second: 1
(protocols:choices:post:_citrix_authentication)\Controller Action Average Time (Microseconds) : 488
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Calls / second: 1
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds): 138

The Controller Action Calls /second was “1”, obviously I’m doing just a single test.  For the action average time I got 711 microseconds.  Extremely fast.


$stage = “Domain Pass-Through and Smart Card Authentication”
$store + “DomainPassthroughAuth/Login”

Response:

Two more perf counters hit (rate + time):

Citrix Delivery Services Web Application(citrixfederationauthentication:authenticate:post:_citrix_authentication)\Controller Action Calls / second
Citrix Delivery Services Web Application(citrixfederationauthentication:authenticate:post:_citrix_authentication)\Controller Action Average Time (Microseconds)

Results:
(citrixfederationauthentication:authenticate:post:_citrix_authentication)\Controller Action Calls / second: 1
(citrixfederationauthentication:authenticate:post:_citrix_authentication)\Controller Action Average Time (Microseconds): 4,850


$stage = “Resource Enumeration”
$store + “Resources/List”

Response:

This call returns a list of all the resources for which you have access.  The counters important to this call are:
“Citrix Receiver for Web\List resources Average Time (microseconds)”
“Citrix Dazzle Resources Controller\List Whole Body Calls / second”
“Citrix Dazzle Resources Controller\List Whole Body Average Time (Microseconds)”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Calls / second”
“Citrix Delivery Services Web Application(dazzleresources:list:get:_citrix_store)\Controller Action Calls / second”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds)”
“Citrix Delivery Services Web Application(dazzleresources:list:get:_citrix_store)\Controller Action Average Time (Microseconds)”
“Citrix Receiver for Web\List resources Calls / second”
“Citrix Xml Service Communication(10.10.10.11)\Network Traffic Calls / second”
“Citrix Xml Service Communication(10.10.10.11)\Network Traffic Average Time (Microseconds)”
“Citrix Xml Service Communication(10.10.10.12)\Network Traffic Calls / second”
“Citrix Xml Service Communication(10.10.10.12)\Network Traffic Average Time (Microseconds)”
“Citrix Xml Service Communication(10.10.10.13)\Network Traffic Calls / second”
“Citrix Xml Service Communication(10.10.10.13)\Network Traffic Average Time (Microseconds)”
“Citrix Xml Service Communication(10.10.10.14)\Network Traffic Calls / second”
“Citrix Xml Service Communication(10.10.10.14)\Network Traffic Average Time (Microseconds)”
“Citrix Xml Service Communication(xenapp5.bottheory.local)\Network Traffic Calls / second”
“Citrix Xml Service Communication(xenapp5.bottheory.local)\Network Traffic Average Time (Microseconds)”
“Citrix Xml Service Communication(xenapp65t.bottheory.local)\Network Traffic Calls / second”
“Citrix Xml Service Communication(xenapp65t.bottheory.local)\Network Traffic Average Time (Microseconds)”

We have lots of exciting things happening here.  We’re reaching back and querying the XML brokers, some kind of authentication occurs, etc.

The results of these counters were:

Citrix Receiver for Web\List resources Average Time (microseconds) : 249,172
Citrix Dazzle Resources Controller\List Whole Body Calls / second : 1
Citrix Dazzle Resources Controller\List Whole Body Average Time (Microseconds) : 182,206
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Calls / second : 1
(dazzleresources:list:get:_citrix_store)\Controller Action Calls / second : 2
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds) : 4,549
(dazzleresources:list:get:_citrix_store)\Controller Action Average Time (Microseconds) : 8,391
Citrix Receiver for Web\List resources Calls / second : 1
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Average Time (Microseconds) : 0
Citrix Xml Service Communication(10.10.10.12)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(10.10.10.12)\Network Traffic Average Time (Microseconds) : 36,820
Citrix Xml Service Communication(10.10.10.13)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(10.10.10.13)\Network Traffic Average Time (Microseconds) : 37,622
Citrix Xml Service Communication(10.10.10.14)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(10.10.10.14)\Network Traffic Average Time (Microseconds) : 43,121
Citrix Xml Service Communication(xenapp5.bottheory.local)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(xenapp5.bottheory.local)\Network Traffic Average Time (Microseconds) : 95,575
Citrix Xml Service Communication(xenapp65t.bottheory.local)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(xenapp65t.bottheory.local)\Network Traffic Average Time (Microseconds) : 149,888

With these counters we can see how long it took the brokers to respond.  My broker that has a few hundred applications took the longest, followed by a broker of an older farm with dozens of apps, then three where only a single app each is published from them.  I got a 0.0 for enumeration for one service.  I’m not sure what that’s about (maybe it exceeded enumeration timeout and is now excluded?).  Total time to list resources was 0.249 seconds.


$stage = “Get User Name”
$store + “Authentication/GetUserName”

Response:

This call simply returns your display name (or logon name if no display name is specified).

Two counters were hit:
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action  Calls / second”
“Citrix Delivery Services Web Application(tokenvalidation:tokenvalidation:get:_citrix_authentication)\Controller Action Average Time (Microseconds)”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds)”

The results of these counters were:
(tokenvalidation:tokenvalidation:get:_citrix_authentication)\Controller Action  Calls / second : 2
(authservice:tokenservices:post:_citrix_authentication)\Controller Action  Calls / second : 1
(tokenvalidation:tokenvalidation:get:_citrix_authentication)\Controller Action Average Time (Microseconds) : 2,200
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds) : 4,551


$stage = “AllowSelfServiceAccountManagement”
$store + “ExplicitAuth/AllowSelfServiceAccountManagement”

Response:

This call appears to return True or False.

“Citrix Delivery Services Web Application(selfserviceaccountmanagement:allowselfserviceaccountmanagement:get:_citrix_authentication)\Controller Action  Calls / second”
“Citrix Delivery Services Web Application(selfserviceaccountmanagement:allowselfserviceaccountmanagement:get:_citrix_authentication)\Controller Action Average Time (Microseconds)”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds)”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action  Calls / second”

The results of these counters were:
(selfserviceaccountmanagement:allowselfserviceaccountmanagement:get:_citrix_authentication)\Controller Action  Calls / second : 2
(selfserviceaccountmanagement:allowselfserviceaccountmanagement:get:_citrix_authentication)\Controller Action Average Time (Microseconds) : 2,247
(authservice:tokenservices:post:_citrix_authentication)\Controller Action  Calls / second : 1
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds) : 6,014


 

$stage = “Download all the icons”
$store + $iconurl

Response:

<downloads all icon files>

This call hits the following counters:

“Citrix Receiver for Web\Get icon Average Time (Microseconds)”
“Citrix Receiver for Web\Get icon Calls / second”

The results of these counters were:

Citrix Receiver for Web\Get icon Average Time (Microseconds) : 285.351
Citrix Receiver for Web\Get icon Calls / second : 152.259


$stage = “ICA Launch.GetLaunchStatus”
$store + $appToLaunch.launchstatusurl

Response:

This call hits the following counters:

Citrix Receiver for Web\Get launch status Calls / second
Citrix Receiver for Web\Get launch status Average Time (Microseconds)
Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Calls / second
Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Average Time (Microseconds)
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Calls / second
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Average Time (Microseconds)

The results of these counters were:

Citrix Receiver for Web\Get launch status Calls / second : 1
Citrix Receiver for Web\Get launch status Average Time (Microseconds) : 152,721
Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Calls / second : 1
Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Average Time (Microseconds) : 5,091
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Calls / second:  2
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Average Time (Microseconds) : 65,709

Storefront will only fetch the status for the application from the broker that the application belongs.  If I run this query multiple times, hitting different brokers, the XML service communication populates more information.  The information here can be useful for you to find which broker is getting hit the most by looking at the “Network Traffic Calls / second” and you can find whether it’s overloaded by the “Network Traffic Average Time” being a number that grows as your load grows.  As with everything, it’s probably best to get some “base” numbers when load is low and then again when load is high to get your range.


$stage = “ICA Launch.LaunchIca”
$store + $appToLaunch.launchurl

Response:

We get an ICA file.  The performance counters that are important:

Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Calls / second : 1
Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Average Time (Microseconds) : 5,453
Citrix Receiver for Web\Get Ica file Calls / second : 1
Citrix Receiver for Web\Get Ica file Average Time (Microseconds) : 1,255,237
Citrix Xml Service Communication(10.10.10.14)\Network Traffic Calls / second : 2
Citrix Xml Service Communication(10.10.10.14)\Network Traffic Average Time (Microseconds) : 530,153

 


Final results:

Taking all the operations and the time it took, I sorted the list according to duration.

The following operations list the total duration of that specific task:

“Get Ica file”
“List resources”
“List Whole Body”
“Get launch status”
Excluding them shows us the sub-tasks that actually consumed the majority of the time spent:

The XML communications is what consumes the majority of the processing time.  The items starting in ‘brackets ()’ are the Citrix Storefront processing time that it spent.  Being in the single “thousands” of microseconds is super fast.  This essentially shows us that it takes next to no time at all for Storefront to execute the tasks that it needs.


Analysis:

Storefront has some great new performance counters that can help you determine your pain points.  Access to Citrix is usually talked about in terms of ‘rate’ so the per-second and average time counters are excellent in helping determine what is the component taking so long to do whatever task.  The breaking down of how long individual XML brokers take is a HUGE plus as you should be able to accurately gauge which Citrix FARM is now causing slowness in logon, enumeration or launching for your users.

Read More

Citrix Storefront – Performance and Tuning – Part 2 – PNA Traffic simulation

2017-05-30
/ /
in Blog
/

In part 1 I created a script to simulate users going through their web browser, logging in to Storefront, and launching an application.

In this post I’m going to simulate “PNA traffic”.  “PNA” is the icons delivered to your desktop or start menu via Citrix Receiver.

When connecting to Storefront or Web Interface the following can occur:

  1. This is your first connection and you need to download/cache the icons from the server
  2. This is a subsequent connection and you only need to validate you have all the icons
  3. You connect and PNA discovers you are missing an icon or have a new application assigned and proceeds to download any missing/different icons

All connections from PNA to Storefront/Web Interface start with “config.xml” .  From there, it queries for icons and downloads them.  This is what each of the 3 scenarios described look like:

 

The first time you connect via PNA

 

Subsequent connections after icons have been cached.

 

2 new applications published to the user

 

The first connection “config.xml” returns an xml file detailing the properties of the store.  Show icons on desktop/start menu? What are the URL’s for resource enumeration, reconnection, change password, etc.?  What logon methods are available?

All these questions and more are answered by the config.xml.

The second connection “enum.aspx” with a body size of 175 bytes is a POST request for the PRELAUNCH stage of PNA.  Are there any applications configured for PRELAUNCH?  POST request looks like this:

If you don’t have any applications configured for PRELAUNCH you receive a response like this:

The next enum.aspx request is a POST request that asks for all the icons:

And the response is a rather large one (1,758,490 bytes in my example) with application properties and details.  Part of the reason it’s so large is the icon data is stored as ASCII.  Here’s a small snippet of just 2 applications:

The subsequent POST request for icons (most roughly ~21,000 bytes in size) look like this:

With the response being:

Lastly, application launches over PNA are a POST request that look like this:

The content of that POST request looks like this:

And the response is a ICA file that Receiver executes.


With each of these requests we can see that PNA sends your username/password/domain to the server to get the relevant information.  This means we can script these requests using WCAT as the data being POST’ed is actually static.

Using the wcat scenario creator, download the WCATScenarioGenerator.dll and save it to: “C:\Program Files (x86)\Fiddler2\Scripts”

 

Open Fiddler.  Capture PNA operation by closing Citrix Receiver and reopening it.  Logon (if necessary) and launch an application.  If you want a more intensive logon, ensure your icon cache is cleared so that it forces new icons to be downloaded.  This will be the ‘scenario’ that we will be executing against.  Clean up any requests that don’t go to your web interface/storefront.  My Fiddler looks like this:

Select your items and in the black bar enter “wcat addtrans” and hit enter.

In the same bar enter “wcat save” and hit enter.

 

This will create a file called “fiddler.wcat” in your “C:\Program Files (x86)\Fiddler2” folder.

This file will actually require some cleanup as the WCAT scenario generator captures things literally and the scenario file requires escaping of double-quotes.

So in the wcat file, lines like this:

 

Need to be modified to look like this:

 

To execute our script, install wcat and launch the controller via the command line like so:

What each of these settings means:

-t = scenario file to use
-f = settings file to use (just using default)
-s = server to target
-p = port to connect
-c = clients that need to connect (this is the number of wcclient.exe’s that will connect to the controller)
-v = number of virtual clients that will connect
-w = warmup time (clients and virtual clients will connect in the amount they have / warmup time)
-u = duration for full load (all clients should be connected)
-n = cooldown, clients start disconnecting (opposite of warmup)

For my example, if I use my parameters for the controller then I need to start 10 clients be executing:

(Win7.bottheory.local is running the controller).

This command can be executed 10 times on the same box, even the same one as the controller, or on some random combination of 1-10 different boxes as long as you match the number of “clients” to the controller “client” excepted number of connections.


Lastly, here is a powershell script to measure the performance of each action of the PNA request.  You will need to modify it for your domain/(webinterface/storefront URL)/username/password.  This script will measure the amount of time each stage takes and the total, overall time and saves it as a CSV.

Here’s an example of the CSV output:

Some example commands:

Examples of “stress component” output:

And the script:

 

 

Read More

Citrix Storefront – Performance Testing and Tuning – Part 1

2017-05-24
/ /
in Blog
/

I’ve used the Web Capacity Analysis Tool (WCAT) in the past to measure the performance of Citrix Web Interface with some success.  So I thought this tool would be perfect for load testing Storefront.  I loaded up Fiddler, set up the WCAT extension, captured a web logon and then application launch.  The whole process looked like this:

Pretty simple right?

I logged on with Domain Passthrough authentication, I clicked on an application and I was done.  I have my customizations added in as well.  But what is actually happening?  How long does each step of the actual process take?  To determine this answer, we go to Fiddler to capture our flow so we can examine each step.

I truncated all the icon resource calling.  You can see it calls around 120 individual icon URLs.  I have my two custom ‘helpers’ (ADInfo and LogonType) that determine logon preference and whether we should get workspace control enabled or disabled (LogonType.aspx and GroupMembership.aspx).

So the actual calls to Storefront revolve around 7 unique queries to the Storefront server.  They are:

Fortunately, Citrix has documented how these need to be configured to successfully call these services.

With all this setup, I took my scenario file and executed it.  Nothing appeared to happen.  I broke down the scenario file into the individual calls and found where it was breaking.  This Citrix documentation explains it nicely:

Cross-site request forgery token

To protect against cross-site request forgery (CSRF) attacks, the Web Proxy APIs require that a CSRF token be supplied by the client, unless specified otherwise. This is a random string generated by the Web Proxy for the duration of the session and communicated to the client using a session cookie. Clients must read the value of this cookie and send it back to the Web Proxy in most API calls, as either the value of a header named Csrf-Token (note the hyphen) for POST requests, or as the value of a query string parameter named CsrfToken for GET requests.

 

The part in bold and underlined, is troublesome with WCAT.  WCAT does not appear to have this ability (read the value of a cookie and set a header to send it back to the web proxy).  What Storefront does, is send back a set-cookie back to the client which WCAT has no problem with…  but the data Storefront sends back is multiple values within that set-cookie command.  And this is a problem.

This is supposed to take this ‘Set-Cookie’ command and set two different values:

Cookie Value
CsrfToken FE2148E03989CB263CCD82A5888BF039
path /Citrix/StoreWeb/

But what WCAT does is create a single cookie that looks like this:

Cookie Value
cookie CsrfToken=FE2148E03989CB263CCD82A5888BF039; path=/Citrix/StoreWeb/;

WCAT creates a cookie with the entirety of a string value instead of separating them out.  Is there a way to parse this Set-Cookie command so that these are stored correctly?

Unfortunately, I was not able to find a way to do this with WCAT.

However, we can use Powershell to accomplish this job.  Ryan Butler has created a script to query the Storefront services to generate an ICA file.  This script is about 90% of what I need, however I’m not interested in doing an explicit logon, I want to do Domain Passthrough (integrateWindows) authentication, and I want to simulate the process as was captured by Fiddler, so I’ll be calling the additional services (GetUserName, AllowSelfServiceAccountManagement, etc.) and capture the time required for each section.

My Storefront Logon/Stress testing script:

 

And this is the output:

 

We can now run this script concurrently to simulate multiple clients.  Or we could run it with a command like so:

To stress an individual component.  By stressing the individual component we can actually determine which process on the Storefront server deals with which service.  So which components equals which service?

Get Auth Methods:

This component stresses “Citrix Receiver for Web”


Domain Pass-Through and Smart Card Authentication:

This component stresses “Citrix Receiver for Web” and “Citrix Delivery Services Authentication”


Resource Enumeration:

This component stresses “Citrix Receiver for Web” and “Citrix Delivery Services Resources”


Get User Name:

This component stresses “Citrix Receiver for Web”


AllowSelfServiceAccountManagement:

This component stresses “Citrix Receiver for Web” and “Citrix Delivery Services Authentication”


“GetLaunchStatus”

This component stresses “Citrix Delivery Services Resources”


LaunchIca

This component stresses “Citrix Delivery Services Resources”


Now that we have this script and we can see the where individual components cause stress, we can begin to push our Storefront server to find its limits and how the different configurations are impacted.  I’ll write up my findings on the limits of Storefront next.

Read More

Citrix Storefront – Adventures in customization – Query group membership from aspx

2017-05-15
/ /
in Blog
/

I’ve written a script that can tie into your environment for Storefront or other web service.  This is preferable over the powershell HTTP listener (IMHO) because it can just run on the IIS server and doesn’t need to rely on any external program/service.  It’s a simple script to pull out whether a user is a part of a group or not.  However, it does require impersonation to be able query Active Directory if your environment does not allow anonymous queries (I believe most do not).  Impersonation will make the request come from the machine account, which typically does have authorization to query AD.

In order to set this up, I’ve created a web application in IIS with impersonation set.