PowerShell

Citrix Provisioning Service – Network Service Starting/Stopping services remotely

2018-05-02
/ /
in Blog
/

Citrix Provisioning Services has a feature within the “Provisioning Services Console” that allows you to stop/restart/start the streaming service on another server:

 

This feature worked with Server 2008R2 but with 2012R2 and greater it stopped working.  Citrix partially identified the issue here:

 

I was exploring starting and stopping the streaming service on other PVS servers from the Console and I found this information was incorrect.  Adding the NetworkService does NOT enable the streaming service to be stop/started/restarted from other machines.  The reason is the NETWORKSERVICE is a LOCAL account on the machine itself.  When it attempts to reach out and communicate with another system it is translated into a proper SID, which matches the machine account.  Since that SID communicating across the wire does not have access to the service you get a failure.

In order to fix this properly we can add either the machine account permissions for each PVS Server on each service OR we can add all machine accounts into a security group and add that as permissions to manipulate the service on each PVS Server.

I created a PowerShell script to enable easily add a group, user or machine account to the Streaming Service.  It will also list all the permissions:

An example adding a Group to the permissions to the service:

And now we can start the service remotely:

 

In order to get this working entirely I recommend the following steps:

  1. Create a Group (eg, “CTX.Servers.ProvisioningServiceServer”)
  2. Add all the PVS Machine Accounts into that group
  3. Reboot your PVS server to gain that group membership token
  4. Run the powershell script on each machine to add the group permission to the streaming service:
  5. Done!

And now the script:

 

Read More

Check for NUMA within a VM guest with Powershell

2017-07-06
/ /
in Blog
/

I have only tested this in VMWare, but it seems to work.  I have shamelessly stole the main code from here:

 

I was going to use this to detect NUMA and configure a network adapter RSS based upon NUMA configuration…   But I’m getting lazy and am going to ignore it.  But I don’t want this code to go away.  So here it is.  A way to detect if you are on a NUMA system in a guest VM in Powershell.

If you have greater than “1” for the NumaNode count then NUMA is present.

CoreInfo.exe result on the same system:

 

Read More

Remove Ghost devices natively with Powershell

2017-06-28
/ /
in Blog
/

We’ve been looking at using Base Image Scripting Framework (BISF) as our new preparation and personalization platform for our PVS farm.  In our current world we have a bunch of features and tweaks that are not apart of BISF so I’ve been writing some additions so that we achieve feature parity.

One of the features I wanted to take a good hard look at was removing ‘Ghost’ devices — or devices that aren’t present in your system.  Ghost devices look like this:

Our current world accomplishes this task by using a script written by Simon Price and devcon.exe.  There is nothing wrong with this method per se but BISF is native Powershell and I want to stick to that without having outside dependencies.  Can this requirement be achieved with nothing but PowerShell?  Fortunately, google came to rescue and pointed me here.  A script written by Alexander Boersch got me 80% of the way there (whoo hoo!).  He wrote the method and ability to access setupapi.dll which gives us the functions and methods necessary to query and manipulate devices in Device Manager.  PowerShell is supposed to have the ability to do C# code natively and his example was perfect for taking me where I needed to go.

How does it work?  What’s the output?

 


 

notice the “filter match” text

 


 

And a brief video of it in action:

 

Lastly, the script:

 

 

Read More

Citrix Storefront – Performance Testing and Tuning – Part 6 – Testing Peak Load

2017-06-13
/ /
in Blog
/

In part 5 I figured out our peak load against our existing infrastructure.  In this part I’m going to test against a Storefront server to see if it can handle the baseline load.

This is the counters used for PNA launches:

The loads to test are:

Peak Rates (in per second)
PNA Logon PNA App Launches Web Logon Web LaunchICA Connections per second
7.531 17.94 1.02 1.78 28.271

I am going to use WCAT to test PNA as WCAT is fairly easy to configure.  From the counters we can easily target “ICA Launches Calls / second” and “Find all resources Calls / second” (and divide by ~2.5) to get to our PNA load count.

Now, I can test the PNA portion with WCAT, which is good because it can maintain a nice, steady rate.  Web Logon’s and launches though, require a PowerShell script because of some cookie manipulation.

Testing a single Web Interface server, I tested just PNA app launches (20 per second) and our Web Interface server was unable to maintain that rate.  The maxmium number a single Web Interface server (4vCPU 12GB RAM) was ~15 application launches per second.  At that point my Powershell script was getting errors during its requests.

Since we load balance we could sustain ~30 PNA app launches per second with Web Interface.  How does Storefront do?  Can a single server maintain our peak load?

 


Ok, first off, I have to say I love it when the math adds up.

I was hoping for an average of 8 PNA app launches (ICA Launch Calls / second) and I got 8.6, and I was hoping for ~19 “Find All resources Calls / second” and I got 17.7.  So I believe I’m pretty close to simulating my real peak load traffic.

So we can see the load testing definitely put an impact on Storefront server.  We had some peaks, which varied between requesting icons and requesting the ICA file.  Getting the Config.xml file didn’t move the needle neither did find PRELAUNCH applications.  I’m not entirely sure why we have some spikes but overall I think I’m pretty satisfied with this performance.

When does Storefront break (in my environment)?

I set WCAT to do 1000 connections. At 32 connections per second my response time was starting to increase dramatically.  This was with a 4vCPU system.

Monitoring my XML brokers showed they were not breaking a sweat.  So the bottle neck has to be Storefront.  Looking at my CPU graphs showed this to be a CPU bottle neck.

So what happens if I up the vCPU to 8?  When does the bottleneck push to the brokers or no longer become the CPU?

 

At 8vCPU StoreFront scaled nearly linearly.  At 32 connections per second It was handling requests at around 7 seconds.  At 38 connections per second though, it was starting to choke, going at about 10 seconds per request.

 

And this is the last up I can do.  I moved the StoreFront up to 16 vCPU with a whole host for itself to prevent CPU oversubscription.

This configuration enabled us to handle 64 connections per second before consistently breaking the 10 second limit.

 

 

If we set a target of a “fail” when the response time exceeds 10 seconds this is what my maximum number of requests can be per vCPU:

So Storefront seems to scale pretty well with CPU, with the biggest gain going from 4vCPU to 8vCPU.  At 4vCPU a single StoreFront server should be able to handle our entire peak load, albeit barely.  At 8vCPU a single server should have more than enough capacity to handle almost double our peak loading.

Read More

Citrix Storefront – Performance Testing and Tuning – Part 5 – Establishing a Peak Load

2017-06-06
/ /
in Blog
/

In part 1 I setup a script to load test our Citrix Storefront server, as a user logging in through the website.  Part 2 was users connecting via PNA.  In this part I’m going to examine our existing load.  In order to do so, I only want to count the following users:

  1. Users must have logged into Web Interface.  This ensures that we count users who have actually logged in to launch an application.  Some of our users may have the Web Interface set as their homepage so this way we can avoid idle user counts.
  2. Users who have launched an application.  I’m interested in the users who have actually launched an application via Web Interface.
  3. PNA connections
  4. PNA launches

I want to get these user counts sorted via time in order to determine our peak load.

To get these counts I decided to key into some unique web URL’s or resources.  For #1, I decided to search the IIS logs for the “Logoff.png”.

This icon will only be pulled down for a user who has logged into the Web Interface.

For #2, I decided to look at the IIS logs and search for “launch.ica”.  This will get us the application launches via the sites.

For #3, I decided to look at the IIS logs and search for “config.xml”.  This file is pulled each time PNA is launched or a user logs onto the machine.

For #4, I decided to look at the IIS logs and search for “launch.aspx”.

Users must have logged into Web Interface.

For #1 I created a powershell script to parse the IIS log file and export my data:

I then opened it in Excel and created time ranges per-hour and then counted how many of each item was found in that range.  For the Logoff.PNG it was accessed like this:

The peak count was 2775 logons over a course of an hour.  This equals ~45 logons per minute or 1 logon every 1.29 seconds.

In order to get even more precise, I broke down the hour that had the 2775 logons and looked at how many logons occurred over 15 minute time spans.

The peak rate over a 15 min span was:

1 logon every 0.974 second. (1.02 logons per second)


Users who have launched an application

 

For users who launch an application we are at 7593 launches as a peak over an hour.

The peak access to Web Interface is 2.10 launches per second.

If we do a failover strategy to ensure , say in the future, we unified our architecture to a single location (or single cloud?) what would our combined rate be?

1 launch every 0.561 second. (1.78 launches per second)

Citrix has Storefront guidance that says it can handle connections/users in the ten’s of thousands.  I thought we were a large organization, but apparently Storefront has been designed for something even bigger! (Or web-based front ends are rarely the bottle neck…)

So these numbers are purely for logons through the web site.  But previous testing for PNA functionality showed that it’s a *much* heavier traffic to Web Interface.  Why is the traffic heavier for PNA?  Because each time a user logs on to a workstation it kicks in.  Whereas accessing the web interface requires some work the user must do, PNA is automatic.  It’s also more convenient for users as they prefer to launch desktop/start menu icons.


PNA connections

How am I going to determine our concurrent connection rate for PNA?  This is what a PNA connection looks like:

What I’m going to do is query the IIS logs for ‘config.xml‘, as that file gets called only once when Receiver is launched or upon login.  Once I get that I can search for ‘enum.aspx’ command that has been called and divide by the number of config.xml’s.  This should get me the number of icon requests per user.

After doing the math I found Receiver makes 2.63 enum.aspx request for every config.xml requests.

Lastly,

My thought process here is once we get our peak ‘Receiver’ connection rate to the Storefront server we can create another script to simulate it.  I know WCAT can do this as no funny cookie business occurs for Receiver.

So what does the PNA Receiver numbers look like?

We have a peak PNA connection rate of 7.531 config.xml connections per second.  Multiplying by the additional 2.63 enum.aspx requests give us a total of 19.806 connections per second for a peak rate.

For PNA application launches, our peak rate is 30,075 launches in a single hour.  This was measured by parsing for “launch.aspx”.  Breaking this hour down to 15 minute intervals and a single 15min window contained 53% of the launches.  For our peak 15 minute window this is 16148 launches.  This gives us a total of 17.94 application launches per second.

Peak Rates (in per second)
PNA Logons PNA App Launches Web Logons Web LaunchICA Connections per second
7.531 17.94 1.02 1.78 28.271

 

 

Read More

Citrix Storefront – Performance Testing and Tuning – Part 4 – PerfMon Counters, PNA Logon

2017-06-02
/ /
in Blog
/

In looking at the performance of the Citrix Storefront Server, one of the thing I want is to understand what Storefront is doing at each stage of a session life cycle.  There are two session-types that I’m curious about, a web browser based connection and a PNA connection.  I’m going to examine a PNA based connection in this post.  See this post for a browser based examination.

Using my powershell script to simulate a user connection I put ‘pauses’ between each stage.  I then setup Perfmon to capture counters from the following objects:

Read More

Citrix Storefront – Performance Testing and Tuning – Part 3 – PerfMon Counters, browser logon

2017-06-01
/ /
in Blog
/

In looking at the performance of the Citrix Storefront Server, one of the thing I want is to understand what Storefront is doing at each stage of a session life cycle.  There are two session-types that I’m curious about, a web browser based connection and a PNA connection.  I’m going to examine a web browser based connection first.

Using my powershell script to simulate a user connection I put ‘pauses’ between each stage.  I then setup Perfmon to capture counters from the following objects:

I’ve added 6 farms to the storefront server to examine the load “in a real world” environment as Storefront does do some magic with concurrent enumeration.  What I love that Citrix has done, is actually time the transactions AND gives you the ‘rate’ the transactions are occurring at.  This will make it much easier to baseline what your load is vs how I had to do it to measure the ‘rates’ for Web Interface.


$stage = “Initial Connection”
$store = “http://storefront.bottheory.local/Citrix/StoreWeb/”

So for a web browser based connection, I ‘connected’ to the site and saw nothing.  Storefront does very little work for the initial connection.


$stage = “Client Configuration”
$store + “Home/Configuration”

Result:

This request is to get the client configuration.  Essentially, it pulls down an xml file from the store containing important links to various locations in Storefront. Again, no perf counter seems to occur and it doesn’t seem to generate any load.


$stage = “Get Authentication Methods”
$store + “Resources/List”

 

So this command starts to get a little exciting.  It’s a bit of a misnomer to call it “Resources/List” as it doesn’t actually return resources, yet.  It returns the challenge to logon.  We get our first Peformance Counter hit.

The four counters that register are

“Citrix Receiver for Web\List resources Calls / second”
“Citrix Receiver for Web\List resources Average Time (microseconds)”
“Citrix Delivery Services Web Application(dazzleresources:list:get:_citrix_store)\Controller Action Calls / second”
“Citrix Delivery Services Web Application(dazzleresources:list:get:_citrix_store)\Controller Action Average Time (Microseconds)”
Results:
List resources Calls / second: 1
List resources Average Time (microseconds): 3,714
(dazzleresources:list:get:_citrix_store)\Controller Action Average Time (Microseconds): 607
(dazzleresources:list:get:_citrix_store)\Controller Action Calls / second: 1


$stage = “Get Auth Methods”
$store + “Authentication/GetAuthMethods”

Response:

Another exciting call.  This call hits these four performance counters:

“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Calls / second”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds)”
“Citrix Delivery Services Web Application(protocols:choices:post:_citrix_authentication)\Controller Action Calls / second”
“Citrix Delivery Services Web Application(protocols:choices:post:_citrix_authentication)\Controller Action Average Time (Microseconds)”

Results:
(protocols:choices:post:_citrix_authentication)\Controller Action Calls / second: 1
(protocols:choices:post:_citrix_authentication)\Controller Action Average Time (Microseconds) : 488
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Calls / second: 1
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds): 138

The Controller Action Calls /second was “1”, obviously I’m doing just a single test.  For the action average time I got 711 microseconds.  Extremely fast.


$stage = “Domain Pass-Through and Smart Card Authentication”
$store + “DomainPassthroughAuth/Login”

Response:

Two more perf counters hit (rate + time):

Citrix Delivery Services Web Application(citrixfederationauthentication:authenticate:post:_citrix_authentication)\Controller Action Calls / second
Citrix Delivery Services Web Application(citrixfederationauthentication:authenticate:post:_citrix_authentication)\Controller Action Average Time (Microseconds)

Results:
(citrixfederationauthentication:authenticate:post:_citrix_authentication)\Controller Action Calls / second: 1
(citrixfederationauthentication:authenticate:post:_citrix_authentication)\Controller Action Average Time (Microseconds): 4,850


$stage = “Resource Enumeration”
$store + “Resources/List”

Response:

This call returns a list of all the resources for which you have access.  The counters important to this call are:
“Citrix Receiver for Web\List resources Average Time (microseconds)”
“Citrix Dazzle Resources Controller\List Whole Body Calls / second”
“Citrix Dazzle Resources Controller\List Whole Body Average Time (Microseconds)”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Calls / second”
“Citrix Delivery Services Web Application(dazzleresources:list:get:_citrix_store)\Controller Action Calls / second”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds)”
“Citrix Delivery Services Web Application(dazzleresources:list:get:_citrix_store)\Controller Action Average Time (Microseconds)”
“Citrix Receiver for Web\List resources Calls / second”
“Citrix Xml Service Communication(10.10.10.11)\Network Traffic Calls / second”
“Citrix Xml Service Communication(10.10.10.11)\Network Traffic Average Time (Microseconds)”
“Citrix Xml Service Communication(10.10.10.12)\Network Traffic Calls / second”
“Citrix Xml Service Communication(10.10.10.12)\Network Traffic Average Time (Microseconds)”
“Citrix Xml Service Communication(10.10.10.13)\Network Traffic Calls / second”
“Citrix Xml Service Communication(10.10.10.13)\Network Traffic Average Time (Microseconds)”
“Citrix Xml Service Communication(10.10.10.14)\Network Traffic Calls / second”
“Citrix Xml Service Communication(10.10.10.14)\Network Traffic Average Time (Microseconds)”
“Citrix Xml Service Communication(xenapp5.bottheory.local)\Network Traffic Calls / second”
“Citrix Xml Service Communication(xenapp5.bottheory.local)\Network Traffic Average Time (Microseconds)”
“Citrix Xml Service Communication(xenapp65t.bottheory.local)\Network Traffic Calls / second”
“Citrix Xml Service Communication(xenapp65t.bottheory.local)\Network Traffic Average Time (Microseconds)”

We have lots of exciting things happening here.  We’re reaching back and querying the XML brokers, some kind of authentication occurs, etc.

The results of these counters were:

Citrix Receiver for Web\List resources Average Time (microseconds) : 249,172
Citrix Dazzle Resources Controller\List Whole Body Calls / second : 1
Citrix Dazzle Resources Controller\List Whole Body Average Time (Microseconds) : 182,206
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Calls / second : 1
(dazzleresources:list:get:_citrix_store)\Controller Action Calls / second : 2
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds) : 4,549
(dazzleresources:list:get:_citrix_store)\Controller Action Average Time (Microseconds) : 8,391
Citrix Receiver for Web\List resources Calls / second : 1
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Average Time (Microseconds) : 0
Citrix Xml Service Communication(10.10.10.12)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(10.10.10.12)\Network Traffic Average Time (Microseconds) : 36,820
Citrix Xml Service Communication(10.10.10.13)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(10.10.10.13)\Network Traffic Average Time (Microseconds) : 37,622
Citrix Xml Service Communication(10.10.10.14)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(10.10.10.14)\Network Traffic Average Time (Microseconds) : 43,121
Citrix Xml Service Communication(xenapp5.bottheory.local)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(xenapp5.bottheory.local)\Network Traffic Average Time (Microseconds) : 95,575
Citrix Xml Service Communication(xenapp65t.bottheory.local)\Network Traffic Calls / second : 1
Citrix Xml Service Communication(xenapp65t.bottheory.local)\Network Traffic Average Time (Microseconds) : 149,888

With these counters we can see how long it took the brokers to respond.  My broker that has a few hundred applications took the longest, followed by a broker of an older farm with dozens of apps, then three where only a single app each is published from them.  I got a 0.0 for enumeration for one service.  I’m not sure what that’s about (maybe it exceeded enumeration timeout and is now excluded?).  Total time to list resources was 0.249 seconds.


$stage = “Get User Name”
$store + “Authentication/GetUserName”

Response:

This call simply returns your display name (or logon name if no display name is specified).

Two counters were hit:
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action  Calls / second”
“Citrix Delivery Services Web Application(tokenvalidation:tokenvalidation:get:_citrix_authentication)\Controller Action Average Time (Microseconds)”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds)”

The results of these counters were:
(tokenvalidation:tokenvalidation:get:_citrix_authentication)\Controller Action  Calls / second : 2
(authservice:tokenservices:post:_citrix_authentication)\Controller Action  Calls / second : 1
(tokenvalidation:tokenvalidation:get:_citrix_authentication)\Controller Action Average Time (Microseconds) : 2,200
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds) : 4,551


$stage = “AllowSelfServiceAccountManagement”
$store + “ExplicitAuth/AllowSelfServiceAccountManagement”

Response:

This call appears to return True or False.

“Citrix Delivery Services Web Application(selfserviceaccountmanagement:allowselfserviceaccountmanagement:get:_citrix_authentication)\Controller Action  Calls / second”
“Citrix Delivery Services Web Application(selfserviceaccountmanagement:allowselfserviceaccountmanagement:get:_citrix_authentication)\Controller Action Average Time (Microseconds)”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds)”
“Citrix Delivery Services Web Application(authservice:tokenservices:post:_citrix_authentication)\Controller Action  Calls / second”

The results of these counters were:
(selfserviceaccountmanagement:allowselfserviceaccountmanagement:get:_citrix_authentication)\Controller Action  Calls / second : 2
(selfserviceaccountmanagement:allowselfserviceaccountmanagement:get:_citrix_authentication)\Controller Action Average Time (Microseconds) : 2,247
(authservice:tokenservices:post:_citrix_authentication)\Controller Action  Calls / second : 1
(authservice:tokenservices:post:_citrix_authentication)\Controller Action Average Time (Microseconds) : 6,014


 

$stage = “Download all the icons”
$store + $iconurl

Response:

<downloads all icon files>

This call hits the following counters:

“Citrix Receiver for Web\Get icon Average Time (Microseconds)”
“Citrix Receiver for Web\Get icon Calls / second”

The results of these counters were:

Citrix Receiver for Web\Get icon Average Time (Microseconds) : 285.351
Citrix Receiver for Web\Get icon Calls / second : 152.259


$stage = “ICA Launch.GetLaunchStatus”
$store + $appToLaunch.launchstatusurl

Response:

This call hits the following counters:

Citrix Receiver for Web\Get launch status Calls / second
Citrix Receiver for Web\Get launch status Average Time (Microseconds)
Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Calls / second
Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Average Time (Microseconds)
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Calls / second
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Average Time (Microseconds)

The results of these counters were:

Citrix Receiver for Web\Get launch status Calls / second : 1
Citrix Receiver for Web\Get launch status Average Time (Microseconds) : 152,721
Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Calls / second : 1
Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Average Time (Microseconds) : 5,091
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Calls / second:  2
Citrix Xml Service Communication(10.10.10.11)\Network Traffic Average Time (Microseconds) : 65,709

Storefront will only fetch the status for the application from the broker that the application belongs.  If I run this query multiple times, hitting different brokers, the XML service communication populates more information.  The information here can be useful for you to find which broker is getting hit the most by looking at the “Network Traffic Calls / second” and you can find whether it’s overloaded by the “Network Traffic Average Time” being a number that grows as your load grows.  As with everything, it’s probably best to get some “base” numbers when load is low and then again when load is high to get your range.


$stage = “ICA Launch.LaunchIca”
$store + $appToLaunch.launchurl

Response:

We get an ICA file.  The performance counters that are important:

Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Calls / second : 1
Citrix Delivery Services Web Application(dazzleresources:launchica:post:_citrix_store)\Controller Action Average Time (Microseconds) : 5,453
Citrix Receiver for Web\Get Ica file Calls / second : 1
Citrix Receiver for Web\Get Ica file Average Time (Microseconds) : 1,255,237
Citrix Xml Service Communication(10.10.10.14)\Network Traffic Calls / second : 2
Citrix Xml Service Communication(10.10.10.14)\Network Traffic Average Time (Microseconds) : 530,153

 


Final results:

Taking all the operations and the time it took, I sorted the list according to duration.

The following operations list the total duration of that specific task:

“Get Ica file”
“List resources”
“List Whole Body”
“Get launch status”
Excluding them shows us the sub-tasks that actually consumed the majority of the time spent:

The XML communications is what consumes the majority of the processing time.  The items starting in ‘brackets ()’ are the Citrix Storefront processing time that it spent.  Being in the single “thousands” of microseconds is super fast.  This essentially shows us that it takes next to no time at all for Storefront to execute the tasks that it needs.


Analysis:

Storefront has some great new performance counters that can help you determine your pain points.  Access to Citrix is usually talked about in terms of ‘rate’ so the per-second and average time counters are excellent in helping determine what is the component taking so long to do whatever task.  The breaking down of how long individual XML brokers take is a HUGE plus as you should be able to accurately gauge which Citrix FARM is now causing slowness in logon, enumeration or launching for your users.

Read More

Citrix Storefront – Performance and Tuning – Part 2 – PNA Traffic simulation

2017-05-30
/ /
in Blog
/

In part 1 I created a script to simulate users going through their web browser, logging in to Storefront, and launching an application.

In this post I’m going to simulate “PNA traffic”.  “PNA” is the icons delivered to your desktop or start menu via Citrix Receiver.

When connecting to Storefront or Web Interface the following can occur:

  1. This is your first connection and you need to download/cache the icons from the server
  2. This is a subsequent connection and you only need to validate you have all the icons
  3. You connect and PNA discovers you are missing an icon or have a new application assigned and proceeds to download any missing/different icons

All connections from PNA to Storefront/Web Interface start with “config.xml” .  From there, it queries for icons and downloads them.  This is what each of the 3 scenarios described look like:

 

The first time you connect via PNA

 

Subsequent connections after icons have been cached.

 

2 new applications published to the user

 

The first connection “config.xml” returns an xml file detailing the properties of the store.  Show icons on desktop/start menu? What are the URL’s for resource enumeration, reconnection, change password, etc.?  What logon methods are available?

All these questions and more are answered by the config.xml.

The second connection “enum.aspx” with a body size of 175 bytes is a POST request for the PRELAUNCH stage of PNA.  Are there any applications configured for PRELAUNCH?  POST request looks like this:

If you don’t have any applications configured for PRELAUNCH you receive a response like this:

The next enum.aspx request is a POST request that asks for all the icons:

And the response is a rather large one (1,758,490 bytes in my example) with application properties and details.  Part of the reason it’s so large is the icon data is stored as ASCII.  Here’s a small snippet of just 2 applications:

The subsequent POST request for icons (most roughly ~21,000 bytes in size) look like this:

With the response being:

Lastly, application launches over PNA are a POST request that look like this:

The content of that POST request looks like this:

And the response is a ICA file that Receiver executes.


With each of these requests we can see that PNA sends your username/password/domain to the server to get the relevant information.  This means we can script these requests using WCAT as the data being POST’ed is actually static.

Using the wcat scenario creator, download the WCATScenarioGenerator.dll and save it to: “C:\Program Files (x86)\Fiddler2\Scripts”

 

Open Fiddler.  Capture PNA operation by closing Citrix Receiver and reopening it.  Logon (if necessary) and launch an application.  If you want a more intensive logon, ensure your icon cache is cleared so that it forces new icons to be downloaded.  This will be the ‘scenario’ that we will be executing against.  Clean up any requests that don’t go to your web interface/storefront.  My Fiddler looks like this:

Select your items and in the black bar enter “wcat addtrans” and hit enter.

In the same bar enter “wcat save” and hit enter.

 

This will create a file called “fiddler.wcat” in your “C:\Program Files (x86)\Fiddler2” folder.

This file will actually require some cleanup as the WCAT scenario generator captures things literally and the scenario file requires escaping of double-quotes.

So in the wcat file, lines like this:

 

Need to be modified to look like this:

 

To execute our script, install wcat and launch the controller via the command line like so:

What each of these settings means:

-t = scenario file to use
-f = settings file to use (just using default)
-s = server to target
-p = port to connect
-c = clients that need to connect (this is the number of wcclient.exe’s that will connect to the controller)
-v = number of virtual clients that will connect
-w = warmup time (clients and virtual clients will connect in the amount they have / warmup time)
-u = duration for full load (all clients should be connected)
-n = cooldown, clients start disconnecting (opposite of warmup)

For my example, if I use my parameters for the controller then I need to start 10 clients be executing:

(Win7.bottheory.local is running the controller).

This command can be executed 10 times on the same box, even the same one as the controller, or on some random combination of 1-10 different boxes as long as you match the number of “clients” to the controller “client” excepted number of connections.


Lastly, here is a powershell script to measure the performance of each action of the PNA request.  You will need to modify it for your domain/(webinterface/storefront URL)/username/password.  This script will measure the amount of time each stage takes and the total, overall time and saves it as a CSV.

Here’s an example of the CSV output:

Some example commands:

Examples of “stress component” output:

And the script: