scripting

Printers and their impact on logon duration

2019-03-05
/ /
in Blog
/

40 second logons.  100 second logons.  200 second logons.  

Our users had an average logon time of 8-11 seconds.  But some users were hitting 40-200 seconds.  Users were frustrated and calling the help desk.

Why?

I did not want to individually examine all of the users whom had a long logon duration.  There was dozens of dozens of users, but after sampling 5 of them, the root cause was consistent.  Printers were causing long logons.  

What I found was users were encountering the long logons on specific workstations.  Further investigation revealed these workstations had globally mapped printers.  Some workstations had dozens of printers.  Some of the printers were long gone, having been moved/re-IP’ed, shutdown, or re-provisioned and the entry in the workstation not updated.  Citrix, depending on Receiver version, attempts to do a check to validate the printer’s operation before mapping and this would wait for a network timeout (or a crash of Receiver).  When I stopped the print spooler on the local workstation, logons came back down to the proper range.

What I sought to do was create something to make identifying this root cause easier, quicker, and with more specificity — in other words was there a particular printer causing the delay?

Citrix Printing Methods

Citrix has two methods of printing.  “Direct connection” and local printing.  Direct connection has the potential for larger impact on logon duration as it executes additional steps and actions on the Citrix server.

To enable “Direct Connection”, Citrix has a policy “Direct connection to print servers”.  By default, this policy is enabled.

With this policy enabled, it changes the behavior of how Citrix connects printers you have mapped locally as network printers.  Citrix has a diagram here:

But I feel this is missing temporal information.  I’ve created a video to highlight the steps.

Direct Connection Process

Why is this important?

Citrix has an option that is an absolute requirement for some applications.  
“Wait for printers to be created (server desktop)” (Virtual Apps and Desktops 7.x) or “Start this application without waiting for printers to be created. (Unchecked)” (Citrix XenApp 6.5).

This policy needs to be enabled for numerous applications that require a specific printer is present before the application starts.  This is usually due to applications that have pre-defined printers like label printers and do a check on app launch.  If you have an application like that you probably  have this feature enabled.

Citrix has many options for managing printers and they can impact your logon process.

Focusing on Direct Connection being enabled, if “Wait for printers to be created (server desktop)” policy is also enabled, care MUST be taken to minimize logon times.

Direct connection does something that can have a very adverse affect on logon time that is enabled by default.  When it connects directly to the print server it will check and install the print driver on the Citrix server.  The installation activity is visible via process monitor.

The DrvInst.exe process is the installation of printer drivers

The installation of printer driver, by default, will only occur if the driver is inbox or prestaged on the Citrix server.  But the check to see if a print driver needs to be installed occurs every time a session is created.  This behavior can be changed by policy.

Why does this impact logon times?  Driver installation can take a while, especially if there is communication issues between the Citrix server and print server.  Or if the print server is under stress and just simply slow to respond.  Or if the driver package that needs to be loaded is especially large, has lots of forms or page formats, multiple trays, etc.  This all adds up.

How can I tell what the impact of printers might be on Logon Duration?

I’ve been working on updating a Script-Based Action (SBA) originally posted by Guy Leech on the Citrix blogs.  This enhancement to the SBA will output more information breaking down what’s consuming your logon times.  Specifically, this tackles how long printers take in the ControlUp column: “Logon Duration – Other”

I’ve created a video showing how to enable the additional logging and how to run the new Script Based Action and what the output looks like:

What’s new in the output?

For the individual printers, direct connection printers now show two lines, the Driver load time and Printer load/connection time. 

The driver load time is titled with Driver and then the UNC path to the printer. 

The Printer and UNC path shows the the actual connection, printer configuration and establishment time.  All Direct Connection printers will be shown with UNC paths.

Citrix Universal Print Driver (UPD) mapped printers using the Citrix Universal Print Driver (print jobs that get sent back to the client for processing) are shown with their “Friendly Name” and do not have a matching “Driver” component as no driver loading is required.  In the example output above, “\\printsrv\HP Color LaserJet 5550” is a direct connection printer and “Zebra R110Xi HF (300 dpi)” is a Citrix UPD printer.

Here is an example of the full output:

The “Connect to Printers” is a sub-phase under “Pre-Shell (Userinit)“.  Each “Printer:” and “Driver:” is a sub-phase under “Connect to Printers“.  In this example screenshot, “Pre-Shell (Userinit)” was consuming 117.6 seconds of a 133.2 second logon duration.  Prior to this script that was all that was known.  Now we can see that connecting to printers consumes 97.7 of those 117.6 seconds!

Awesome!  What’s the catch?

In order to generate this information, some more verbose logging is required on the Citrix server side.   I specifically worked to ensure that no 3rd party utilities would be required so we simply need to enable 4 additional features.

Command Line Audit
PrintServer/Operational Logs
Audit Process Creation/Termination

I’ve written a batch script to enable these features:

Easy!

Wait a minute!  Windows Server 2008R2 doesn’t have Command Line Auditing!

Good catch!  Windows Server 2008R2 doesn’t have command line auditing so it will miss the driver load information.  There is no way around this using native tools, but modification could be done to use a 3rd party tool like sysmon.  The output on a 2008R2 system would show the following (with all other features enabled).

Awesome!  Let me at this script!

This script is available on the ControlUp site here.  If you have ControlUp you can run simply add it via the “Script Based Actions” button and further derive more information on your users logon performance!  

Last Gotcha

In some instances the “Connect to Printers phase” may start before “Pre-Shell”. This has been observed and is actually happening! The operating system is attempting to start *some* printer events asynchronously in certain situations and so the connect to Printers may start before userinit, but overall connecting to printers may still be blocking logons while it trys and completes.

Steve Elgan kindly pointed out that it’s important you size your event logs to ensure the data is present for the users whom you want to monitor. If your event logs are sized too small than the data being queried by the script won’t be present. So ensure you have your Security and the Printer event logs sized large enough to capture all of this data! For an idea of scale, you can look at your log size and the oldest entry there. If your log is 1MB in size and your oldest entry is from an hour ago, than sizing your log to 24MB should capture about a day’s worth of information.

Hope this helps you reduce your long logon durations!

Read More

Citrix Logon Simulator’s – Part 2

2019-02-06
/ /
in Blog
/

In my previous post I was looking at utilizing a Logon Simulator to setup some proactive monitoring of a Citrix environment.  I setup some goals for myself:

  1. Minimize the number of VM’s to run the robots
  2. As little resource consumption as possible
  3. Still provide operational alerts
  4. Operate on-premise

I want the footprint of these robots to be tiny.  This must done on Server Core.

I want to run multiple instances of the logon simulator concurrently.

I need to be able to test “Stores” that do not have “Receiver for Web” sites enabled.

I want it so if I reboot the robot he picks up and starts running.

The Choice.

In order to successfully hit these specified targets I opt’ed to use ControlUp’s Logon Simulator.  It can target the Store Service so it works with our “Receiver for Web”-less stores.  It also has the features to generate events that can be targeted to send out notifications of an application launch failure.

The Setup

In order to achieve my goals I need the following:

  • A Service Account that will be logging onto the Citrix servers
  • The robot (Windows 2019 Server Core)

I installed Server Core 2019 and added it to the domain.

Configure Autologon

I configured group policy preferences to setup AutoLogon for my service account.  This group policy object is set to the OU the robots reside in.

Group Policy Preferences settings to configure Autologon for our service account.

However, I did not include the required “DefaultPassword” registry with the password.  In order to embed the password in a more secure fashion, I had to manually use Sysinternals AutoLogons.  This keeps the password from being stored in plain text in the registry but this does need to be manually executed on each robot.

Configuring Autologon

The account MUST be a regular user and not a member of the “Administrators” group.  This is a requirement of the ControlUp Logon Simulator.

Prerequisites gotcha’s

Because I selected to use pure Server Core, there are some components that require fixing for full compatibility.  This can actually be alleviated immediately by installing the Feature On Demand (FoD) “Server App Compatibility”, but this would increase both memory utilization and consume more disk space for our robot.  However, if you prefer the easy way out, adding the FoD fixes everything and you can skip the “Fixes” section.  Or just run the Logon Simulator on a operating system with the desktop experience.  Otherwise, follow the steps in each of the solutions.

Fixes

Unable to install Citrix Reciever/Workspace App

“TrolleyExpress.exe – System Error”
The code execution cannot proceed because oledlg.dll was not found.  Reinstalling the program may fix this problem.

Solution

Copy oledlg.dll from the SysWow64 in either the install.wim or another “Windows Server 2019 (Desktop Experience)” install and put it in the C:\Windows\SysWow64 folder of your robot.

 

ControlUp Logon Simlulator is cropped on smaller displays

Solution

Set the resolution larger in your VM.

 

ControlUp Logon Simulator errors when you attempt to save the configuration

“The logon simulator failed for the following reason: Creating an instance of the COM component with CLSID {C0B4E2F3-BA21-4773-8DBA-335EC946EB8B} from the IClassFactory failed due to the following error: 80040111 ClassFactory cannot supply requested class (Exception from HRESULT: 0x80040111 (CLASS_E_CLASSNOTAVAILABLE)).  An application event log has been written to handle this crash.”

Solution

Copy ExplorerFrame.dll from the SysWow64 in either the install.wim or another “Windows Server 2019 (Desktop Experience)” install and put it in the C:\Windows\SysWow64 folder of your robot.  Add the following registry:

ControlUp Logon Simulator detects admin rights

Admin rights detected The logon simulator should not be run as an administrator, please restart the app as a standard user.

Solution

Run the logon simulator as a standard user.


Configuration

Once you’ve implemented all the fixes, install Citrix Workspace App and ControlUp Logon Simulator with an account that is an administrator.

Configure ControlUpLogonSim.  With the simulator open, enter your Storefront details, ensuring to use the “Store” account as seen in the Storefront console.

 

 

For the “Resource to Launch” ensure the name matches the display name in Storefront:

 

In order to avoid session stealing in the simulator, each application will require a unique user name.  Setup a unique account per application you are going to test.

 

From here, enter your logon credentials for the account associated with the application.  Run your first test by clicking the green triangle and ensure it works correctly.

 

Now that we have a successful run we set “Repeat Test” to ON and save the configuration.

I then created another application to monitor by renaming the “Resource to Launch” as another application and saved a second configuration.  I saved all my files to a C:\Swinst folder.

 

The point of all of this is to ensure the simulator is running in an automated fashion.  To do so, we need to be able to configure the simulator to “launch” multiple different applications when the operating system starts.  We have already configured autologon, we’ve setup our configuration files for each application we want to monitor, now we need to set the monitor to auto-start.

Add the following registry key:

And create a file “C:\Swinst\StartAppMonitors.cmd” with the following contents:

And watch the magic fly!

And so the final question and the point of all this work, how much does this consume for our resources?

 

1.1GB of RAM for the ENTIRE system, a peak CPU consumption of 7%, and the processes required to do the monitor use no CPU and only ~55MB of RAM.  Each Citrix process consumes ~20MB of memory and is the most significant consumer of CPU but at the single digit % range.

I anticipate doing some more stress testing to determine what the maximum amount of monitors I can get on one system, but I’m thrilled with these results.  With this one box I would expect to be able to monitor dozens of application…  Maybe a hundred?

In the end, this was a fair bit of work to get this setup on Server Core, but I do believe the savings in resource consumption and overhead reduction will pay off.

Read More

Citrix Provisioning Service – Network Service Starting/Stopping services remotely

2018-05-02
/ /
in Blog
/

Citrix Provisioning Services has a feature within the “Provisioning Services Console” that allows you to stop/restart/start the streaming service on another server:

 

This feature worked with Server 2008R2 but with 2012R2 and greater it stopped working.  Citrix partially identified the issue here:

 

I was exploring starting and stopping the streaming service on other PVS servers from the Console and I found this information was incorrect.  Adding the NetworkService does NOT enable the streaming service to be stop/started/restarted from other machines.  The reason is the NETWORKSERVICE is a LOCAL account on the machine itself.  When it attempts to reach out and communicate with another system it is translated into a proper SID, which matches the machine account.  Since that SID communicating across the wire does not have access to the service you get a failure.

In order to fix this properly we can add either the machine account permissions for each PVS Server on each service OR we can add all machine accounts into a security group and add that as permissions to manipulate the service on each PVS Server.

I created a PowerShell script to enable easily add a group, user or machine account to the Streaming Service.  It will also list all the permissions:

An example adding a Group to the permissions to the service:

And now we can start the service remotely:

 

In order to get this working entirely I recommend the following steps:

  1. Create a Group (eg, “CTX.Servers.ProvisioningServiceServer”)
  2. Add all the PVS Machine Accounts into that group
  3. Reboot your PVS server to gain that group membership token
  4. Run the powershell script on each machine to add the group permission to the streaming service:
  5. Done!

And now the script:

 

Read More

Citrix Storefront – Adventures in customization – Add a help button to your Storefront UI

2017-12-27
/ /
in Blog
/

This customization is pretty easy.  Add the following to your custom.js file:

Replace “http://www.google.ca” with the URL you want your help screen to be.

Read More

Citrix Storefront – Adventures in customization – Default to “Store” view if you have no favourited app’s

2017-12-22
/ /
in Blog
/

We are in the process of migrating users from Web Interface to Storefront.  We have identified a potential issue; new users are directed to the “Favourites” view which doesn’t have any applications be default, instead it has instructions on how to add apps to the favourites view.

New users might say, “Where did my apps go?!”

The concern is users may become confused because Web Interface shows all your applications, and this new view shows none.  What we want to do to solve this is default to the “Store” view if you have no favourite apps, and default to the favourites view if you have at least 1 app favourite.

 

We can do this.

 

Just add the code above to your custom.js file and the default view will be changed to the store if you have no favorited apps.  Done!

Read More

AppV 5 – Raiser’s Edge 7.96 – Run-time error -2147024770 (800707e)

2017-12-15
/ /
in Blog
/

We are in the process of upgrading Blackbaud’s Raiser’s Edge to 7.96 and we encountered an error:

Run-time error ‘-2147024770 (8007007e)’:
Automation error
The specified module could not be found.

This error is giving us a few clues as to what might be happening.  The most obvious message is the “8007007e” which is a standard windows error hex code which translates to:

8007007E = FileNotFoundException

So RE7.exe is not finding a file it’s looking for.  With most AppV packages we can suss out the file it’s missing by using procmon and tracing for “FILE NOT FOUND” in the result field.  Unfortunately, my searching for this message did NOT result in finding a file that wasn’t resolved by another path.  In other words, all files were accounted for.  But the error message very clearly states that a file is missing.  So the next step is to install the application locally and compare the launch differences between the local install and the AppV install.  Again, process monitor makes this easy by using the “loaded modules” option.

The differences I found between a local install of this application and the AppV launch looked like so:

The launches were identical, until the highlighted points.  The local install, which works without issue, has an extra file that gets loaded.  bbcor7.dll.

It appears, somehow, this file is getting loaded and registered dynamically on a local install, but this is not happening with the AppV install.  I don’t see the file get searched for at all with the AppV install and tracing with procmon.  However, executing a regsvr32 /s “C:\Program Files (x86)\Blackbaud\The Raisers Edge 7\DLL\bbcor7.dll” during sequencing does do all the necessary work to register and allow RE7.exe to find and load the file in an AppV bubble.

So, long story short, execute:

While sequencing your AppV package and this should fix this issue.

Here is my entire sequencing script:

 

Read More

Citrix Storefront – Adventures in customization – Define a custom resolution for a specific application

2017-11-14
/ /
in Blog
/

Currently, Storefront does not grant the ability to define applications with specific resolutions.  In order to configure the resolution, Citrix recommends you modify the default.ica file.  This is terrible!  If you had specific applications that required specific resolutions, what are you to do?  Direct users to a variety of stores depending on the resolution required?!

Fortunately, again, we can extend StoreFront to make it so we can configure custom resolutions for different applications on the same store.  The solution is a Storefront extension I’ve already written.

The steps to set this up:

  1. Download the Storefront_CustomizationLaunch.dll.
  2. Copy the file to C:\inetpub\wwwroot\Citrix\Store\bin
  3. Edit the web.config in the Store directory and enable the extension
  4. We need to enable Header pass-through for DesiredHRES, DesiredVRES, and TWIMode in the “C:\inetpub\wwwroot\Citrix\StoreWeb\web.config” file:
  5. Lastly, add the following to the custom.js file in your StoreWeb/custom folder:
  6. And enjoy the results!  🙂

Read More

Citrix Storefront – Adventures in customization – Prepopulate Explicit Logon Credentials

2017-10-31
/ /
in Blog
/

Citrix Storefront allows you to prepopulate the credentials for your Explicit Logon.  The explicit logon screen is generally seen here:

And you can prepopulate the Username/Password fields.  If you don’t want to prepopulate the password, that’s fine too.  There are 3 properties and none are required.  Username, Password and Domain.  In order to prepopulate you must pass your credentials through to Storefront somehow, either as a cookie, header or as a URL search query.  I will demo it in the URL search query since I already have that code for pulling the parameters.  You must have “Explicit Authentication” enabled, aka, “User name and Password”:

Put the following code into your custom.js file:

The url to query is:

And the result:

Read More

Citrix Storefront – Adventures in customization – Login via credentials in URL search query

2017-10-30
/ /
in Blog
/

If you use a 3rd party service to connect to your Citrix Storefront environment, you may want to “pass-through” credentials without using domain authentication or whatever.  This post illustrates how you can login to your Storefront environment using nothing more than a URL with your credentials embedded in them.  To enable this functionality, this code must be in your custom.js file.

You MUST have HTTP Basic enabled as an authentication method on your Citrix Storefront Store.

The URL to login would look like this:

Put it all together:

Read More

Check for NUMA within a VM guest with Powershell

2017-07-06
/ /
in Blog
/

I have only tested this in VMWare, but it seems to work.  I have shamelessly stole the main code from here:

 

I was going to use this to detect NUMA and configure a network adapter RSS based upon NUMA configuration…   But I’m getting lazy and am going to ignore it.  But I don’t want this code to go away.  So here it is.  A way to detect if you are on a NUMA system in a guest VM in Powershell.

If you have greater than “1” for the NumaNode count then NUMA is present.

CoreInfo.exe result on the same system:

 

Read More