printing

Printers and their impact on logon duration

2019-03-05
/ /
in Blog
/

40 second logons.  100 second logons.  200 second logons.  

Our users had an average logon time of 8-11 seconds.  But some users were hitting 40-200 seconds.  Users were frustrated and calling the help desk.

Why?

I did not want to individually examine all of the users whom had a long logon duration.  There was dozens of dozens of users, but after sampling 5 of them, the root cause was consistent.  Printers were causing long logons.  

What I found was users were encountering the long logons on specific workstations.  Further investigation revealed these workstations had globally mapped printers.  Some workstations had dozens of printers.  Some of the printers were long gone, having been moved/re-IP’ed, shutdown, or re-provisioned and the entry in the workstation not updated.  Citrix, depending on Receiver version, attempts to do a check to validate the printer’s operation before mapping and this would wait for a network timeout (or a crash of Receiver).  When I stopped the print spooler on the local workstation, logons came back down to the proper range.

What I sought to do was create something to make identifying this root cause easier, quicker, and with more specificity — in other words was there a particular printer causing the delay?

Citrix Printing Methods

Citrix has two methods of printing.  “Direct connection” and local printing.  Direct connection has the potential for larger impact on logon duration as it executes additional steps and actions on the Citrix server.

To enable “Direct Connection”, Citrix has a policy “Direct connection to print servers”.  By default, this policy is enabled.

With this policy enabled, it changes the behavior of how Citrix connects printers you have mapped locally as network printers.  Citrix has a diagram here:

But I feel this is missing temporal information.  I’ve created a video to highlight the steps.

Direct Connection Process

Why is this important?

Citrix has an option that is an absolute requirement for some applications.  
“Wait for printers to be created (server desktop)” (Virtual Apps and Desktops 7.x) or “Start this application without waiting for printers to be created. (Unchecked)” (Citrix XenApp 6.5).

This policy needs to be enabled for numerous applications that require a specific printer is present before the application starts.  This is usually due to applications that have pre-defined printers like label printers and do a check on app launch.  If you have an application like that you probably  have this feature enabled.

Citrix has many options for managing printers and they can impact your logon process.

Focusing on Direct Connection being enabled, if “Wait for printers to be created (server desktop)” policy is also enabled, care MUST be taken to minimize logon times.

Direct connection does something that can have a very adverse affect on logon time that is enabled by default.  When it connects directly to the print server it will check and install the print driver on the Citrix server.  The installation activity is visible via process monitor.

The DrvInst.exe process is the installation of printer drivers

The installation of printer driver, by default, will only occur if the driver is inbox or prestaged on the Citrix server.  But the check to see if a print driver needs to be installed occurs every time a session is created.  This behavior can be changed by policy.

Why does this impact logon times?  Driver installation can take a while, especially if there is communication issues between the Citrix server and print server.  Or if the print server is under stress and just simply slow to respond.  Or if the driver package that needs to be loaded is especially large, has lots of forms or page formats, multiple trays, etc.  This all adds up.

How can I tell what the impact of printers might be on Logon Duration?

I’ve been working on updating a Script-Based Action (SBA) originally posted by Guy Leech on the Citrix blogs.  This enhancement to the SBA will output more information breaking down what’s consuming your logon times.  Specifically, this tackles how long printers take in the ControlUp column: “Logon Duration – Other”

I’ve created a video showing how to enable the additional logging and how to run the new Script Based Action and what the output looks like:

What’s new in the output?

For the individual printers, direct connection printers now show two lines, the Driver load time and Printer load/connection time. 

The driver load time is titled with Driver and then the UNC path to the printer. 

The Printer and UNC path shows the the actual connection, printer configuration and establishment time.  All Direct Connection printers will be shown with UNC paths.

Citrix Universal Print Driver (UPD) mapped printers using the Citrix Universal Print Driver (print jobs that get sent back to the client for processing) are shown with their “Friendly Name” and do not have a matching “Driver” component as no driver loading is required.  In the example output above, “\\printsrv\HP Color LaserJet 5550” is a direct connection printer and “Zebra R110Xi HF (300 dpi)” is a Citrix UPD printer.

Here is an example of the full output:

The “Connect to Printers” is a sub-phase under “Pre-Shell (Userinit)“.  Each “Printer:” and “Driver:” is a sub-phase under “Connect to Printers“.  In this example screenshot, “Pre-Shell (Userinit)” was consuming 117.6 seconds of a 133.2 second logon duration.  Prior to this script that was all that was known.  Now we can see that connecting to printers consumes 97.7 of those 117.6 seconds!

Awesome!  What’s the catch?

In order to generate this information, some more verbose logging is required on the Citrix server side.   I specifically worked to ensure that no 3rd party utilities would be required so we simply need to enable 4 additional features.

Command Line Audit
PrintServer/Operational Logs
Audit Process Creation/Termination

I’ve written a batch script to enable these features:

Easy!

Wait a minute!  Windows Server 2008R2 doesn’t have Command Line Auditing!

Good catch!  Windows Server 2008R2 doesn’t have command line auditing so it will miss the driver load information.  There is no way around this using native tools, but modification could be done to use a 3rd party tool like sysmon.  The output on a 2008R2 system would show the following (with all other features enabled).

Awesome!  Let me at this script!

This script is available on the ControlUp site here.  If you have ControlUp you can run simply add it via the “Script Based Actions” button and further derive more information on your users logon performance!  

Last Gotcha

In some instances the “Connect to Printers phase” may start before “Pre-Shell”. This has been observed and is actually happening! The operating system is attempting to start *some* printer events asynchronously in certain situations and so the connect to Printers may start before userinit, but overall connecting to printers may still be blocking logons while it trys and completes.

Steve Elgan kindly pointed out that it’s important you size your event logs to ensure the data is present for the users whom you want to monitor. If your event logs are sized too small than the data being queried by the script won’t be present. So ensure you have your Security and the Printer event logs sized large enough to capture all of this data! For an idea of scale, you can look at your log size and the oldest entry there. If your log is 1MB in size and your oldest entry is from an hour ago, than sizing your log to 24MB should capture about a day’s worth of information.

Hope this helps you reduce your long logon durations!

Read More

About the XenApp 6.5 Group Policy Client Side Extensions (CSE)

2016-08-02
/ /
in Blog
/

TLDR; using a newer Citrix Group Policy Management (GPM) than 1.7.X on XenApp 6.5 will cause your policies to disappear if you upgrade your Client Side Extensions to a version higher than 1.7.6.

The Citrix Client Side Extension (CSE) are the ‘Citrix Group Policy Engine’.  The CSE takes whatever policies you set through Active Directory, locally or the AppCenter console and apply them to your server or Citrix session (if a user policy).  There is some oddity with the CSE and ‘Citrix Group Policy Management’ portions of the Citrix products.  You see, they are interoperable, but in certain scenario’s they are not.

The split appears to be for the XenApp 6.5 product for CSE 1.7.6+.  My Citrix TRM informed me that the Active Directory Group Policy Schema changed for CSE 1.7.6+.  If you intend to use CSE 1.7.6+ you will need to upgrade your Group Policy Mangement (GPM) to 1.7.11.  To upgrade your AD policy seems simple enough.  Citrix says open the policy on a computer with GPM 1.7.11 and then close it and it will become updated.

But here’s a bit of the rub.  Citrix supports and encourages “some” mixing and matching of some components.  Specifically, the Citrix Universal Print Server (UPS) and Client (UPC).

And here’s my story:

We wanted to use the 7.6 version of UPS/UPC as it had some improvements we deemed critical.  We had not upgraded the version of the Citrix GPM/CSE from what came with the Citrix XenApp 6.5 (1.5.0).  When we downloaded UPS/UPC 7.6 we found we could not configure the Group Policy  settings for the Universal Print Client… Until we upgraded the GPM that came with XenApp 7.6.  Then the UPM policies appeared, ready to configure.  The version of GPM included with XA7.6 was 2.2.0.

Only on reboot, with the policies set, we found they were still not applying.  At this time, I found you need version 1.7.0 of the CSE to recognize the new policies.  We installed CSE 1.7.0 and it recognized all the policies and we were flying.

Fast forward a year or so later and we decided to ‘get up to date’ with our operational software.  Essentially, we wanted to ensure we had all the bug fixes enhancements of all of the latest and greatest for XA6.5 so we can survive for the next couple years while we transition to whatever Citrix will have out by then.  So the latest and greatest CSE is 1.7.6+ and I installed it, and all my policies went poof.  This prompted my earlier post.

During the course of troubleshooting my issue I installed various versions of the CSE’s and GPM’s that came with the various versions of Citrix XenApp.  Since we had GPM 2.2.0 installed, nothing from the 1.7.6 CSE branch recognized any of the policies.  BUT, installing any of the CSE’s from XA7.5+ recognized and applied the 6.5 policies and everything on top of that.  So I started asking our Citrix TRM if it was supported to have the CSE from the newer XenApp 7+ on 6.5 and if they included all the policies.  The answer was ‘Maybe it works, probably not supported’.  So I asked why the policies of 2.2.0+ don’t work with CSE 1.7.6 and the answer I got was the schema changed for the GPO’s.  This is implied in CTX202233:

Note: This fix addresses the issue for AD policies you create after installing this update. It also addresses it for existing policies where Citrix settings were configured before Microsoft settings. It does not address it for existing AD policies where Microsoft settings were configured before Citrix settings. For those AD policies, you must open the affected policies and save the Citrix settings.

Opening and saving the policies updates the schema.

 

Read More

Microsoft Patch (KB3170455) breaks PrintBRM importing printers with drivers

2016-07-29
/ /
in Blog
/

We have an application (ARIA MO) that has some special requirements.  The application requires all printers used by it in the different locations to be manually loaded on the Citrix server with all drivers.  This totals around 220 or some odd printers installed with around 15 different drivers loaded.  The printers must have a local port and cannot be mapped via TCPIP or network mapping.

Our design goals for our Citrix environment are to minimize the various PVS images we use so we use various ‘layers’ to allow a single master image to be able to host various unique and difficult configurations.  For this application we use AppV as our layering technology to put the application on the server, but for the printers we use a script to load them onto the server.  What we have is a print server that hosts all the printers needed by this application and we can export the printers into a file using Print Management.  Then we save that file on a network share somewhere.  When the Citrix server boots, I can take that file and manipulate its contents to change the queues to ‘local’ queues then import that modified file to the Citrix server.  I configured the script to take two parameters, a print file name and a server name.  I call the script with a command line like this:

The powershell command it calls is here:

So, what does this have to do with KB3170455?  Well, since installing KB3170455 it prevents importing printer files with print drivers embedded.  This is what it looks like with KB3170455 installed:

Screen Shot 2016-07-29 at 9.04.48 AM

Without:
Screen Shot 2016-07-29 at 9.07.35 AM

And the failure import with 3170455 installed:

Remove KB3170455 and the import works without issue.

Read More