Uncategorized

EventID 6003 – The winlogon notification subscriber was unavailable to handle a critical notification event.

2016-05-31
/ / /

We updated one of our Citrix XenApp servers and this message started flooding our Application event log:
“The winlogon notification subscriber <TrustedInstaller> was unavailable to handle a critical notification event.”

So what’s going on here?

Examining the registry on a ‘good’ working system and the ‘bad’ system revealed the following:

2

Good TrustedInstaller – No Error 6003

 

1

Bad TrustedInstaller – Error 6003

How did that value get there?

It turns out we installed Internet Explorer 11 with our patch cycle — but that in and of itself did not cause our issue.  Additional components for IE 11 were installed as well:
3

“Microsoft Windows English Spelling Package” and “Microsoft Windows English Hyphenation Package”

Neither of these packages are present on any of the ‘working’ systems.  I tested to determine which of them placed the registry key there…

It turns out both of them do.  If you uninstall *either* package it will remove the ‘CreateSession’ value.  Since these packages are not in our standard build we are removing both.

Read More

WSUS clients fail with WARNING: Exceeded max server round trips: 0x80244010

2016-03-14
/ / /

J C Hornbeck/Joe Tindale touched on this topic on a Microsoft blog post here.

In that post he touches on what’s actually happening:

Cause: The error, 0x80244010, means WU_E_PT_EXCEEDED_MAX_SERVER_TRIPS and happens when a client has exceeded the number of trips allowed to a WSUS server.  We have defined the maximum number of trips as 200 within code and it cannot reconfigured.  A “trip” to the server consist of the client going to the server and saying give me all updates within a certain scope.  The server will give the client a certain number of updates within this trip based on the size of the update metadata.  The server can send 200k worth of update metadata in a single trip so it’s possible that 10 small updates will fit in that single trip.  Other larger updates may require a single trip for each update if they exceed the 200k limit.  Due to the way Office updates are published you are more likely to see this error if you’re syncing Office updates since their metadata is typically larger in size.
I’ve bolded the more important information.  This is hardcoded and cannot be reconfigured.  This, to me, is a bit ridiculous that it can’t be reconfigured.
I have a WSUS client where we ‘reset’ the Windows Update client.  After resetting the client we were getting an error “WARNING: Exceeded max server round trips: 0x80244010”.  We would try multiple times but this error wouldn’t go away and prevented us from running Windows Update on this system.  So I started to investigate.  The first thing I did was finish reading that blog entry.  Hornbeck continues:
The client takes these new updates as they trickle down and inserts them into a small database to cache them for future use.  So during the first client synchronization with WSUS the client may get 75% of the available updates, put them into the database, and then fail at some point due to the number of max trips being exceeded.  The good news is the second synchronization cycle will not need to start from the beginning since the client has already cached 75% of the updates into its database.  The second cycle will pick up where it left off and most likely finish getting all the updates from the server.  There have been a few rare cases where a third scan cycle is needed but more often than not two is sufficient.
Again, I have bolded and underlined the important parts.
I started my investigation by trying to replicate the problem.  I started up Windows Update and ‘checked for updates’.
Ok, no problem.  So I checked again.
Well, Hornbeck/Tindale did say it may take a couple passes.  Let’s try again.
I’m getting worried now…  Let’s try a fourth time.
Hmmm…   At this point I wanted to better understand what Windows Update was doing.  I originally installed Wireshark to trace the conversation but it was difficult and time consuming to try and count the traffic back and forth to the WSUS server.  So I reverted my system and installed Fiddler2 on it.
From the video you can actually see the traffic from the WSUS server.  The request for the ‘Updates’ starts at item 3 and completes at 203.  Exactly 200 round trips.
Since my previous Windows Update attempts at the WSUS server failed after a few tries I thought I would trace the traffic with Fiddler for the multiple attempts.  My logic was I wanted to know if the traffic was ‘looping’; repeating itself and getting to the limit preventing updates.  Or, would each send/receive be unique and thus, simply, more is needed?
The first bit of the first run.  If the second run as identical or near identical traffic ‘packet sizes’ I would be concerned it’s looping…
I reset Fiddler and started the second run.  Completely different!
When I started the second run I was happy to see it was a completely different result.  I cleared Fiddler for a 3rd time ran the ‘Check for Updates’ until it timed out again, and cleared Fiddler again.  I then thought to just let Fiddler capture everything.  There really is no need to clear it each time.  I monitored the Fiddler output looking for loops or patterns.  The update check timed out a forth time (as before).  There was no looping I could see.
Finally, on the fifth run:
Updates!  We have updates!
In the end, when the Microsoft blog post was written (2008) there probably was only enough updates that two or three passes would go through all of them.  As time as gone on and more and more updates have been deployed to systems this hardcoded maximum is doing a huge disservice.  Our Windows 2008R2 SP1 systems require FIVE passes of clicking/waiting/clicking/waiting/etc.
A natural solution to this is to expose the “max server round trips” variable and allow it to be programmed by the organization according to their needs.  The present state of this issue is unnecessarily confusing and arbitrarily limited.
Read More

The winlogon notification subscriber TermSrv failed a critical notification event.

2016-02-03
/ / /

When launching a Citrix application a user was getting a the ‘Citrix launch bar’ then, after a few seconds, it would immediately go away.  Checking the Application log in the event viewer displayed these entries:

The winlogon notification subscriber TermSrv failed a critical notification event. Event ID 6001
The winlogon notification subscriber Sens failed a notification event. Event ID 6004

The root cause?

A faulty path for the homedrive AD attribute:

See that ‘Connect’ H: To ‘Path’?  That was a bad path.

Removing the attribute or fixing it so that it pointed to a path that the user has access to and exists resolved our issue immediately.

Read More

Citrix Director 7.7 is giving me artifacts!

2016-01-19
/ / /

We are still in a pilot-preupgrade phase of Citrix Director 7.7 and found an issue where we were getting artifacts with IE11.  The artifacts manifested themselves like so:

This is wrong.
When the search box should look like this:
This is right.
This issue appears to be caused by a policy our organization uses to enable compatibility view for sites on the Local Intranet:
The Culprit
Citrix says the following about Director and ‘Compatibility View’:

Director does not support Internet Explorer compatibility mode. Please use the recommended browser settings. When you install Internet Explorer, accept the default to use the recommended security and compatibility settings. If you already installed the browser and chose not to use the recommended settings, go to Tools > Internet Options > Advanced > Reset and follow the instructions

Because we use a group policy to push this setting out, disabling it and re-enabling it on a site-per-site configuration isn’t acceptable.  There is an option to set the ‘zone assignment’ of our Citrix Director server to be on ‘Trusted Sites’ instead of ‘Local Intranet’ but this would be another policy that would have to be pushed out to the ~80,000 workstations we have.  Instead, there is another option.  We can edit the default.html file in the Citrix Director folder and add a line in thesection to tell IE to exclude this site from compatibility mode.  To execute this:
Edit the Default.html file (“C:inetpubwwwrootDirectordefault.html”) and add this line:

To the ‘head’ section.  Example:

Add the line around here
Delete your cache and now your website will work without issue.
Read More

Citrix Director 7.7 & XenApp 6.5

2016-01-15
/ / /

Citrix has ‘replaced’ Edgesight with a new tool called Citrix DesktopDirector. It’s a monitoring tool that provides help desk or other type of technicians with data on what’s going on in user sessions (when referring to XenApp). Although vastly inferior to ControlUp, it is free.

But it’s not without poor documentation and its own tales of woe. We have Citrix DesktopDirector 2.1 currently live in our environment. Our environment is purely XenApp 6.5 (at the moment).

For version 2.1, logins are incredibly slow. Enumerating any sort of information on DesktopDirector is horribly slow. It’s just incredi-bad.

Enter Citrix Director 7.6 7.7; lots of promises to overcome the failings of the previous version.

First, installing Citrix Director 7.7 for XenApp 6.5:

1) Launch the AutoRun (AutoSelect.exe) installation file

2)
Select ‘Start’ for XenApp

3)
Select ‘Citrix Director

4)
Agree to the licensing agreement and click ‘Next

5)
Select ‘Next

6)
Without entering any information, select ‘Next

7)
Select ‘Next

8)
Select ‘Next’ for the firewall rules

9)
Select ‘Install

10)
Wait for the install to complete

11)
Open IIS Manager on the Director Server. Select the Director site under Default Web Site and double-click on ‘Application Settings

12)
Under ‘Actions’ click ‘Add

13)
For ‘Name’ enter ‘Service.AutoDiscoveryAddressesXA’ and for value put the IP of the local ZDC server

14)
If you are not setting up certificates on the server for SSL, you can disable the SSL verification by changing the UI.EnableSslCheck to false.

15) Reset IIS from the command line.

16)
On EACH XenApp server, install the Director WMI Provider (DirectorWMIProvider_x64.msi for 64bit OS’s and DirectorWMIProvider_x86.msi for 32bit OS’s). No reboot is needed, it takes effect immediately.

17) Enable WinRM on each XenApp server. In a command prompt run:
‘winrm qc’

18) Configure WinRM with the appropriate permissions:
ConfigRemoteMgmt.exe /configwinrmuser HEALTHYDesktopDirector.TST /all

**A NOTE ABOUT CONFIGREMOTEMGMT.EXE**

Ensure you use the latest version you can find. For XenApp/XenDesktop 7.7 the version that comes with it is:

It has been revised as time has gone on and numerous bug fixes have been implemented. We implemented a script to update our WinRM so all of our XenApp servers would have the user configured. We didn’t include checking, we just assumed if a group was added it would not be added again. With an older version of ConfigRemoteMgmt.exe this was an incorrect assumption and our WinRM SDDL became so large that the command line that ConfigRemoteMgmt.exe generates for winrm.cmd breaks the command line. You see, ConfigRemoteMgmt.exe is also a front end to “C:\Windows\system32\winrm.cmd“. Here is the command it generates:

C:\Windows\system32\cmd.exe /c “”C:\Windows\system32\winrm.cmd” set winrm/config/service @{RootSDDL=”O:NSG:BAD:P(A;;GA;;;S-1-5-21-38857442-2693285798-3636612711-99999999)(A;;GA;;;BA)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)”}”

The part I bolded is the GUID of the object you pass in the /configwinrmuser. The ConfigRemoteMgmt.exe takes your object and converts it to a GUID and then sets the SDDL using the native Windows tools. The bug in a pervious version of this tool is that it would add another GUID, even if one already existed. So in the buggy version the next command would be this:

And so on. Eventually, this causes WinRM to fail and it will just hang at this point:

To check your WinRM Root SDDL, execute this command:

If you have an output similar to this, then you have a problem:

Fortunately, it’s a super easy fix. If you run the command with a single GUID it all gets reset:

And now you can execute the ConfigRemoteMgmt.exe command again. The new version of ConfigRemoteMgmt.exe doesn’t seem to just ‘stack’ the new version on top.

/**A NOTE ABOUT CONFIGREMOTEMGMT.EXE**

Now that you have WinRM configured, Director installed, you login and it works. So you try with a HelpDesk account and it fails:

In order to trouble shoot this, we need to turn logging on for Director. To do so, go into IIS Manager Console, to the ‘Director‘ Site, and double click on Application Settings. Turn on the following “Log.” settings:

Now, create the folder you specified the log will be captured in and set the following permissions on it:

Note: The permissions are %COMPUTERNAME%IIS_USERS

Restart IIS and attempt to login again. You should get a log file generated with content. A failed login had this output:

And a succesful login had this output:

I highlighted the lines in the successful output that showed the command succeeding and the output from that. So why did this fail? The one account “adtest91” is setup as a ‘help desk’ style account and has ‘custom permissions’; essentially ‘View-only’ but with some categories removed altogether. Could this be permissions based? The odd thing is DesktopDirector 2.1 works without issue with this exact same configuration.

Fortunately, Citrix tries to make this easy to troubleshoot. I will login to the ZDC and start a Citrix PowerShell session with “adtest91” and run the command it supposedly fails on:

Well, at least it appears consistent. So, apparently, we need to get this command working. My first try at troubleshooting was to login to the AppCenter console, go to Administrators and get properties on this account, and change it’s ‘Privileges’ from Custom to Full:

When I attempted to login it worked! So it’s definitely a permission issue. I went back to Powershell and executed the same command:

Success!

So if the Powershell command specified in the log file works then ‘Director’ should work, and so far that is what it looks like. Now I wanted to narrow down *which* permission does this as we do not want our Help Desk to have full control over the farm. The, very obvious, first thing that stood out was:

Edit Zone Settings.” We are having issues enumerating Zones, perhaps this setting enables you to *read* zones? I enabled this setting and retested:

Success! And our ‘Help Desk’ account can now login to Director.

We don’t really want Help Desk to have the ability to ‘Edit’ Zones but I don’t see a ‘Read-only’ permission so we may have to live with it.

So now we’ve logged into Director and we enumerate an account but we get this message:

Cannot retrieve the data. Cannot find the referenced object. View Director server event logs for further information.

The IIS logging shows:

And the event log shows:

The requested data could not be found in the data ‘The virtual desktop via WinRM service reported an exception. See the event log for more information.’ (‘http://WSCTXZDC301T.healthy.bewell.ca:5985/wsman’).

User: ‘HEALTHYadtest91’

Console operation: ‘Retrieving running application details for IMA Session…’

Additional information:

‘Exception of type ‘Citrix.Dmc.Common.NotFoundException’ was thrown.’

When I got this message with my ‘Administrator’ account I installed the Director WMI Provider and it started working. But with this ‘limited’ user account I was getting an this error. Amazingly, this error goes away as soon as a user logs in to Citrix Director webpage, who would also be a local administrator on the Director server. After searching and searching and troubleshooting and scanning and trying various things to get this working, the only thing that would work was to grant our Director group admin privileges on the Director web server. I finally stumbled across another article that reaffirmed that the only solution is to have the users be local admins on the web server.

https://support.software.dell.com/kb/154066

To test WinRM the following command can be used:

winrm get winrm/config -r:%server%      — Non-administrator user gets access is denied making WinRM queries

This error, though, is only if you set Connector.WinRM.Identity as ‘User’ and those users are not Administrators.  The fix, is to put your users into a group and make them a local admin on the Director server.

Read More

Help! My Citrix application is running slow (Meditech, Citrix, Imprivata)

2015-12-19
/ / /

We have an application (Meditech) that users have reported as performing poorly in Citrix.  They were able to confirm the poor performance by comparing it to a desktop that had the same software installed.  I was brought on to try and understand these performance differences and why Citrix would be performing so much worse.  To measure the performance, the user took me to a screen where they held down a key on the keyboard to increment a counter in the software.  When holding the key down on the desktop the counter was quick and incremented at a steady pace, on Citrix it seemed OK at first but then slowed as time went on.   To illustrate these performance differences, I made a video:

So we definitely have a problem.  My first steps on troubleshooting this problem was to compare the differences between the desktop and the VM.  The VM had server processors but they were older, and at less GHz at 2.6GHz for the server.  The desktop had 3.4GHz processors with a newer generation.  If the processing is NOT latent sensitive then the faster processor’s can make a difference, and the CPUMark had them at 1500 and 2100 (~40% difference).  At first glance, this seems like it could be the difference in our timings but it’s still too drastic at ~5s vs ~20s, a 400% difference.  To try and narrow the difference I took the application to a completely bare server with no software on it what-so-ever and reran the test.  It completed in about ~5-6s.  The processor it ran on was comparable to the original server processor but ran at 2.7GHz instead.  The processor was not running at nearly enough speed to make up the difference, something else must be consuming those cycles.

At this point I procmon’ed the benchmark and came up with the following:

(PID 6660 is the process that the benchmark runs against)

Very obviously, the pattern of each key stroke on the Citrix server is present with the initial pattern highlighted:

About 400ms from each key stroke to when the next one is registered.  So what is delaying the 400ms?  From my experience, unexplained delays are CPU related.  I then looked at the process activity summary to see if I can find the bottleneck:

Again, very obviously we are seeing the CPU ramp up, and it can also explain the faster, initial iterations of the GUI as the CPU ‘ramps’ up and doesn’t spike to the it’s maximum.  None of the other graphs show any activity at the time of the benchmark so the suspect is highly on the CPU.  When hovering over the graph we see the CPU percentage.

This is a 4 core box, so ~25% equals one full core for a single threaded application.  Again, this points to the application being bottlenecked by the processor, but again, the difference is too large to consider just the CPU at this point.  We need to find what part of the process is consuming these resources.  Fortunately, ProcExp (process explorer) can help us determine what is going on within a process.

I started a new run and got properties of the process:

MGUI.DLL is consuming all of the CPU.  That is a DLL utilized by the application, clicking on ‘Stack’ gives us the hierarchy of commands being utilized by that thread.

From this, I can understand that ntoskrnl.exe, ntdll.dll are native Microsoft Windows functions, MGUI.DLL is utilized by Meditech, but what is ISXHook.dll?

Doing a search within the process shows that it’s utilized by Imprivata, a single sign-on solution we utilize to try and increase user efficiency.  It works by ‘screen scraping’ to determine fields that it needs to populate with user credentials to try and speed up user logins.  Logically, this sounds like it could be causing the delay by screen scraping every time a key is stroked or a change is registered.  To confirm this, we need to remove Imprivata from the application.  Fortunately, it’s hooked in by services that can easily be terminated.

I’m going to terminate everything that says ‘Imprivata Inc.’

With the processes terminated I reran my test.  Using Process Explorer and getting properties on the process, immediately CPU usage went from 25% down to 15% at a peak:

And getting Stack information showed a much cleaner stack:

In conclusion, I may need to investigate into Imprivata to determine if I can reduce its polling rate or find some way of allowing it to ‘stop’ polling the CSMagic process *after* the accelerated login.  Its current settings (which I’m not familiar with, sadly) is not acceptable and causes a significant slow down.  Fortunately, the root cause has been determined and we can work towards a full resolution.

Read More

AppV5 – Sequencing an application that has Com+ Applications

2015-12-04
/ / /

AppV has a few weaknesses, one of them is Com+ Objects.  It doesn’t like Com+ Objects and sequencing an application with them generally results in this error message:

The Resolution seem easy enough:
1) Try extracting and installing the COM+ component natively on both sequencing station and the client
2) Thoroughly test your virtual application package to verify functionality.
But how do you extract the COM+ components?
1) Open ‘Component Services’
2) Expand until you find your Com+ Applications, right-click and select ‘Export’
3) Click ‘Next’
4) Enter a path to save your MSI then click ‘Next’
5) Click ‘Finish’
Since this application had two Com+ Applications I exported my second one as well:
To ensure they work, I took them to our XenApp Test server and installed them.
Both *appeared* to have installed correctly.  But this application requires the Com+ Application to run under a service account.  Checking the original install:
I can see under the ‘Identity’ tab that it is configured for ‘This user:’.  I then checked the MSI install on the XenApp Test server and looked at the Identity tab:

 

It appears it defaults to ‘System Account’ even though I selected ‘Export user identities with roles’.  To change the user account silently, I used this script supplied by the vendor (as a encrypted vbe):
This script changes the user account on the Com+ applications to use the proper account.  When browsing the properties of the components of the SFSQL object I saw it referenced the dll to it:

 

 

Fortunately, it seems these files are captured by the MSI export of the Com+ Application.
To test everything now, I must now create my AppV package.  Before doing the install I installed my prerequisites (the Com+ Applications) into the Sequencer so they are not captured by the package.
I ran this script to do my prereq install:
Now I startup the sequencer and do my sequencing first steps.
I selected a batch file I created for the install (per the vendor’s initial instructions):
Once the install completed I saved my package and took it to my XenApp Test server and published the application and ran it.
No, this is not correct
For some reason my application was failing.  I double checked the COM+ Applications are installed and appear to be working, so I opened Procmon.exe to see if I can find where it’s going wrong:
I’ve used procmon enough now to practically pull the old’ Matrix, “blonde, brunette, red head“.  Working from bottom to top, the message box is generated from the ‘Text’ (LanguagePack) back to the KernelBase.dll.  So we can exclude that has the cause, this is telling us *of* the issue.  So if we look at the lines preceding it, one of them must be causing our message.
Looking at the first keys, the OLE keys, I can compare them to my sequencer and they match exactly, so the likelihood of them causing our issues is none.
So we go back a little further to the AppID key and jumping to the *SUCCESS* location to look at it’s value…
I am on a server called WSCTXAPP301T but the registry key for this value is pointing to the system I did my sequencing on.  I changed the value to the actual server name and did a registry search for “WSAPVSEQ07” to see if it appeared anywhere else.  It did in one other location so I modified the value there as well to point to the local server:
And tested my application again:
Success!  The login screen.  This is as far as I can take this app until I get credentials or a user to login and try testing for me.  So, for now, I can create a preflight-script to setup some prerequisites when the application is published.  I modified my preflight script to include the two registry values being updated:

Then I updated the Deployment_Config.xml file for this application:

Lastly, I logged onto a new, fresh, system and manually tested my XML file by publishing from the command line:

Using Procmon to monitor to ensure the prerequisites get installed:

“Exit Status: 0” –> Usually a good thing 🙂

Testing the application:

Looks good

Then testing removal and monitoring with Procmon:

Com+ Application are removed cleanly 🙂

So everything looks good and I can now add this application to our AppV Management Server and deploy it to the machines that need it.

Read More

AppV5 – When Application Compatibility Toolkit doesn’t work

2015-11-27
/ / /

I have an application that embeds an IE window and generates a small HTML file for some content.  This application has some…..   interesting….  configuration settings that require a folder with “EVERYONE:F” permissions.  This folder cannot be in a path that contains spaces, cannot be a environment variable and the vendor recommends to put the folder on the root of the C: drive as C:TMP.  To make it more crazy, when launching the application it will generate a folder under the C:TMP that is locked to the %CLIENTNAME% (e.g., C:TMPComputerName) and every login to the application generates unique files to that session, so technically, each folder underneath TMP is unique to the user.  The vendor actually recommends completely deleting the contents of the TMP folder DAILY.

So why not just store these files in %TEMP% instead?  Great question.  If the vendor made that happen I wouldn’t have this lovely blog post.

(Just for a note this application utilized multiple folders on the root of C: as well, TMP being one, since we move our PackageInstallationRoot to a different drive, TMP isn’t available on C:)

With that all said, what am I looking to solve?

Well, I don’t want to be creating folders on the root of the C: drive as this application will be on our ‘generic’ PVS Citrix servers and minimizing the potential for pollution on the master image is a goal, having to create a folder and configure permissions is possible and I’ve done it before but it’s not very clean or elegant; really if we’re going to do that then we may as well pollute the C: drive.  We want to maintain portability so creating a C:TMP folder on the master image prevents us from publishing this package on desktops or other systems in the future unless this is well documented requirement.

We’ve just learned a new trick though, we can try using Microsoft ACT to create a file path shim that redirects the path to a different one.  So let’s do that…

The problem.

I created the shim using the steps from this post.  (C:TMP;C:ProgramDataMicrosoftAppVClientIntegrationD8E3DB68-4E48-4409-8E95-4354CC6E664BRootVFSProgramFilesX64dlc11.2TMP) I then launched the application and…

Huh.  That doesn’t look right.

And it didn’t work.  I then used procmon.exe to examine to see if it was reading the file correctly:

I see SUCCESS…

Procmon.exe is reporting that it IS reading that HTM file correctly.  So why isn’t it being displayed?

I ran across a similar issue on another application and what I found was that the embedded component appears to be running *outside* the bubble.  Since the shim targets applications (in this case prowin32.exe) all reads from prowin32.exe are being redirected by other processes are not.  I suspect (though I have no proof), in this case, that somehow the IE component is breaking outside the bubble and so it’s NOT getting redirected.

Can we force all programs to get redirected to the proper path?  Yes, we can.

Using symbolic links we can force any access to the directory to be redirected to the AppV package path.  AppV will then see this is a path within the ‘bubble’ and redirect *again* to your local profile where the AppV5 writes will take place.

I removed the shim and then executed:

Symbolic Link
mklink /d C:TMP C:ProgramDataMicrosoftAppVClientIntegrationD8E3DB68-4E48-4409-8E95-4354CC6E664BRootVFSProgramFilesX64dlc11.2TMP

Tracing with procmon.exe now showed us this:

And the application now displayed this:

So the application is now working without any issues, all of these ‘temp’ files are being redirected to a location where they will not be saved between sessions so no cleanup is ever needed.  We need to add the mklink.exe to the DeploymentConfig.xml.

AHS-BDMPHARMACY-PREFLIGHT.CMD:

And we are now done.

Read More

AppV5 – Using Application Compatibility Toolkit to solve issues

2015-11-26
/ / /

C:

We have several applications that install folders in the root of the C: drive.  For most AppV5 implementations, this wouldn’t be an issue but we modify our PackageInstallationRoot folder so the token {AppVPackageDrive} turns into the drive letter specified in the PackageInstallationRoot registry key.  For us, that’s the D: drive.  This causes an issue because when you sequence an application to C: the application has an expectation for it to be there.  Ideally, AppV takes care of that by the use of the tokens, but this breaks down when applications are hardcoded.

So far, we’ve been able to work around these issues by using junctions or setting the PVAD to the folder as the PVAD acts literally on the value specified, essentially becoming a hard coded, custom, token.
But now we have an application with THREE folders that are installed on the root of the C: and the application is hard-coded to look for two of them.
Hello my nemesis’s
If we do nothing this is the error we get after attempting to login to the program:
Looking at the appvve we can see the D: is coming up with those folders.
So this isn’t going to work.  How can we make it work?
After publishing this application on the server, we install ACT on the Citrix server.
Launch the “Compatibility Administrator (32-bit)”
Right-click on “New Database” and select “Create New > Application Fix”
Enter the details and browse to the application and click “Next”
Click ‘Next’ on the Compatibility Mode screen
Select ‘CorrectFilePaths’ and then click ‘Parameters’
In command line, enter the path that should exist, then a semi-colon divider “;” and then the target path:
C:\webforms;C:\ProgramData\Microsoft\AppV\Client\Integration\A3E3C00D-19F1-47CB-9BA1-464DB85DC3CA\Root\VFS\AppVPackageDrive\webforms
and click “OK”
Try a Test Run.
And the application launches without any error messages!
Click ‘Next’
Then Finish.
Click ‘Save’ then name your database and click “OK”:
Save your fix now:
At this stage you now need to put your SDB file somewhere accessible for when the package is published.  We put it on a fileshare.
Now, all we need to is install the fix.
Since we publish our application globally, I added the fix to the DeploymentConfig.xml:

And we are done! The application now works.

Read More

Citrix Receiver 4+ rants and “Your apps are not available at this time. Please try again in a few minutes or contact your help desk with this information”

2015-10-28
/ / /

The environment I’m working in is a mix of XenApp 6.5, 5.0 and Presentation Server 4.5.  The 6.5 and 5.0 farm also have a nearly identical test farm.  We’ve been migrating the applications off the Presentation Server farms and are moving them to XenApp 6.5.  At this point in the migration we have around 5-10 applications left on 4.5 to move, with around 400-450 on the 6.5 farms and probably around 20-30 or so on the 5.0 farms.  These farms utilize a Citrix Webinterface 5.4.2 frontend for web interface and PNA.

We have standardized the environment on mostly Citrix Receiver 3.3 and some 3.4.  Time has marched and we’ve been tasked with getting Receiver 4+ working the Windows 7/8/10 rollout.  We were not able to do so with the earlier versions of Receiver 4 because things like sort icons into custom folders on the desktop and Start Menu.  This feature came in around 4.2.  In our environment memberships to applications are granted through group membership and Citrix PNA allowed the user to ‘roam’ from computer to computer only displaying the applications they have access to, as opposed to a bunch of applications they do not have access with the onus on them to pick and choose the correct applications.

So we started work on planning this migration and have started with the latest greatest (as of today) Citrix Receiver 4.3.  We have been able to come close to simulating all the features of the Enterprise editions of 3.3 and 3.4.  Namely:

Citrix Receiver automatically connects and populates a folder (MyApps) in the Start Menu and on the desktop.
No self-service.  Applications are automatically presented to you and defined by Group Membership.
Single sign-on.  Receiver will take your Windows logon credentials to use for authentication.

We do have some outstanding items.  We’ve set the client to have all applications as ‘MANDATORY’ which does populate the applications in the MyApps folders; but applications marked as ‘Create shortcut on the desktop’ in the applications properties in AppCenter are not created.

Anyways, onto the problem.  Now that we have Receiver 4.3 setup, SSON working, PNA working, we logged onto our system and watched the applications populate.

Slowly.

Really Slowly.

Eventually a dialog popped up.

Then another, and another, and another.

And these dialogs are completely custom!  They are NOT native Windows dialogs!

So if you have multiple ones of them, sometimes clicking the X (close button) or OK doesn’t work because the dialog appears ‘modal’ and you need to click the button on the ‘active’ window.  But you generally don’t know which one that is so you have to go through and select each dialog from the task bar and try clicking ok until you magically get lucky and select the one that has priority.  Then you do it all over again as the next primary window *may not be the one on top*.

1 minute 24 seconds to populate 470 applications

Just for giggles, how fast does Receiver 3.3 populate the same list?

8 seconds.  And all the icons show up.

Alright, so you’ve passed that point and are now looking at your applications.  But they are missing icons!

But not all applications are missing their icons…  Only some.

So let’s find out what’s consuming all this time and maybe, just maybe, we’ll solve our “Your apps are not available” error message.

First thing we need to do is enable Citrix Receiver Logging:

Next is to exit and restart receiver and logs will start to generate.  They are located here:
%USERPROFILE%AppDataLocalCitrixReceiver
%USERPROFILE%AppDataLocalCitrixAuthManager
%USERPROFILE%AppDataLocalCitrixSelfService

The most important log tends to be the ‘SelfService.txt’ log.  If you search that log for the “Your apps are not available” error message it pops up in locations like this:

So this dialog popped up for an application called ‘BMTServe’.  And what does BMTServe look like?

Generic icon!

But BMTServe was not the only application that encountered this dialog.  From my video it popped up numerous times.  Searching the SelfService.txt file for ‘Your apps’ and looking for the application it references points to an application with a blank icon 100% of the time.  Not every application that produces a blank icon causes this prompt, as we literally have ~250 applications with blank icons and the dialog pops up anywhere from 0 to 10 times.  Sometimes it pops up 2 times, sometimes none, sometimes 10 times.

So, why are these icons blank?

Citrix Receiver 4.3 seems to only prefer 32bit icons or icons of a particular size.  I haven’t confirmed what exactly yet, but I do know that 8bit 32×32 icons don’t seems to get ‘translated’.  The Citrix logs all but confirm this as well.

I confirmed with Citrix that icons are required to be 32bit and the order they are checked is 48×48, 32×32 then 16×16.

This is how Receiver processes icons that are formatted correctly:

The icons were processed instantly. 00:00:00. But if they are formatted in a way that Receiver decides it needs to ‘reformat’ them:

This call to get an icon took 11.6 seconds!!! If it doesn’t get the icon formatted in the way it wants, it appears the SelfService.exe setups a queue of icons that it needs and ‘re-requests’ them from the server. Could it be that Receiver is submitting too many queries? The error mentions to check the authmansvr.txt log file. This log file shows the following:

The error appears to start at “CWindowsReceiver::CallARGetConnectedVpnGateway” When this call is successful it returns:

So, I guess it’s possible that trying to re-pull the icon data is causing authmansvr.exe to crash…?  Another crazy thing is I was attempting to automate this process of terminating Receiver and relaunching it to see if I could get a gauge on the frequency of this occurrence and this is what I saw:

Ok, I thought, not so bad.  Just two messages the first couple launches?  It shouldn’t be too much of an issue…  But then I looked at my application folder:

Left is when I get all my apps (and usually the message box) the left is all those ‘successes’

It appears Receiver removed all applications producing that dialog box.  When I was terminating and relaunching receiver it was ONLY populating 195 applications as opposed to 493 it was supposed to. No wonder I wasn’t getting any messages!  On a hunch, I looked at each of the 195 applications it kept and they all had good icons.  I then took a random sampling of about 30 of the 300 or so applications that it did not keep and none of them had proper icons, all blank.  So another bullet towards icons causing my issue.

Read More