User Profile Manager – Unavoidable Delays

/ /
in Blog

I’ve been exploring optimizing logon times and noticed “User Profile Service” always showed up for 1-3 seconds.  I asked why and began my investigation.

The first thing I needed to do is separate the “User Profile Service” into it’s own process.  It’s originally configured to share the same process as other services which makes procmon’ing difficult.

Making this change is easy:

Now that the User Profile Service is running in it’s own process we can use Process Monitor to target that PID.

I logged onto the RDS server with a different account and started my procmon trace.  I then logged into server:

One of the beautiful things about a video like this is we can start to go through frame-by-frame if needed to observe the exact events that are occurring.  Process Monitor also gives us a good overview of what’s happening with the “Process Activity” view:

9,445 file events, 299,668 registry events.  Registry, by far, has the most events occurring on it.  And so we investigate:

  1. On new logins the registry hive is copied from Default User to your profile directory, the hive is mounted and than security permissions are set.

    Setting the initial permissions of the user hive began at 2:14:46.3208182 and finished at 2:14:46.4414112.  Spanning a total time of 121 milliseconds.  Pretty quick but to minimize logon duration it’s worth examining each key in the Default User hive and ensuring you do not have any unnecessary keys.  Each of these keys will have their permissions evaluated and modified.
  2. The Profile Notification system now kicks off.

    The user profile server now goes through each “ProfileNotification” and, if it’s applicable, executes whatever action the module is responsible for.  In my screenshot we can see the User Profile Service alerts the “WSE”.  Each key actually contains the friendly name, giving you a hint about its role:

    It also appears we can measure the duration of each module by the “RegOpenKey” and “RegCloseKey” events tied to that module.

    In my procmon log, the WSE took 512ms, the next module “WinBio” took 1ms, etc.  The big time munchers for my system were:
    WSE: 512ms
    SyncCenter: 260ms
    SHACCT: 14ms
    SettingProfileHandler: 4ms
    GPSvc: 59ms
    GamesUX: 60ms
    DefaultAssociationsProfileHandler: 4450ms (!)
  3. In the previous screenshot we can see the ProfileNotification has two events kicked off that it runs through it’s list of modules: Create and Load.  Load takes 153ms in total, so Create is what is triggering our event.
  4. DefaultAssociationsProfileHandler consumes the majority of the User Profile Service time.  What the heck is it doing?  It appears the Default Association Profile Handler is responsible for creating the associations between several different components and your ability to customize them.  It associates (that I can see):
    ApplicationToasts (eg, popup notifications)
    File Extensions
    The GPO “Set Default Associations via XML file” is processed and the above is re-run with the XML file values.
  5. Do we need these associations?

    Honestly…   Maybe.

    However, does this need to be *blocking* the login process?  Probably not.  This could be an option to be run asynchronously with you, as the admin, gambling that any required associations will be set before the user gets the desktop/app…  Or if you have applications that are entirely single purpose that simply read and write to a database somewhere than this is superfluous.

  6. Can we disable it?


    But I’m on the fence if this is a good idea.  To disable it, I’ve found deleting the “DefaultAssociationsProfileHandler” does work, associations are skipped and we logon 1-4 seconds faster.  However, launching a file directly or shortcut with a url handler will prompt you to choose your default program (as should be expected).

I’m exploring this idea.  Deleting the key entirely and using SetUserFTA to set associations.

We have ~400 App-V applications that write/overwrite approximately 800 different registered applications and file extensions into our registry hive (we publish globally — this puts them there).  This is the reason why I started this investigation is some of our servers with lots of AppV applications were reporting longer UserProfileService times and tying it all together, this one module in the User Profile Service appears to be the culprit.  And with Spectre increasing the duration of registry operations 400% this became noticeable real quick in our testing.

Lastly, time is still being consumed on RDS and server platforms by the infuriating garbage services like GamesUX (“Games Explorer”).  It just tweaks a nerve a little bit when I see time being consumed on wasteful processes.

Read More

AppV 5 – Raiser’s Edge 7.96 – Run-time error -2147024770 (800707e)

/ /
in Blog

We are in the process of upgrading Blackbaud’s Raiser’s Edge to 7.96 and we encountered an error:

Run-time error ‘-2147024770 (8007007e)’:
Automation error
The specified module could not be found.

This error is giving us a few clues as to what might be happening.  The most obvious message is the “8007007e” which is a standard windows error hex code which translates to:

8007007E = FileNotFoundException

So RE7.exe is not finding a file it’s looking for.  With most AppV packages we can suss out the file it’s missing by using procmon and tracing for “FILE NOT FOUND” in the result field.  Unfortunately, my searching for this message did NOT result in finding a file that wasn’t resolved by another path.  In other words, all files were accounted for.  But the error message very clearly states that a file is missing.  So the next step is to install the application locally and compare the launch differences between the local install and the AppV install.  Again, process monitor makes this easy by using the “loaded modules” option.

The differences I found between a local install of this application and the AppV launch looked like so:

The launches were identical, until the highlighted points.  The local install, which works without issue, has an extra file that gets loaded.  bbcor7.dll.

It appears, somehow, this file is getting loaded and registered dynamically on a local install, but this is not happening with the AppV install.  I don’t see the file get searched for at all with the AppV install and tracing with procmon.  However, executing a regsvr32 /s “C:\Program Files (x86)\Blackbaud\The Raisers Edge 7\DLL\bbcor7.dll” during sequencing does do all the necessary work to register and allow RE7.exe to find and load the file in an AppV bubble.

So, long story short, execute:

While sequencing your AppV package and this should fix this issue.

Here is my entire sequencing script:


Read More

AppV5 – System Environment Variables Gotcha’s

/ /
in Blog

We had an application that has its environment configured via ‘system environment variables’.  System environment variables differ from ‘user’ environment variables in that they are global so all users see them, and in a native Windows environment, they are stored in a different location; but still in the registry.  So it seemed, like everything else App-V, these values would be captured (they are just registry values).  Imagine our surprise when they were *not* captured.

The tech who was working on this thought this must be a mistake, exported the registry keys he needed into a file and imported them into the package:




He created a startup script that would modify the environment variables to whatever environment the user wanted to launch (Test, Prod, Dev, etc.).

BUT, no matter what was set the application always launched with the parameters it was sequenced with.  In this case it was always PROD.  We would put a ‘SET’ or ECHO ‘%TT_SYS1%’ or some such into the batch file, after the ‘SET/SETX’ commands we could see the variables were different.  They were what we expected them to be.  But launching a new process with these changed variables resulted with the new process inheriting the original environment variables.  So it was always PROD.  Now, these are SYSTEM environment variables and Microsoft released a utility, setx, which you can use to manipulate them.  This utility still did not work as expected, the variables were not being retained by a child process.

“By default, a child process inherits the environment variables of its parent process.”

When this was brought to my attention I suggested we explore ‘Exporting the manifest and seeing what was stored there’. Changing the registry entries in the package did NOTHING.  So I suspected they weren’t being used at all…

What we found is when we extracted the manifest file:

We saw where the environment variables where being stored.  By deleting this section and reimporting the manifest file we found that we could set environment variables and they would correctly inherit from the parent process from then on.  But if you do NOT delete this section of the Manifest file, then whatever is set for that AppV session will be reset for any new processes created in the bubble.


Final Analysis:

AppV5 treats Environment Variables differently then one would expect from a native system.  My expectation when I enter the bubble is an environment that I could manipulate and then have those changes persist for the duration of that session.  Instead, AppV environment variables are reset for EVERY process in the bubble.  Although you can manipulate them in the current process, creating a new process (cmd.exe/powershell.exe/whatever.exe) results in those changes being reset for variables specified in the manifest file.

Read More

AppV5 – Microsoft Visual C++ Runtime Library – Error R6034

/ /
in Blog

We packaged an application that was giving out an error message at a certain point in the program:

————————— Microsoft Visual C++ Runtime Library ————————— Runtime Error! Program: D:\AppVData\PackageInstallationRoot\7336984D-A43A-4EE3-A223-09E50C2E4A04\8B9E7007-D87F-4620-9B10-F06320BB1B… R6034 An application has made an attempt to load the C runtime library incorrectly. Please contact the application’s support team for more information. ————————— OK —————————


Fortunately, this was a nice repeatable error.

What happens is the program generates this message when it’s importing a word doc into its database.  This error message doesn’t stop the program from importing or causes a failure, and the process actually completes successfully.  Even stranger is after this message appears and you click OK it will not pop up again and you can repeat the process ad infinitum without this prompt…  Until you quit and relaunch the program.  Then it will prompt the first time but then never after.

But having this prompt even once is unacceptable to our users.  So how can we solve it?  Let’s examine the clues we’ve been given.  “An application has made an attempt to load the C runtime library incorrectly”.  So it’s crashing when loading a Visual C++ Runtime Library.  Which file?  Unfortunately, the path got truncated.  But Procmon will tell us!  I opened procmon, set it only look for ‘Load Image’ operations and then traced the process name.  This is where it crashed:


It crashed loading the MSVCR80.dll.  What’s curious is this DLL and the version and everything exist natively in the system.  I’m unsure why it was captured but the program is deferring to the AppV msvcr80.dll as opposed to the natively installed one.

Using Process Explorer I searched for this DLL and confirmed that it’s loaded in multiple places, with all programs except the AppV program using the natively installed one.

What I suspect is happening is that there is some form of DLL sharing occurring and Windows is telling the program that “your loading this file?!  I already have it loaded and used by all these other programs.  Use that instead you dummy!”

So can we force the program to use the natively installed DLL’s?

This program automatically installs some Visual C++ runtime libraries and these are captured by AppV automatically.  There is no way to remove them as they are not a separate installer or anything we can just hack out during an install.

However, we can just manually delete them.  If they don’t exist in the package AppV should automatically look for those files in the native file system.

I then went into the AppV Sequencer and ‘Edit’ on the package, selected ‘Package Files’ and deleted the ‘winsxs’ folder:


Did it work?

This is what procmon showed me for the new package without these folders:

The msvcm80.dll is now using the natively installed version and path!  And I did not get the error message any more and the operation worked successfully!  Another success story.

Read More

ControlUp – Realtime Troubleshooting Example

/ /
in Blog

I’ve been using a troubleshooting tool for some time and I can’t speak highly enough about.  It has saved myself so much time in finding/resolving and remediating issues.  I decided to capture myself utilizing the tool *in realtime* to remediate a reported issue that came in during a weekend call.  I was able to find the servers that had the issues within 5 mins and prevent them from hosting users (effectively remediating the issue).

The root of the issue is we had some servers reboot during the daylight savings time change and this messed up some of our scripts that were relying on some timing to load our AppV packages.  After receiving the call I started my recording.

Youtube link.  Please ensure to select the highest quality.

Without ControlUp, trying to search all the servers was a manual task or a scripted task that was executed serially.  These tasks, in the past, would take tons of time.  Generally, an issue like this would have taken an hour or so to resolve.  With ControlUp this was resolved in less than 5 minutes.

Higher Quality Version here.

Read More

AppV5 – 0xFD01F25-0x2 and deciphering some of these messages

/ /
in Blog

We recently had an issue with some users launching an AppV 5 application.  They were getting an error message:

In order to troubleshoot this issue more effectively, I turned on ‘Event Tracing for Windows‘ for the AppV logs and captured the output.  Searching for the error code revealed the following:


The first line that shows an ‘error’:

I navigated to that path, and sure enough, a ‘Templates’ folder was not present.

I did a procmon trace during that user logon and noticed the folder was never created.  AppV, for some reason, when it did not find this folder threw up an error.  If I create that folder I was able to launch the application without issue.

So what does the error message “0xFD01F25-0x2” mean?  Well, the first portion split by the ‘dash’ is the component that is explained to decipher where in AppV this issue is occuring.  The second string (0x2) is more interesting because it actually tells us something.  Microsoft has these short codes documented here.

0x2 = the system cannot find the file specified.  It’s actually looking for a folder, but the object didn’t exist and that’s the code it generated.  So if you see that second octect in an AppV error, the short system error code may give you a more precise clue to what is occuring and how you can fix it.

Read More

AppV 5.1 Sequencer – Not capturing all registry keys – Update

/ /
in Blog

My previous post touched on an issue we are having with some applications.  The AppV sequencer is not capturing all registry keys.  I have been working with Microsoft on this issue for over 2 years but just recently got some headway with getting this addressed.  And I have good news and bad news.  The good news is the root cause for this issue appears to have been found.

It appears that ETW (Event Tracing for Windows) will capture some of the events out of order and the AppV sequencer will then apply that out of order sequence.  The correct sequence of events should be:

But in certain scenarios it’s capturing the events as:


By capturing the deletion last in the order, the AppV sequencer is effectively being told to ‘Not’ capture the registry key.

Microsoft has corrected this issue in the ‘Anniversary’ edition of Windows 10 (Build 14393+) and sequencing in this OS will capture all the registry keys correctly.

The bad news is Microsoft is evaluating backporting the fix to older versions of Windows.  Specifically Windows 2008 R2.  Windows 2008R2 is still widely used and AppV best practice is to sequence on the OS you plan on deploying but if the older OS sequences unreliably this complicates the ability to ‘trust’ the product.  This fix still needs to be backported to 2012, 2012R2 and the related client OS’s, so hopefully they get it as well.  The reason I was told 2008 R2 may not get the fix is that it is no longer in Mainstream support, but Windows 7SP1 currently is, which is analogous to 2008 R2.  So hopefully the effort to fix this bug will be backported and we’ll have a solid AppV platform where we can sequence our packages in confidence.

Read More

AppV5 – Citrix User Profile Manager exclusions

/ /
in Blog

The Citrix User Profile Manager (UPM) needs a little configuration tweaking to work with AppV specifically it requires:

 You must exclude the following item using Profile management exclusions:
 Profile Management\File system\Exclusion list\directories:
  • AppData\Local\Microsoft\AppV
 If you don’t exclude these items, App-V applications work the first time users access them but they fail, with an error, on subsequent logons.

But what happens when you *don’t* exclude this directory?

We upgraded our Citrix UPM to 5.4.1 and in that process we moved from setting our inclusions/exclusions via the ini file to using Group Policy.  The original thought was simply adding the exclusions would add them to the existing list of default inclusions/exclusions which already has this directory set.  This line of thinking was incorrect.  Citrix’s documentation states:

Important: If you use Group Policy rather than the .ini file (or you are rolling out a Group Policy deployment after a successful test with the .ini file), note that, unlike the installed .ini file, no items are included or excluded by default in the .adm or .admx file. This means you must add the default items manually to the file.

When we enabled Group Policy for the exclusions and set the path (for something unrelated to AppV) then it was the ONLY item being excluded from AppV and we were having the issue described by Citrix.  Our application would launch the first time, or oddly, just for the user on that specific server.  When they launched it again on another server it would fail until their user profile was deleted from the profile share.

I setup AppV5 debug logging and traced a launch of what this failure looked like when our user tried to start an AppV application:

The lesson?

If you are using a Profile Manager ensure your exclusions for AppV are applied correctly!  If you miss you may run into this weird behaviour.

Read More

Office Deployment Toolkit – taking a look

/ /
in Blog

When I attempted to create a virtualized Office application using the Office Deployment Toolkit (ODT) I was happily surprised at how easy it was, and extremely disappointed that it contained a lot of ‘baggage’. That is, the virtualized Office application installed options that I did not have a choice on whether they should be there or not. These included things like ‘Proofing Tools’, ‘Office 2013 Upload Center’, ‘Telemetry Dashboard for Office 2013’ etc.


I wanted to get rid of nearly all of this ‘junk’.  Unfortunately, Microsoft only offers the following apps to be excluded:

So if I wanted to remove these extra apps, which should I select?


This I have not been able to figure out.  BUT what you can do is modify the mcxml’s after you’ve built a package once, removing all these extra programs, and run the ‘flattener.exe’ again.  This will remake your package without them.  So the steps in detail:

  1. Create your config.xml file and exclude everything:
  2. run the ‘Packager’ switch
  3. This leaves you with a ‘blank’ package.  In reality, it leaves you with a package full of the extra stuff Microsoft crams in.  The package itself weighs in around 630MB for an AppV office package without office!  The extra ‘configurations’ we can now look at removing.

This is what was left in the workingDir:

Now explore to “%TEMP%\Microsoft Office 15\root\mcxml” and you’ll see these same mcxml files these extra bits.  This is where they are pulled to the WorkingDir to be processed.  If we remove them here then they won’t be pulled and we can continue without them being installed.  There are a few files we need to keep as they implement the publishing scripts for the package.  These files are:

Now modify your config.xml file and remove the exclude for the app you want to keep (Lync in my case):

Rerun the packager command.  This will ensure your application’s mcxml file is generated.

Now we want to remove all the previously detected extra files from the %Temp%\Microsoft Office 15\root\mcxml directory.  This is what I was left with:


Now, we do NOT want to run the setup.exe as that will re-add all the extra bits.  We just want to run the flattener.exe.  The command line to do so is:

Now we have a package that is only 62MB in size


And if we open it in AppV we see only the components we are really interested in, just Skype For Business and we don’t have any other junk.  At this stage, the package ‘fails’ because some of the dependencies are not in the package.  I hope to be exploring what the exact dependencies are in a future post to explore what the actual *minimum* package size we can get away with for some of these Office applications.


Read More

AppV5 – Connection groups, an example

/ /
in Blog

Connections groups in AppV5 is a feature that allows different AppV packages to have visibility to each other by overlaying them overtop of each other.  Why would you want to do this?  In this real world example, we utilize connection groups for 2 AppV packages.  One of these packages is updated bi-yearly or yearly and one of the packages is updated quarterly.  If we created one monolith package then full revalidation testing needs to be completed each time it’s updated.  This process can take several days.  By breaking it out into component packages we can ensure that only a subset of the package is changed and testing can be reduced to just that area which can usually be accomplished in a few hours.

The packages:
Raiser’s Edge
Address Accelerator

To get started, we create our AppV packages:

Raiser’s Edge

Address Accelerator

With our packages complete, we go onto the AppV Management Server and add the packages to the server:

Then we go to the Connection Groups and ‘ADD CONNECTION GROUP’

We publish the connection group to our targetted servers and both AppV packages ‘merged’ when you open either package.  Now, we can update the ‘Address Accelerator’ portion on the quarterly basis that is required and simply add the new AppV version to the Address Accelerator package in the management console and the next publishing refresh it will be received on the server and the next user launch of the package will have the updated components.  Smooth, easy and now we don’t have to touch the actual program or its guts.

Read More