Blog

Office Deployment Toolkit – taking a look

2016-10-21
/ /
in Blog
/

When I attempted to create a virtualized Office application using the Office Deployment Toolkit (ODT) I was happily surprised at how easy it was, and extremely disappointed that it contained a lot of ‘baggage’. That is, the virtualized Office application installed options that I did not have a choice on whether they should be there or not. These included things like ‘Proofing Tools’, ‘Office 2013 Upload Center’, ‘Telemetry Dashboard for Office 2013’ etc.

1

I wanted to get rid of nearly all of this ‘junk’.  Unfortunately, Microsoft only offers the following apps to be excluded:

So if I wanted to remove these extra apps, which should I select?

 

This I have not been able to figure out.  BUT what you can do is modify the mcxml’s after you’ve built a package once, removing all these extra programs, and run the ‘flattener.exe’ again.  This will remake your package without them.  So the steps in detail:

  1. Create your config.xml file and exclude everything:
  2. run the ‘Packager’ switch
  3. This leaves you with a ‘blank’ package.  In reality, it leaves you with a package full of the extra stuff Microsoft crams in.  The package itself weighs in around 630MB for an AppV office package without office!  The extra ‘configurations’ we can now look at removing.

This is what was left in the workingDir:

Now explore to “%TEMP%\Microsoft Office 15\root\mcxml” and you’ll see these same mcxml files these extra bits.  This is where they are pulled to the WorkingDir to be processed.  If we remove them here then they won’t be pulled and we can continue without them being installed.  There are a few files we need to keep as they implement the publishing scripts for the package.  These files are:

Now modify your config.xml file and remove the exclude for the app you want to keep (Lync in my case):

Rerun the packager command.  This will ensure your application’s mcxml file is generated.

Now we want to remove all the previously detected extra files from the %Temp%\Microsoft Office 15\root\mcxml directory.  This is what I was left with:

6

Now, we do NOT want to run the setup.exe as that will re-add all the extra bits.  We just want to run the flattener.exe.  The command line to do so is:

Now we have a package that is only 62MB in size

8

And if we open it in AppV we see only the components we are really interested in, just Skype For Business and we don’t have any other junk.  At this stage, the package ‘fails’ because some of the dependencies are not in the package.  I hope to be exploring what the exact dependencies are in a future post to explore what the actual *minimum* package size we can get away with for some of these Office applications.

 

Read More

AppV5 – Connection groups, an example

2016-10-21
/ /
in Blog
/

Connections groups in AppV5 is a feature that allows different AppV packages to have visibility to each other by overlaying them overtop of each other.  Why would you want to do this?  In this real world example, we utilize connection groups for 2 AppV packages.  One of these packages is updated bi-yearly or yearly and one of the packages is updated quarterly.  If we created one monolith package then full revalidation testing needs to be completed each time it’s updated.  This process can take several days.  By breaking it out into component packages we can ensure that only a subset of the package is changed and testing can be reduced to just that area which can usually be accomplished in a few hours.

The packages:
Raiser’s Edge
Address Accelerator

To get started, we create our AppV packages:

Raiser’s Edge

Address Accelerator

With our packages complete, we go onto the AppV Management Server and add the packages to the server:

Then we go to the Connection Groups and ‘ADD CONNECTION GROUP’

We publish the connection group to our targetted servers and both AppV packages ‘merged’ when you open either package.  Now, we can update the ‘Address Accelerator’ portion on the quarterly basis that is required and simply add the new AppV version to the Address Accelerator package in the management console and the next publishing refresh it will be received on the server and the next user launch of the package will have the updated components.  Smooth, easy and now we don’t have to touch the actual program or its guts.

Read More

Windows Update – Errors 80070057, 800736B3, 8024400E

2016-10-19
/ /
in Blog
/

We started the new patching Microsoft has put forward (cumulative updates) and one of our Citrix vDisks had an issue with it. Windows Update would say that there were no updates available:

windowsupdatebug

But we very obviously have updates to deploy to it.

When I checked the ‘WindowsUpdate.log’ I saw the following:

0x80070057 = E_INVALIDARG

The ‘CBS’ (Component Based Servicing) is reporting an Invalid Argument.  Microsoft keeps a more verbose log of the component based servicing here: C:\Windows\Logs\CBS

This log reported the following:

An error appeared to occur at this point (there were a few):

Failed to get component version: 6.1.7601.22653
Failed to enumerate store versions on component: x86_microsoft-windows-c..ityclient.resources_31bf3856ad364e35_6.1.7601.22843_da-dk_c1126f6226996014
Failed to enumerate related component versions on component: x86_microsoft-windows-c..ityclient.resources_31bf3856ad364e35_6.1.7601.22843_da-dk_c1126f6226996014

 

The standard MS method of troubleshooting Windows Update is to run the ‘CheckSUR’ utility.  This was the result of running that tool:

windowsupdatebug2

No errors detected.  Awesome.  So I have a problem but this tool reports everything is peachy.

So when I looked at Windows Update I saw numerous ‘failed’ updates.

windowsupdatebug3

I took one that was failed and downloaded it (KB3177467) and attempted to manually install it.

This time I got a different error:

0x800736B3 – ERROR_SXS_ASSEMBLY_NOT_FOUND

Looking at the CBS.LOG showed me the following:

Ok, so now we can see the error but it still doesn’t give us much information.  If I run ProcMon I can find *when* the error occurs somewhat easily because Windows Error Reporting kicks in as soon as the error occurs:

windowsupdatebug4

My apologies for such an ugly screenshot.  The ‘Dark Blue’ highlight is when Windows Error Reporting kicked in.  So the error must have occurred immediately preceding it.  The only line that had a ‘NAME NOT FOUND’ without being a subkey search is the line that ends in v!6.1.7601.18766.  If I browse to that location in the registry, here’s what I find:

windowsupdatebug5

 

So we are definitely missing that key.  I have a bit of an advantage with the patches here as I have a duplicate of this server in another vDisk that was made around a year ago.  Both are supposed to be at the same patch level, but it allows me to look through it’s COMPONENTS registry hive and see if it has that key…

And I see a vastly different set of keys:

windowsupdatebug6

So I import that key and try running the update again…

windowsupdatebug7

 

Manually, it successfully updated.  So now I tried running Windows Update again:

windowsupdatebug8

Code 8024400E.  Which means…  I just need to rerun ‘Try again’ about 6 times.  Once I do:

windowsupdatebug9

Ok, we have updates.  As a test I’m going to do just one:

windowsupdatebug10

 

Success!

windowsupdatebug11

So I’ll try the rest:

windowsupdatebug12

 

Success!

windowsupdatebug13

 

It appears we can install patches again.  I’m concerned the COMPONENTS hives were so different but without a solid understanding of how that hive comes to be (I wish you could rebuild it) I think I may be stuck with assuming that a failed update or maybe a corrupt registry key is at fault for that missing key breaking updates.  I guess we’ll see what happens next month to see if we can patch Windows :/

Read More

AppV5 – Applications being published to “C:\App-V Packages” instead of %PACKAGEINSTALLATIONROOT%

2016-09-21
/ /
in Blog
/

The last little while we’ve had an issue with some applications seemingly being published to “C:\App-V Packages” instead of our defined path “D:\AppVData\PackageInstallationRoot”.

AppV-Packages

Notice the timestamps? Packages published during the same ‘publishing refresh’

 

Video of the issue in action that I was able to setup a repro:

So what’s going on here that would cause this to happen?

I turned to the event viewer to find the time when this package “8FE4C99E…” was published and just looked to see if there was anything surrounding it.

Highlighted is the package publish event

This is the event showing the package being published to “C:\App-V Packages”.  There is an odd thing different about this publish event compared to the rest (that you can see above).  Between ‘Configure Package’ and when the ‘Streaming’ starts there is a ‘Publishing Refresh’ event.

Publishing refresh

Why is there a Publishing Refresh event?  It turns out, it can be triggered by a group policy refresh.

GP Refresh

Notice the time stamps?

So, is there something that occurs during Group Policy refresh that could cause this?

To test this I used procmon to monitor for PackageInstallationRoot registry key and ran a Group Policy Refresh:

screen-shot-2016-09-21-at-8-27-01-am

Procmon_PackageInstallationRoot

The key is deleted then recreated and inbetween the AppV service is trying to read it.

So what is happening here?

At our organization we prefer to set Group Policies using Group Policies Preferences – Registry because of the power and flexibility of Item Level Targetting.  We prefer to only apply registry keys/policies to specific systems in certain groups rather than having a complicated OU structure.  Since I know we have a policy that configures our AppV5 values I went and looked into how this value was being configured:

GroupPolicyManagement

The ‘Action’ is ‘Replace’

screen-shot-2016-09-21-at-8-44-18-am

We have this value set to ‘Replace‘.  What does replace do?

Replace Delete and recreate a registry value or key for computers or users. If the target is a registry value, the net result of the Replace action is to overwrite all existing settings associated with the registry value. If the target is a registry key, the net result is to delete all values and subkeys in the key, leaving only a default value name with no data. If the registry value or key does not exist, then the Replace action creates a new registry value or key.

Well, process monitor is showing EXACTLY that scenario.  Replace is deleting the key and recreating it.  In between that time ‘AppVClient.exe’ is trying to read that value.  If PackageInstallationRoot doesn’t exist, then AppV defaults to ‘C:\App-V Packages” as the PackageInstallationRoot.

screen-shot-2016-09-21-at-10-01-23-am

No “PackageInstallationRoot” key -> “C:\App-V Packages” folder is where packages go.

 

It appears we have a race condition between group policy being applied and when our AppVClient is refreshing.  It is less than 600ms but that’s enough time for a package or two to start their refresh and get caught in ‘default folder’ purgatory.

How can we fix this?

The easiest solution and the one we are going to implement:

We’re going to change the action to “Update”.  When I initially tried to change the action was greyed out.  When “Remove this item when it is no longer applied” is checked, it forces the action to “Replace”.
screen-shot-2016-09-21-at-8-44-27-am

I had to ‘uncheck’ that value

screen-shot-2016-09-21-at-8-44-34-am

And then I could select ‘Update’.

screen-shot-2016-09-21-at-8-44-43-am

After updating the GPO and refreshing policies on the affected system, I ran procmon to verify that it does not delete the value anymore:

GPP_Update

It does not delete the value, it now ‘Sets’ the value to the same string we had before.  I don’t know if the individual ‘Set’ action will cause an issue; I suspect not since the value will always exist and is technically the same value throughout the entire set of actions that this issue is now ‘resolved’ for us.

Read More

AppV 5.1 Sequencer – Not capturing all registry keys

2016-09-20
/ /
in Blog
/

Video of this issue:

https://theorypc-my.sharepoint.com/personal/trententtye_theorypc_onmicrosoft_com/_layouts/15/guestaccess.aspx?guestaccesstoken=vq4pmhseZam8zGPBh6Q8bn%2bgvaZrVuFc4fXo%2fziYmeA%3d&docid=08e53da35fa314929a7ce6578c69bf5c5

The issue is when sequencing an application (100% reproducable on Epic and the VMWare Hypervisor) and then you add a large ‘update’ (for Epic this is the client pack) then not all registry keys are captured.  At 8:20 seconds you can see keys that are present in the local registry are not present in the package.

appv_bug

 

Notice the “0”, “win32” and FLAGS keys are missing in the AppV package.

This is the script I used to compare the local registry vs the package:

 

Read More

Examining Logon Durations with Control Up – Profile Load Time

2016-09-15
/ /
in Blog
/

Touching on my last point with an invalid AD Home Directory attribute, I decided to examine it in more detail as what is causing the slowness on logon.

60s

52 seconds for Profile Load time.  This is caused by this:

bad_attribute

The Home Folder server ‘WSAPVSEQ07’ I created that share and set the home directory to it.  I then simulated a ‘server migration’ by shutting this box down.  This means that the server responds to pings because it’s still present in DNS.

ping

But why does it take so long?

If I do a packet capture on the server I’m trying to launch my application from, and set it to trace the ip.addr of the ‘powered off’ server, here’s what we see:

slow-logon-packet-capture

The logon process attempts to query the server at the ‘28.9’ second mark and stops at the ‘73.9’ second mark.  A total of 45 seconds.  This is all lumped into the ‘User Profile Load Time’.  So what’s happening here?

We can see the initial attempt to connection occurs at 28.9 seconds, then 31.9 seconds, then finally 37.9 seconds.  The time span is 3s between the first and second try then 6s between the second and third try.  This is explained here.

In Windows 7 and Windows Server 2008 R2, the TCP maximum SYN retransmission value is set to 2, and is not configurable. Because of the 3-second limit of the initial time-out value, the TCP three-way handshake is limited to a 21-second timeframe (3 seconds + 2*3 seconds + 4*3 seconds = 21 seconds).

Is this what we are seeing?  It’s close.  We are seeing the first two items for sure, but then in instead of a ‘3rd’ attempt, it starts over but at the same formula.

= 45 seconds.

According to the KB article, we are seeing the Max SYN retransmissions (2) for each syn sent.  This article contains a hotfix we can install to change the value of the Max SYN retransmissions, but it’s a minimum of 2 which it’s set to anyways.  However, there is an additional hotfix to modify the 3 second time period.

The minimum I’ve found is we can reduce the 3 second time period to 100ms.

values

This reduces the logon time to:

faster_logon

19 seconds.

What does this packet capture look like?  Like this:
lower_syn_values

Even with the ‘Initial RTO’ set to 100ms, Windows has a MinRTO value of 300ms:

minrto

After the initial ‘attempt’ there is a 10-12 second delay.

Setting the MinRTO to the minimum 20ms

minrto_20

Reduces our logon time further now because our SYN packets are now about 200ms apart:

minrto_200ms

We are now 16 seconds, 13 seconds spent on the profile upon which 12 seconds was timing out this connection.

16s-logon

 

Would you implement this?  I would strongly recommend against it.  SYN’s were designed so blips in the network can be overcome.  Unfortunately, I know of no way to get around the ‘Home Directory responds to DNS but not to ping’ timeout.  The following group policies have no effect:

gpo1

gpo2

 

And I suspect the reason it has no effect is because the server still responds to DNS so the SYN sequence takes place and blocks regardless of this GPO settings.  Undoubtedly, it’s because this comes into play:
gpo_explain

Since our user has a home directory the preference is to ‘always wait for the network to be initialized before logging the user on’.  There is no override.

Is there a way to detect a dead home directory server during a logon?  Outside of long logon’s I don’t see any event logging or anything that would point us to determine if this is the issue without having to do a packet capture in a isolated environment.

 

 

Read More

ControlUp – Dissecting Logon Times a step further (invalid Home Directory impact)

2016-09-07
/ /
in Blog
/

Continuing on from my previous post, we were still having certain users with logons in the dozens of seconds to minutes.  I wanted to find out why and see if there is anything further that could be done.

60second_profile

 

After identifying a user with a long logon with ControlUp I ran the ‘Analyze Logon Duration’ script:

51-1second_profile

 

Jeez, 59.4 seconds to logon with 51.2 seconds of that spent on the User Profile portion.  What is going on?  I turned to process monitor to capture the logon process:

screen-shot-2016-09-07-at-8-24-18-pm

Well, there appears to be a 1 minute gap between the cmd.exe command from when WinLogon.exe starts it.  The stage it ‘freezes’ at is “Please wait for the user profile service”.

 

Since there is no data recorded by Process Monitor I tested by deleting the users profile.  It made no difference, still 60 seconds.  But, since I now know it’s not the user profile it must be something else.  Experience has taught me to blame the user object and look for network paths.  50 seconds or so just *feels* like a network timeout.  So I examined the users AD object:

screen-shot-2016-09-07-at-8-46-19-pm

 

Well well well, we have a path.  Is it valid?

screen-shot-2016-09-07-at-8-49-42-pm

 

 

It is not valid.  So is my suspicion correct?  I removed the Home Directory path and relaunched:

without_homedir_logon_time

Well that’s much, much better!

So now I want ControlUp to identify this potential issue.  Unfortunately, I don’t really see any events I can key in on that says ‘Attempting to map home drive’.  But what we can do is pull that AD attribute and test to see if it’s valid and let us know if it’s not.  This is the output I now generate:

new_script

 

I revised the messaging slightly as I’ve found the ‘Group Policy’ phase can be affected if GPP Drive Maps reference the home directory attribute as well.

 

So I took my previous script and updated it further.  This time with a check for valid home directories.  I also added some window sizing information to give greater width for the response as ‘Interim Delay’ was getting truncated when there were long printer names.  Here is the further updated script:

Read More

ControlUp – Dissecting Logon Times a step further (Printer loading)

2016-08-31
/ /
in Blog
/

We have applications that require printers be loaded before the application is started.  This is usually because the application will check for a specific printer or default printer and if one is not set (because it hasn’t mapped into the session) then it’ll throw up a dialog or not start entirely.

So we have this value ‘unchecked’ for some applications:

Screen Shot 2016-08-31 at 12.12.08 AM

But how does this impact our logon times?

Well… Our organization just underwent a print server migration/upgrade where some print servers were decommissioned and don’t exist.  But some users still have them or references to them on their end points.  We do have policies that only map your default printer, but some users are on a policy to map ‘all’ printers they have on their system.

What’s the impact?

Screen Shot 2016-08-31 at 12.15.10 AM

Waiting for printers before starting the application…

 

Screen Shot 2016-08-31 at 12.26.50 AM

Without waiting for printers

16 Seconds?  How is that so?

Well, it turns out waiting for printers and the subsystem components to support them add a fair amount of time, and then worse is network printers that don’t go anywhere anymore.  I’ve seen these logons wait for connection before timing out, all the while the user sits there and waits.  The script that comes with ControlUp for analyzing logons is good, but I wanted to know more on why some systems had long logon times and the only clue was Pre-Shell (userinit) taking up all the time.  So I dug into the print logs and found a way to measure their impact.

Screen Shot 2016-08-31 at 12.32.05 AM

With my modified script we can clearly see waiting for the printers takes ~15.4s with a few printers over a few seconds and the rest at 0.5 seconds or so.  One thing about this process is that mapping printers is synchronous.  So when or if 1 stalls, the whole process gets stuck.  All my printers were local except for the ‘Generic / Text Only’ which was a network printer where I powered off the server.  It hung the longest at 5.9 seconds, but I’ve seen ‘non-existant’ network mapped printers hang for 150 seconds or so…

To facilitate finding the printers we need to pass the clientName to the server and the Print Service Logs need to be enabled.

You can enable the print service logs on server 2008R2 by executing the following:

The ControlUp arguments need to look like this now:

Screen Shot 2016-08-31 at 12.40.36 AM

Here is my updated script:

I hope to dig into other startup components and further drill down into what our user launch process looks like.  We wait, and we see 🙂

Read More

ControlUp – List AppV5 recent events on various servers

2016-08-26
/ /
in Blog
/

David Falkus just posted a blog post on using Powershell to combine multiple AppV5 logs into a single view and orders them chronologically so you can see the events as they occurred.

Since this was a PowerShell script we can use ControlUp to import it, tweak it to accept some server variables and then get the output back to us.  Here is a video of this in action:

Here is the recipe for it:

1

2

3

4

 

And the script:

 

Read More

Troubleshooting application error “No Microsoft Outlook profiles have been created”

2016-08-24
/ /
in Blog
/

I was informed an AppV5 application was getting the following error message:

NoMicrosoftOutlookProfilesHaveBeenCreated

So what’s going on here?

The application is trying to create an email message and needs to activate Outlook to add the attachment.  This error can be worked around by launching Outlook *first*, which creates our profile, but I would prefer to not launch programs to use resources in the background if it can be avoided.

 

 

What I’d like to do is silently create our Outlook profile and then continue launching the application.  I didn’t find a particularly good solution for this, but I did eventually stumble across one with some minor modifications to meet my needs:

Or via cmd.exe:

This powershell script will launch launch Outlook into the ‘Inbox’ and terminate.  Since it’s done through ’embedded’ commands the only thing you may see is a brief blip of Outlook with the ’embedded’ icon in the taskbar.

 

Read More