Windows Update

Windows Update – 0x80070308 ERROR_REQUEST_OUT_OF_SEQUENCE

2016-10-31
/
by trententtye
/
in Blog
/
No Comments

We are getting this error when trying to install KB3172605, which was re-released in September of this year.  This patch was originally installed in August without issue but the re-release fails to install.  The WindowsUpdate.log reports:

Examining the CBS.LOG for more information has revealed the following:

The error starts here:

One of the nice things about Windows Update is it kicks up Windows Error Reporting immediately when it detects an error.  So if you run procmon.exe you can find when the error occurs and work backwards only a little bit to, usually, find why it crashed.

In this case I ran procmon while I tried installing this update, went to 11:25:10 and did a search for “Windows Error Reporting”

windowsupdatebug1

The actual Windows Error Reporting kicked in at the check to see if it is disabled (DisableWerReporting).  The last key it read before it crashed was:

“HKLM\COMPONENTS\DerivedData\Components\amd64_0e471cf709070f76ea5797942bb36096_31bf3856ad364e35_6.1.7601.23455_none_9b9ebc8fa6659c8e”

If I open that key in Regedit here is what it looks like:

windowsupdatebug2

And what another key in that same list looks like:

windowsupdatebug3

 

It appears the key that is in error is missing some information.  In order to fix this I’m going to another system and try to grab they values needed for identity and S256H.  I don’t know how these keys are generated but I’m hoping they are generated by the file it’s referring to which *should* mean that a same or similar level patched system should have the same files and thus the same generated hash’s.

I went to another system and exported the values.  In order to find the proper key, we find it is referenced by the previous key in the procmon trace:
“amd64_microsoft-windows-b..environment-windows_31bf3856ad364e35_6.1.7601.23455_none_c7bdc8a2bca7…”

The key path was present in both systems because they were at the same patch level but the look of the key is different because of GUIDs are generated:

windowsupdatebug10

Working system – Notice the highlight is different

By tracing back in the working system I exported the key that “c!” is referencing.

Working System has S256H, identity keys

Working System has S256H, identity keys

I exported out the key from the working system and had to edit it so the GUID matched the broken key’s values.  I took the ‘working’ registry:

And the broken key:

I added the S256H and identity values to the ‘broken’ key:

And then added the last “c!” line by following these next steps:

  1. copied the line out:
    “c!c4ebacc5355..93c10e6175a_31bf3856ad364e35_6.1.7601.23455_8459b72d1e2f2700″=hex:
  2. Pasted the value from the registry “key” underneath with the “broken” value:
  3. Starting at the ellipse, counted the number of characters from the “55..” to “93′ underneath and created the value:
  4. Lastly, removed the ‘_none” from the string:
  5. I could then add “c!” in front and added it to my ‘broken’ registry file:
    I then tried to install the patch:

    windowsupdatebug9

And it worked!

Read More

Windows Update – Errors 80070057, 800736B3, 8024400E

2016-10-19
/
by trententtye
/
in Blog
/
No Comments

We started the new patching Microsoft has put forward (cumulative updates) and one of our Citrix vDisks had an issue with it. Windows Update would say that there were no updates available:

windowsupdatebug

But we very obviously have updates to deploy to it.

When I checked the ‘WindowsUpdate.log’ I saw the following:

0x80070057 = E_INVALIDARG

The ‘CBS’ (Component Based Servicing) is reporting an Invalid Argument.  Microsoft keeps a more verbose log of the component based servicing here: C:\Windows\Logs\CBS

This log reported the following:

An error appeared to occur at this point (there were a few):

Failed to get component version: 6.1.7601.22653
Failed to enumerate store versions on component: x86_microsoft-windows-c..ityclient.resources_31bf3856ad364e35_6.1.7601.22843_da-dk_c1126f6226996014
Failed to enumerate related component versions on component: x86_microsoft-windows-c..ityclient.resources_31bf3856ad364e35_6.1.7601.22843_da-dk_c1126f6226996014

 

The standard MS method of troubleshooting Windows Update is to run the ‘CheckSUR’ utility.  This was the result of running that tool:

windowsupdatebug2

No errors detected.  Awesome.  So I have a problem but this tool reports everything is peachy.

So when I looked at Windows Update I saw numerous ‘failed’ updates.

windowsupdatebug3

I took one that was failed and downloaded it (KB3177467) and attempted to manually install it.

This time I got a different error:

0x800736B3 – ERROR_SXS_ASSEMBLY_NOT_FOUND

Looking at the CBS.LOG showed me the following:

Ok, so now we can see the error but it still doesn’t give us much information.  If I run ProcMon I can find *when* the error occurs somewhat easily because Windows Error Reporting kicks in as soon as the error occurs:

windowsupdatebug4

My apologies for such an ugly screenshot.  The ‘Dark Blue’ highlight is when Windows Error Reporting kicked in.  So the error must have occurred immediately preceding it.  The only line that had a ‘NAME NOT FOUND’ without being a subkey search is the line that ends in v!6.1.7601.18766.  If I browse to that location in the registry, here’s what I find:

windowsupdatebug5

 

So we are definitely missing that key.  I have a bit of an advantage with the patches here as I have a duplicate of this server in another vDisk that was made around a year ago.  Both are supposed to be at the same patch level, but it allows me to look through it’s COMPONENTS registry hive and see if it has that key…

And I see a vastly different set of keys:

windowsupdatebug6

So I import that key and try running the update again…

windowsupdatebug7

 

Manually, it successfully updated.  So now I tried running Windows Update again:

windowsupdatebug8

Code 8024400E.  Which means…  I just need to rerun ‘Try again’ about 6 times.  Once I do:

windowsupdatebug9

Ok, we have updates.  As a test I’m going to do just one:

windowsupdatebug10

 

Success!

windowsupdatebug11

So I’ll try the rest:

windowsupdatebug12

 

Success!

windowsupdatebug13

 

It appears we can install patches again.  I’m concerned the COMPONENTS hives were so different but without a solid understanding of how that hive comes to be (I wish you could rebuild it) I think I may be stuck with assuming that a failed update or maybe a corrupt registry key is at fault for that missing key breaking updates.  I guess we’ll see what happens next month to see if we can patch Windows :/

Read More

Microsoft Patch (KB3170455) breaks PrintBRM importing printers with drivers

2016-07-29
/
by trententtye
/
in Blog
/
1 Comment

We have an application (ARIA MO) that has some special requirements.  The application requires all printers used by it in the different locations to be manually loaded on the Citrix server with all drivers.  This totals around 220 or some odd printers installed with around 15 different drivers loaded.  The printers must have a local port and cannot be mapped via TCPIP or network mapping.

Our design goals for our Citrix environment are to minimize the various PVS images we use so we use various ‘layers’ to allow a single master image to be able to host various unique and difficult configurations.  For this application we use AppV as our layering technology to put the application on the server, but for the printers we use a script to load them onto the server.  What we have is a print server that hosts all the printers needed by this application and we can export the printers into a file using Print Management.  Then we save that file on a network share somewhere.  When the Citrix server boots, I can take that file and manipulate its contents to change the queues to ‘local’ queues then import that modified file to the Citrix server.  I configured the script to take two parameters, a print file name and a server name.  I call the script with a command line like this:

The powershell command it calls is here:

So, what does this have to do with KB3170455?  Well, since installing KB3170455 it prevents importing printer files with print drivers embedded.  This is what it looks like with KB3170455 installed:

Screen Shot 2016-07-29 at 9.04.48 AM

Without:
Screen Shot 2016-07-29 at 9.07.35 AM

And the failure import with 3170455 installed:


Remove KB3170455 and the import works without issue.

Read More

WSUS clients fail with WARNING: Exceeded max server round trips: 0x80244010

2016-03-14
/
by trententtye
/
in Blog, Uncategorized
/
2 Comments

J C Hornbeck/Joe Tindale touched on this topic on a Microsoft blog post here.

In that post he touches on what’s actually happening:

Cause: The error, 0x80244010, means WU_E_PT_EXCEEDED_MAX_SERVER_TRIPS and happens when a client has exceeded the number of trips allowed to a WSUS server.  We have defined the maximum number of trips as 200 within code and it cannot reconfigured.  A “trip” to the server consist of the client going to the server and saying give me all updates within a certain scope.  The server will give the client a certain number of updates within this trip based on the size of the update metadata.  The server can send 200k worth of update metadata in a single trip so it’s possible that 10 small updates will fit in that single trip.  Other larger updates may require a single trip for each update if they exceed the 200k limit.  Due to the way Office updates are published you are more likely to see this error if you’re syncing Office updates since their metadata is typically larger in size.
I’ve bolded the more important information.  This is hardcoded and cannot be reconfigured.  This, to me, is a bit ridiculous that it can’t be reconfigured.
I have a WSUS client where we ‘reset’ the Windows Update client.  After resetting the client we were getting an error “WARNING: Exceeded max server round trips: 0x80244010”.  We would try multiple times but this error wouldn’t go away and prevented us from running Windows Update on this system.  So I started to investigate.  The first thing I did was finish reading that blog entry.  Hornbeck continues:
The client takes these new updates as they trickle down and inserts them into a small database to cache them for future use.  So during the first client synchronization with WSUS the client may get 75% of the available updates, put them into the database, and then fail at some point due to the number of max trips being exceeded.  The good news is the second synchronization cycle will not need to start from the beginning since the client has already cached 75% of the updates into its database.  The second cycle will pick up where it left off and most likely finish getting all the updates from the server.  There have been a few rare cases where a third scan cycle is needed but more often than not two is sufficient.
Again, I have bolded and underlined the important parts.
I started my investigation by trying to replicate the problem.  I started up Windows Update and ‘checked for updates’.
Ok, no problem.  So I checked again.
Well, Hornbeck/Tindale did say it may take a couple passes.  Let’s try again.
I’m getting worried now…  Let’s try a fourth time.
Hmmm…   At this point I wanted to better understand what Windows Update was doing.  I originally installed Wireshark to trace the conversation but it was difficult and time consuming to try and count the traffic back and forth to the WSUS server.  So I reverted my system and installed Fiddler2 on it.
From the video you can actually see the traffic from the WSUS server.  The request for the ‘Updates’ starts at item 3 and completes at 203.  Exactly 200 round trips.
Since my previous Windows Update attempts at the WSUS server failed after a few tries I thought I would trace the traffic with Fiddler for the multiple attempts.  My logic was I wanted to know if the traffic was ‘looping’; repeating itself and getting to the limit preventing updates.  Or, would each send/receive be unique and thus, simply, more is needed?
The first bit of the first run.  If the second run as identical or near identical traffic ‘packet sizes’ I would be concerned it’s looping…
I reset Fiddler and started the second run.  Completely different!
When I started the second run I was happy to see it was a completely different result.  I cleared Fiddler for a 3rd time ran the ‘Check for Updates’ until it timed out again, and cleared Fiddler again.  I then thought to just let Fiddler capture everything.  There really is no need to clear it each time.  I monitored the Fiddler output looking for loops or patterns.  The update check timed out a forth time (as before).  There was no looping I could see.
Finally, on the fifth run:
Updates!  We have updates!
In the end, when the Microsoft blog post was written (2008) there probably was only enough updates that two or three passes would go through all of them.  As time as gone on and more and more updates have been deployed to systems this hardcoded maximum is doing a huge disservice.  Our Windows 2008R2 SP1 systems require FIVE passes of clicking/waiting/clicking/waiting/etc.
A natural solution to this is to expose the “max server round trips” variable and allow it to be programmed by the organization according to their needs.  The present state of this issue is unnecessarily confusing and arbitrarily limited.
Read More

ERROR 0x8024402c Windows Update

2013-05-16
/
by trententtye
/
in Blog, Uncategorized
/
No Comments

Recently, I was applying Windows Update to a 2008 system and it failed on 4 updates after being successful for months.  I was unsure why, but the updates were Office updates.  I don’t think that the fact they were Office updates are important, but it’s something to mention anyways.

Symptoms of the issues I found and the resolution for this issue.

1) Getting “ERROR 8024402C” when running Windows Update.
2) Checking %WINDIR%\WindowsUpdate.log reveals lines like:

To determine the cause of the issue, I used the nicely revamped Event Viewer and looked at the BITS-Client logs.  Which was a waste, nothing showed up there.  I checked the WindowsUpdateClient log and nothing was there either.  I then learned BITS uses WinHTTP when I was googling for this issue and saw there was a WinHTTP log file.  (You may have to enable analytics and debug logs).  I enabled the Diagnostics Log.

When reattempting to execute Windows Update I went back into the log and scanned through it.  I found the following:

Windows update was going to the wrong server!  The event viewer said it was going to wswsus02.XXXX.ab.ca.  This was our old server address and we since replaced it with going directly to the IP of that server via GPO.

Checking regedit for the WU preferences showed it was pointing to the correct server, but for some reason Windows Update wasn’t picking up the new server.  Rebooting the machine and refreshing the GPO did not resolve the issue.

This is the correct settings

Saman suggested some fixes:

net stop wuauserv
net stop BITS
net start wuauserv
net start BITS
wuauclt /resetauthorization /detectnow
wuauclt /reportnow

These did not appear to work however.  But, we did try the following:
esentutl /p %windir%securitydatabasesecedit.sdb /o
Gpupdate /force

And I believe this worked.  After running this command, WinHTTP reported that it was pulling the Windows Update from the Microsoft servers, not our WSUS server:

au.download.windowsupdate.com is not our WSUS server

At this point I could have probably ran the net stop and net start commands and it may have worked, but I rebooted the server instead.

Upon the server coming back up I reran Windows Update and confirmed it was pulling from our WSUS server:

Success!

Windows Update then downloaded and installed the updates successfully!

Read More