BSOD

Corrupt Registry Repair with Citrix Provisioning Services

2018-04-02
/ /
in Blog
/

I encountered an interesting issue and worked through a solution with a corrupt registry.  The issue seemed innocuously enough, we upgraded PowerShell to 5.1.  Upon reboot I encountered a bluescreen with code 0xF4:

I have encountered this issue before, but I haven’t recorded my troubleshooting steps until now.

Since this was a PVS target device, the easy method was deleting the version and trying to upgrade PowerShell to 5.1, which resulted in the same BSOD.  So it was easily reproducible.  So I tried it a few more times, because, why not?

I then deleted this version, and booted into the system and looked at Event Viewer for hints of what could be at fault.  Going through it was pretty obvious that it was registry corruption:

Filtering for EventID 5 shows all the attempts of booting to BSOD:

 

I mean, it literally says the Registry was corrupted 🙂

Where can you find this corruption?  With PVS it’s fairly simple but I believe the same process can exist for other systems including physical.  The first step was to mount the vDisk to a VM.

 

Mount the registry hive you suspect with corruption.

Next is to scan the registry for corruption.  Thus far, I’ve only found corruption to be detectable if it’s a key or value that cannot be read.  If the data on a value can be read but contains garbage it’s much harder to detect.  In order to avoid permissions being a problem, I open a PowerShell prompt as SYSTEM using PSEXEC.  If you don’t elevate permissions, some keys maybe restricted from the Admins group and this will be detected as a failure.

Once at this stage, it’s a one-liner to scan the registry:

In my experience, corruption can be detected as “Permission Denied”, “Access Denied”,”Path does not exist” or some such:

 

At this point you can examine the text file to see the last path it explored:

 

At this point you can open regedit (as SYSTEM) and examine the keys within that path.  Clicking through each one revealed the corrupted key:

Attempting to get Permissions on this key reveals it also exists on the ACL level:

“The requested security information is either unavailable or can’t be displayed”.

Deleting the key may fail as well:

At this point you need to evaluate how to manage the corruption.  If you cannot delete the key, rename it, or in some way replace it you may have an option like I did…  You can rename a higher up branch in the tree, go to a existing system with the same keys (with PVS I can go to the previous version and export that tree) and reimport.

Unload the hive and boot up the system –> and you may have a fully working system!

I’ve used this trick here, and on a corrupt COMPONENTS hive in the past.  With the COMPONENTS hive I got lucky I could replace the corrupted keys with ones from a branched vDisk.  Other machines didn’t have the same key in COMPONENTS so I got lucky.

 

Read More

PNP_DETECTED_FATAL_ERROR

2015-09-11
/ / /

I rebooted my computer to this lovely Blue Screen Of Death (BSOD) message:

PNP_DETECTED_FATAL_ERROR
Attempting to reboot into Safe Mode also resulted in the same message.  I was able to boot into ‘Recovery Mode’ which is a ‘Windows PE’ mode that runs a stripped down version of Windows in RAM.  From here I enabled the network ‘Kernel Debugging’ by configuring some parameters in the BCD file.
The two parameters I set where:

bcdedit /store C:\boot\bcd /debug on
bcdedit /store C:\boot\bcd /dbgsettings net hostip:192.168.1.101 port:49152


I needed to set the “/store” parameter to ensure I was manipulating my non-booting BCD file, and not the BCD file that Windows Recovery boots from.  Write down the key or save it someplace, you’ll need it on the ‘host’ computer (see in the above screenshot).

Once here I downloaded and installed ‘WinDBG.exe‘.  Open windbg.exe and choose “File > Kernel Debug“.  On the ‘NET’ tab, enter your ‘Port’ number and ‘Key’ (everything to right of the equal sign) and click ‘OK’.
Even though I ‘enabled’ debug in my BCD file, I found I still needed to tap the ‘F8’ key while booting and select ‘Debugging Mode’.  Once selected, my windbg.exe on my host computer sprang to life!

It turns out you need to enable symbols or else you get an incomplete picture.  After enabling symbols and running !analyze -v I got the following:

ctxusbm.  This is a Citrix driver for their Receiver client that passes through USB to a Citrix session. I had updated Receiver to 14.3.0.5014 last month and I probably hadn’t rebooted my computer until Windows Update made me.  So that’s probably why I’m experiencing this issue now.  To fix this issue, I rebooted into the Windows Recovery mode and deleted all instances of ‘ctxusbm’ from the SYSTEM hive.  Specifically, I deleted these locations:
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{CF2A3345-050B-41D0-BAF5-CD558EFAAE3B}
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Enum\Root\LEGACY_CTXUSBM
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\ctxusbm

Upon the next reboot, my computer came back cleanly and operates without any issues.  I am going to keep this module removed until the next version of Receiver is released, hopefully, I won’t have any more issues.  Issues with ctxusbm seem relatively prevalent with Citrix.

Read More