Citrix Provisioning Services – Convert your VHD’s to VHDX

/ /
in Blog

Per my previous post, we have been upgrading our storage backend and one of the features is 4K sector sizes being used.  This is an issue as VHD files will not mount when they reside on 4K storage.  Since we want this feature our two choices are to downgrade our storage to 512B sector sizes or move to the VHDX format.  I would prefer to move to the VHDX format as it has a small performance advantage (~7.5% was what I measured) and future proofs us a little bit for when we upgrade our PVS servers to 2012.

The VHDX supports 512e on 4K storage which is only supported on the following operating systems:

  • Windows Vista
  • Windows 7
  • Windows Server 2008*
  • Windows Server 2008 R2*
  • Windows Server 2012
  • Windows 8

So Windows Server 2003 is out, we are going to have to deal with not being able to mount their vDisks.  For our Windows Server 2008 R2 servers we can convert our VHD files to VHDX.  This is the process I have developed.  Because we are going to be mounting the VHD and VHDX files you will need your files to reside on 512B storage.  Without further ado, here are the steps to convert your VHD to VHDX:

  1. Create your VHDX file.  Ensure to select 512B if you are migrating an operating system older than Server 2012.
  2. Mount your VHD file and your VHDX file.  Format and set the partition as active for the VHDX if needed.
  3. Use a clone utility (DriveImage XML, Windows Server Backup or HDDRawCopy) to clone your VHD to the VHDX via the mounted paths.
    1718 1920
  4. Take this opportunity to defrag your cloned VHDX.
  5. Assign your vDisk to your target device.
  6. Done!


Read More

Citrix Provisioning Services 7.7 – Unable to mount VHD files

/ /
in Blog

We recently upgraded our storage where our PVS vDisk’s reside.  I was attempting to inject a file using ‘Mount vDisk’ from the PVS Console and I got an error message:


The text of the message states:
“When accessing a new tape of a multivolume partition, the current block size is incorrect”.

Google searching this message does not bring any helpful results.

I then tried to mount the vDisk via the command line:


Since we still had our ‘older’ storage I tried mounting the same vDisk.  It worked.  So something with our new storage was preventing us from mounting our VHD’s.  We could still create new versions and boot our target devices off the new versions without issue, but mounting the vDisk’s to add a file or change a single registry key is so much more efficient and faster.  We have 5 different datacenters with different storage at each, with 1 of the datacenters about to undergo another storage migration (so 6 different storage ‘filers’ in total).  I attempted to mount a vDisk from each:


A pattern immediately emerged on the far right of this procmon trace.  The ‘filers’ that allowed us to mount the VHD file had the ‘QuerySizeInformationValue’ report back a ‘BytesPerSector’ of 512.  The filers that failed had a BytesPerSector of 4,096.  Immediately, it became obvious that the new storage had a 4K format and the older storage had 512B format.  Microsoft has a statement about VHD files and 4K support:

Effect when hosting VHDs on native 4K disks

The VHD driver today assumes that the size of the physical sector of the disk is 512 bytes and issues 512-bytes IOs. This makes the VHD driver incompatible with these disks. Therefore, the VHD stack does not open the VHD files on physical 4 KB sector disks.

Well, there you go.  You cannot mount VHD files when they reside on 4K storage.  Fortunately, PVS 7.7 supports VHDX files with 512e support with Windows 2008R2 so we can move our VHD’s to VHDX’s.  I will document how to do so in my next post.

Read More

Group Policy Preferences – Scheduled Task fails to apply

/ / /
We had a couple issues with scheduled tasks not applying when submitted as a GPP (Group Policy Preference).  We turned on tracing via local gpedit.msc (Administrative Templates > System > Group Policy > Logging and tracing).  From here we turned on the Scheduled Task logging and events were then stored in the eventvwr.msc (we also turned on tracing which stored a computer.log file here: C:\ProgramData\Group Policy\Trace)
The first error we got was:
So it can’t map between user ID’s.  It’d be nice if it told us which mapping failed, but it gives us a pretty good hint. Looking at the XML file the GPP creates (stored here:  “C:\ProgramData\Microsoft\Group Policy\History\\Machine\Preferences\ScheduledTasks\ScheduledTasks.xml” )
We saw the following:
Everything validates.  Googling for BUILTIN\SYSTEM brought up that several people were getting the same error when using BUILTIN\SYSTEM.  Which makes some sense as “BUILTIN\SYSTEM” isn’t a real account.  We renamed it to NT AUTHORITY\SYSTEM.  This time we got a new error message:
This doesn’t tell us a whole lot of information.  What is the unexpected node? Looking again at the XML file it looked like so:
The difference that I can see:
The SYSTEM account is NOT a group.  We changed how we selected the SYSTEM account by “Browsing” AD, going into the root of the domain, going into the Builtin OU, and selecting SYSTEM.  This populated as “NT AUTHORITY\Well-Known-Security-Id-System”.  This will fail because there is no such user account called “Well-Known-Security-Id-System”.  At this point we renamed it to “NT AUTHORITY\SYSTEM”.
Boom, GPP Scheduled task now worked without issue.  Checking the XML the difference by manually selecting the SYSTEM account changed
If you are having issues with your GPP Scheduled task item running as the SYSTEM account I would HIGHLY recommend you check your XML file and confirm it is set as “NT AUTHORITY\SYSTEM” and it is surrounded by UserId NOT  GroupId.
Read More

Move ALL Windows 7/2008/2012 log files to another drive

/ / /

Read More

Powershell script to manipulate “Register this connection’s addresses in DNS”

/ / /

We run a multihome NIC setup with our Citrix PVS servers and the “Provisioning” Network is a seperate VLAN that is only used by the PVS servers and goes no where.  Unfortunately, however, the “Provision” NIC can register itself in the DNS, causing devices outside of the Provisioning network (everyone) to resolve to the incorrect address.  To resolve this I ran this script across all my vDisks to remove the ability of the provision network to register itself as available in DNS.:

As you can see above, we have two NICs, “Provision” and “Production”.  We do not want “Provision” registerted so it gets the $false,$false set, whereas “Production” gets $true,$false.

Read More

SCVMM: Install VM components Failed

/ / /

I’m attempting to deploy a virtual machine from a template but I get this error: Install VM components: Failed.



SO…  Long story short; if you are encountering this error, I would suggest booting your VHD file in a VM and re-sysprep /generalize it.  If you’ve maxed out on sysprep’s I have a post earlier in my blog on how to get around the 3-times limit and rerun sysprep.  Alternatively, you can try what I did, but I can’t gaurantee your success and replace the BCD file in your Library VHD with a BCD you *know* has been sysprepp’ed and try redeploying it.

I’ve ensured that HTTP and HTTPS is not blocked (firewall is disabled) and the agent on my SCVMM machine is installed and running.  So this error message is somewhat useless.

So I took my two boxes, 2012-SCVMM (the SCVMM server) and S5000VXN-SERVER (the Hyper-V host) and procmon’ed them while it was attempting to “Install VM Components” to try and understand what it’s trying to do.

After reinitating the task, we can see that the vmmAgent.exe on the Hyper-V host accesses the file about 2 seconds after I submit the command to retry the job.

A few seconds after this, it appears to “CreateFile” in the WindowsTemp directory; but this is not actually correct.

What it is really doing is *mounting* the VHD into a folder in the WindowsTemp directory.  You can actually watch this in action if you view Disk Management on the Hyper-V host while you execute the task.

So now that we know it’s mounting the file as a volume this helps us narrow down on what Hyper-V is attempting to do…  And I suspect what it is attempting to do is “offline servicing” of the attached vdisk/vhd.

After attaching the vdisk the next thing it does it query the BCD file on the system.  Maybe it needs to be in a certain mode to operate?  I’m not sure…

Continuing on we can see that events are written to the Microsoft-Windows-FileShareShadowCopyProvider Operational.evtx, System.evtx, and Microsoft-Windows-Bits-CompactServer Operational.evtx event logs.  Examining each log at the time stamp showed the FileShareShadowCopyProvider and System log were just noticed of volume shadow copy starting, but the BITS event log was interesting.



It showed that it was doing something with the BCD file.  The Hyper-V host was *serving* it out.  I suspect it was serving it to the SCVMM.
Looking back at the SCVMM server we see that was executing some WinRM commands.  Sadly, we do not know what commands it was trying to send.

If I mount the vhd file and check out the BCD file I can see that it appears to be corrupted in that it doesn’t know what the proper boot device should be.


It’s not actually corrupted though; the reason why the devices are unknown is because bcdedit isn’t finding the disk signature of the volume I mounted.  But it has the disk signature because I can boot with it without issue.

Continuing on…

In order to try and find out what commands WSMAN was sending I enabled debugview on the SCVMM server:

I then reproduced the error by rerunning the Create Virtual Machine job.

DebugView gave me more information to narrow down what was happening.  It appears that the process is failing with:

Doing some googling on this took me to a Korean Microsoft page where the following was stated:


Thinking about it, I do not think my VHD was SysPrep’ed.  I find it interesting that sysprep appears to do something to the BCD file.  To find out what sysprep does to the BCD file I booted up my 2012 VHD into Windows and ran sysprep, generalize and shutdown.

And……….?  Lets go to DebugView:


Voila!  It appears much better than before.  It found the drive correctly and checked the BCD file and found it is the “Generalize” state.  The *actual* image I originally made was NOT sysprep’ed and all I did was replace the BCD file, but it allowed it to continue beyond and the machine actually completed the imaging process properly.  It joined the domain and whatever else the answer file was I gave it in SCVMM.

It appears I was correct in my earlier assumption about what SCVMM is doing.  It goes and grabs the BCD file and transfers it from the target machine to itself, “fixes it up” (not sure what it’s doing at this stage precisely), then sends it back for injection.  Certainly a bit of a complicated process with a fair bit that can go wrong, but shame on Microsoft for having such a poor error message.  I suspect it wouldn’t have required much effort to push out a real message; something to the effect of, “The BCD of this vDisk does not appear to have been through the sysprep /generalize process.  Please rerun sysprep against the image and try again”.

SO…  Long story short; if you are encountering this error, I would suggest booting your VHD file in a VM and re-sysprep /generalize it.  If you’ve maxed out on sysprep’s I have a post earlier in my blog on how to get around the 3-times limit and rerun sysprep.  Alternatively, you can try what I did, but I can’t gaurantee your success and replace the BCD file in your Library VHD with a BCD you *know* has been sysprepp’ed and try redeploying it.

Read More

ERROR 0x8024402c Windows Update

/ / /

Recently, I was applying Windows Update to a 2008 system and it failed on 4 updates after being successful for months.  I was unsure why, but the updates were Office updates.  I don’t think that the fact they were Office updates are important, but it’s something to mention anyways.

Symptoms of the issues I found and the resolution for this issue.

1) Getting “ERROR 8024402C” when running Windows Update.
2) Checking %WINDIR%\WindowsUpdate.log reveals lines like:

To determine the cause of the issue, I used the nicely revamped Event Viewer and looked at the BITS-Client logs.  Which was a waste, nothing showed up there.  I checked the WindowsUpdateClient log and nothing was there either.  I then learned BITS uses WinHTTP when I was googling for this issue and saw there was a WinHTTP log file.  (You may have to enable analytics and debug logs).  I enabled the Diagnostics Log.

When reattempting to execute Windows Update I went back into the log and scanned through it.  I found the following:

Windows update was going to the wrong server!  The event viewer said it was going to wswsus02.XXXX.ab.ca.  This was our old server address and we since replaced it with going directly to the IP of that server via GPO.

Checking regedit for the WU preferences showed it was pointing to the correct server, but for some reason Windows Update wasn’t picking up the new server.  Rebooting the machine and refreshing the GPO did not resolve the issue.

This is the correct settings

Saman suggested some fixes:

net stop wuauserv
net stop BITS
net start wuauserv
net start BITS
wuauclt /resetauthorization /detectnow
wuauclt /reportnow

These did not appear to work however.  But, we did try the following:
esentutl /p %windir%securitydatabasesecedit.sdb /o
Gpupdate /force

And I believe this worked.  After running this command, WinHTTP reported that it was pulling the Windows Update from the Microsoft servers, not our WSUS server:

au.download.windowsupdate.com is not our WSUS server

At this point I could have probably ran the net stop and net start commands and it may have worked, but I rebooted the server instead.

Upon the server coming back up I reran Windows Update and confirmed it was pulling from our WSUS server:


Windows Update then downloaded and installed the updates successfully!

Read More

Slow Group Policy Client Side Extensions login with Windows 7

/ / /

We experienced an issue when we modified a GPO to include item-level filtering on an AD group.  The issue was that Windows 7 machines with this GPO applied to where suddenly taking minutes to login.  Windows XP machines, however, logged in almost instantly.

When going through the event logs for group policy on Windows 7 we were able to identify the CSE causing this issue.  For us it was the “File processing extension”.

When we looked at the group policy we saw that the item-level filtering was filtering on a group with 11,000+ objects in it.  We had two tasks in the GPO that were filtering on that group.  When I attempted to open the group utilizing ActiveRoles Server (ARS) it was taking 40-50 seconds to populate each object in the group.  I theorized that it appeared Windows 7 was iterating through each object like ARS was.  To test this I installed Wireshark on the Windows 7 and XP machines and ran “GPUPDATE /FORCE”.  This triggered the CSE to execute.  The following are the traces:

XP Capture.  It queries (highlighted) the group then continues on.


Windows 7 Capture.  It queries the group then all objects within the group.

Obviously, with 11,000+ objects in the AD group Windows 7 will have a significantly slower logon if it’s querying every object within the group.  Fortunately, Microsoft has put out a fix for this:

You experience a long domain logon time in Windows Vista, Windows 7, Windows Server 2008 or Windows Server 2008 R2 after you deploy Group Policy preferences to the computer

So if you are experiencing slow login times with Windows 7 it maybe worth it to try this fix.

Read More

Internet Explorer Maintenance Odd Behaviour

/ / /

Recently, I was experiencing some odd behaviour on some of our Windows boxes.  We were getting redirected to a proxy server but RSOP.MSC and group policy showed that we should have been good in the GUI.

The issue is IE Maintenance will store it’s settings in a INS file as opposed to the registry and for some reason IE Maintenance in GPEDIT.MSC doesn’t display the values.  This document details it more:


The INS files created via GPO are placed in the following locations:
2003-“C:\Documents and Settings\trententtye\Local Settings\Application Data\Microsoft\Internet Explorer”
2008-“C:\Users\trententtye\AppData\Local\Microsoft\Internet Explorer”

Read More

iSCSI, iPXE, Microsoft iSCSI Software Target, Microsoft DHCP

/ / /

I was recently struggling to get iSCSI working for myself.  Here were my problems and how I eventually fixed them.

To get iSCSI working the first thing I did was install the Microsoft Software Target 3.3 on a server machine. It’s pretty simple.  Go to iSCSI Targets and create a new target.  Whatever you name it will be on the IQN so I’ve kept it short.  The iSCSI Initators Identifiers is like MAC Address filtering for a DHCP server, if you don’t match you can’t connect to the iSCSI target.  So set the SII to whatever IP, MAC Address or IQN you plan on using.  If you’re like me and wasn’t sure what the IQN will be set to, preferring to have iPXE’s generated one; the Windows event viewer Application log will tell you what the IQN is when it fails to connect.  Look for Event ID 22.

Highlighted is the IQN that tried to connect and failed.  Add it to the SII and you’ll be good to go the next time.

When you are done creating the target you need to create the Virtual Disk (VHD) for the target.  Right-click on the target and go “Create Virtual Disk for iSCSI Target” and follow the prompts until you have your vDisk associated with the target.

We are now done on the server side.  To test the connection I launched iSCSI initiator on the same server and tried to connect to the iSCSI Software Target. It had a “Target Error” as the error message.  To get around this I enabled loopback.

I could now connect to the iSCSI target that resided on the same server and validate the software target was working correctly.

Now to get the client side working.  My goal with this is to PXE boot some client machines with differencing VHD’s off a golden Windows 7 image.  I encountered numerous issues attempting to do this and I’ll share with you my problems and how I solved them.

The first thing I did was configure PXE booting off the Microsoft DHCP server.  I was going to make it launch PXELinux prior to iPXE and may modify my DHCP to do so in the future, but below is a screenshot of my working settings:

I don’t think 043 Vendor Specific Info is required.  I put it in when I was trying to get PXELinux working.  This is a working DHCP config.

When I was trying to get PXELinux working with iPXE I ran into numerous problems.  Before I get too far ahead here, this was my PXELinux default config file:

# These default options can be changed in the geniso script
SAY iPXE ISO boot image
DEFAULT ipxe.lkrn
LABEL ipxe.lkrn
 KERNEL ipxe.krn
 INITRD ipxe.ipxe

Can you spot the error already?

  set keep-san 1
  sanhook –drive 0x80 iscsi:
  sanboot –nodescribe –drive 0xe0

My ipxe.ipxe script file.  Everything I read said this should work.

PXELinux was booting the iPXE kernel, The iSCSI target wasn’t detected natively by the Windows 7/2008 DVD’s (which everything I read said that it should be).  To correct this I made a modified WinPE with the iSCSI Initiator builtin.  When I booted up that WinPE I could start the MSISCSI service and connect to the target.  I thought things looked good and I could format the target and make folders/files on it.  I could even bootsect it.  I then mapped a network drive to the Windows installer files (which iPXE recommends) then ran setup.exe.  I got to the part where I could choose the hard disk but Windows wouldn’t let me install on it giving me an error message.

“Windows cannot be installed to Disk <#> Partition <#>. (Show details)” -> “Windows cannot be installed to this disk. iSCSI deployment is disabled since no NICs referenced in the iBFT can be resolved to actual NT-visible devices. Windows cannot be installed to this disk. This computer’s hardware may not support booting to this disk.

Alright, at least I was making progress.  From here I had to find out what a iBFT was and why can’t Windows resolve it when I’m staring straight at the hard disk.  The importance of it is found here.

“When the pre-boot loader starts, it loads a real-mode network stack driver. The loader contains an iSCSI initiator which logs on to the iSCSI target and mount the boot disk to the system. Another function of the loader is to populate the iSCSI Boot Firmware Table (iBFT), which is required for iSCSI boot. The Boot Parameter driver in Windows will load the parameters from the iBFT, and the Microsoft iSCSI Software Initiator will be able to connect to the iSCSI target using the parameters set in the iBFT. The importance of the iBFT is to be able to share the parameters between the iSCSI boot initiator (which establishes the session in the preboot phase) and the Microsoft iSCSI initiator (which establish the session after Windows boots).”

So my iPXE isn’t storing it’s values in the iBFT.  This seemed weird to me because I could see it preserving the connection when I booted.  Everything I read said the “set keep-san 1” is the magic to storing the iPXE values in the iBFT.

Isn’t that what Registered SAN Device 0x80 means?  That it’s registered in the iBFT? 

So it appeared to be working and I thought maybe it’s Hyper-V.  I read other people using Hyper-V were able to get this to work, but I’m stumped.  It won’t work for me.  So I tried VMWare Player.  I had the same issue there.  No NICs referenced in the iBFT even though I could use the MS Initiator to connect to the target.

At this point I was getting pretty frustrated.  For all intent and purposes this should work.  So I went back to ipxe.org and read their Windows Deployment Service guide and setup WDS and made a WDS wim thinking maybe it had some magic touch; even though I had read that the standard install media should work.

Well, the WDS image didn’t work either.  At this point I went back to ipxe.org and followed their PXE Chainloading guide exactly.  Well, mostly exactly.  I didn’t want to make a reservation so I applied the setting across the entire DHCP by entering everything in Server Options (see the first screenshot of this post).

Like magic, as soon as I changed the kernel from iPXE.KRN to undionly.kpxe it showed me something different on the Hyper-V console:

This *feels* more promising.

Like Magic the undionly.kpxe kernel spewed forth more information than the iPXE.KRN.  I now had a problem where I wanted to boot from the local CDROM and not from the SAN Device, but *after* it became registered.  This is my script:


#  set skip-san-boot 1
set keep-san 1

If I uncomment skip-san-boot then it will boot from the local CD-ROM.  You’ll need to recomment it if you want it to boot from the SAN.  Magically, the Windows 7 install DVD picked up the SAN device during the install, no hacked WinPE required.

And that’s it.  Booting from the UNDIONLY.KPXE works where iPXE.KRN does not.

And that will get you a iSCSI booting disk.  I suspect I can change PXELinux to boot the UNDIONLY.KPXE.  I prefer to use PXELinux because if I want to burn a bootable CD/DVD or USB stick I can just transfer my entire PXELinux folder structure over for SYSLinux or ISOLinux as a boot loader.

Next I’m going to try differencing disks.  The goal for this little project is to see if I can setup a render farm by iSCSI booting computers and have them join automatically.  This way if you need more power added to your farm you can just add any hardware, boot it up via PXE and it will launch into an environment where everything is pre-configured and ready to go.


I’ve been able to get pxelinux to boot iPXE.KRN.  The trick is the ipxe launch file needs the net0 to have the keep-san set.  Setting it globally didn’t appear to make it set in net0 (which, I assume, becomes stored in the iBFT)

This is my ipxe.ipxe that works with iPXE.KRN after been chainloaded by pxelinux.0:

sanhook ${root-path}
set net0/skip-san-boot 1
set net0/keep-san 1
prompt –key 0x02 –timeout 4000 Press Ctrl-B for the iPXE command line… && shell ||

I set the skip-san-boot so I could boot from a regular Windows CD.  You’ll want to remove that line to actually boot from the SAN.

Keep in mind there is a gateway issue with MS’s iSCSI implementation.  It will add a static route to the SAN  IP so you will either need to clear the gateway via the iPXE script or have the software target on the same location as your router (or have your router route local LAN traffic).

Read More