Blog

Citrix PVS 7.7 VHD to VHDX performance difference

2016-06-13
/ /
in Blog
/

I was asked to justify upgrading our PVS vDisks to VHDX from VHD.  There are a few ‘feature’ / technical reasons:

  1. Use of native tools to mount/compress VHDX from Windows Server 2012.  VHD files created with 16MB block sizes require a custom Citrix tool which does not do compress.
  2. VHDX is the format ‘forward’.
  3. VHDX is supposed to perform better.

Test setup: VHD file is Citrix PVS 7.1SP2, VHDX file is a clone of the VHD with the tools upgraded to 7.7.

So I know VHDX is supposed to perform better but I was curious by how much.  Apparently, the modification Citrix made to the VHD format to a 16MB block size is ‘4K’ aligned as well.

fsutil fsinfo ntfsinfo C: reports the following for the different vDisks:

VHD_fsinfo

16MB block size VHD file

VHDX_fsinfo

32MB block size VHDX file

I set my target devices to the different vDisks and set the ‘Cache to RAM’ feature to 4096MB.  Ideally, all writes should be to RAM but this will still tax the filesystem.

And what is the performance between the two?  I used the DiskSpd utility from Microsoft to measure the differences.

VHD DiskSpd Test

VHD DiskSpd Test

VHDX DiskSpd Test

VHDX DiskSpd Test

Summary

Summary

The VHDX format appears to be around 7.5% faster in our setup.

The boot speed (the amount of time it takes the vDisk to power up and start the ‘Citrix PVS Device Service’) is even more dramatic:

VHD Boot Speed

VHD Boot Speed

 

VHDX Boot Speed

VHDX Boot Speed

How much of this is the tools vs the format?  I’m not sure, I didn’t have the time to reverse image and upgrade a VHD to 7.7.  Regardless, the combination of upgrading to 7.7 from 7.1SP2 AND the VHDX format brought a dramatic boot time improvement and consistently faster disk speed.

Read More

EventID 6003 – The winlogon notification subscriber was unavailable to handle a critical notification event.

2016-05-31
/ / /

We updated one of our Citrix XenApp servers and this message started flooding our Application event log:
“The winlogon notification subscriber <TrustedInstaller> was unavailable to handle a critical notification event.”

So what’s going on here?

Examining the registry on a ‘good’ working system and the ‘bad’ system revealed the following:

2

Good TrustedInstaller – No Error 6003

 

1

Bad TrustedInstaller – Error 6003

How did that value get there?

It turns out we installed Internet Explorer 11 with our patch cycle — but that in and of itself did not cause our issue.  Additional components for IE 11 were installed as well:
3

“Microsoft Windows English Spelling Package” and “Microsoft Windows English Hyphenation Package”

Neither of these packages are present on any of the ‘working’ systems.  I tested to determine which of them placed the registry key there…

It turns out both of them do.  If you uninstall *either* package it will remove the ‘CreateSession’ value.  Since these packages are not in our standard build we are removing both.

Read More

Citrix Provisioning Services – Convert your VHD’s to VHDX

2016-05-13
/ /
in Blog
/

Per my previous post, we have been upgrading our storage backend and one of the features is 4K sector sizes being used.  This is an issue as VHD files will not mount when they reside on 4K storage.  Since we want this feature our two choices are to downgrade our storage to 512B sector sizes or move to the VHDX format.  I would prefer to move to the VHDX format as it has a small performance advantage (~7.5% was what I measured) and future proofs us a little bit for when we upgrade our PVS servers to 2012.

The VHDX supports 512e on 4K storage which is only supported on the following operating systems:

  • Windows Vista
  • Windows 7
  • Windows Server 2008*
  • Windows Server 2008 R2*
  • Windows Server 2012
  • Windows 8

So Windows Server 2003 is out, we are going to have to deal with not being able to mount their vDisks.  For our Windows Server 2008 R2 servers we can convert our VHD files to VHDX.  This is the process I have developed.  Because we are going to be mounting the VHD and VHDX files you will need your files to reside on 512B storage.  Without further ado, here are the steps to convert your VHD to VHDX:

  1. Create your VHDX file.  Ensure to select 512B if you are migrating an operating system older than Server 2012.
    23
  2. Mount your VHD file and your VHDX file.  Format and set the partition as active for the VHDX if needed.
    16
  3. Use a clone utility (DriveImage XML, Windows Server Backup or HDDRawCopy) to clone your VHD to the VHDX via the mounted paths.
    1718 1920
  4. Take this opportunity to defrag your cloned VHDX.
    21
  5. Assign your vDisk to your target device.
    22
  6. Done!

 

Read More

Citrix Provisioning Services 7.7 – Unable to mount VHD files

2016-05-13
/ /
in Blog
/

We recently upgraded our storage where our PVS vDisk’s reside.  I was attempting to inject a file using ‘Mount vDisk’ from the PVS Console and I got an error message:

1

The text of the message states:
“When accessing a new tape of a multivolume partition, the current block size is incorrect”.

Google searching this message does not bring any helpful results.

I then tried to mount the vDisk via the command line:

2

Since we still had our ‘older’ storage I tried mounting the same vDisk.  It worked.  So something with our new storage was preventing us from mounting our VHD’s.  We could still create new versions and boot our target devices off the new versions without issue, but mounting the vDisk’s to add a file or change a single registry key is so much more efficient and faster.  We have 5 different datacenters with different storage at each, with 1 of the datacenters about to undergo another storage migration (so 6 different storage ‘filers’ in total).  I attempted to mount a vDisk from each:

3

A pattern immediately emerged on the far right of this procmon trace.  The ‘filers’ that allowed us to mount the VHD file had the ‘QuerySizeInformationValue’ report back a ‘BytesPerSector’ of 512.  The filers that failed had a BytesPerSector of 4,096.  Immediately, it became obvious that the new storage had a 4K format and the older storage had 512B format.  Microsoft has a statement about VHD files and 4K support:

Effect when hosting VHDs on native 4K disks

The VHD driver today assumes that the size of the physical sector of the disk is 512 bytes and issues 512-bytes IOs. This makes the VHD driver incompatible with these disks. Therefore, the VHD stack does not open the VHD files on physical 4 KB sector disks.

Well, there you go.  You cannot mount VHD files when they reside on 4K storage.  Fortunately, PVS 7.7 supports VHDX files with 512e support with Windows 2008R2 so we can move our VHD’s to VHDX’s.  I will document how to do so in my next post.

Read More

Multi-homed server, lock ICA and RDP to a specific NIC

2016-04-21
/ /
in Blog
/

We implement a multi-homed setup with Citrix Provisioning Services.  We have all of our production traffic on one NIC and all PVS traffic on a second nic.  This helps us in troubleshooting when doing packet captures but does introduce other sets of challenges.  One of these challenges is when I uninstalled and then installed updated VMWare tools on one of our vDisks it caused the NIC’s to renumber and reorder themselves.  You’ve probably read some articles saying to ‘show hidden devices’ and uninstall any ‘ghost’ devices; with a multi-homed setup this may not resolve your issue.  My specific issue is my NIC’s now went from “0x1 and 0x2” to “0x3 and 0x4” in the LANATABLE.  We apply a GPO to the ICA-TCP and RDP-TCP to force them to only utilize our ‘Production’ NIC which we decided was going to be the second NIC.

Provision = 1st Nic, Production = 2nd Nic

I uninstalled the ghost devices but because this change is all ‘in the registry’ it wasn’t immediately noticeable by myself that the LANATABLE and NIC ordering had changed.  I promoted my vDisk and then tried to RDP into it:

Well…  I knew this wasn’t right.  So I logged onto the server and checked the LANATABLE values:

Provision NIC
Production NIC

 

Provision NIC LanaID = 0x3 after the VMWare Tools upgrade

 

Production NIC LanaID = 0x4 after the VMWare Tools upgrade

 

RDP targets LanAdapter 0x1 (!?)

 

ICA targets LanAdapter 0x2

A couple issues popped out.   The first was that the LanaID’s were wrong.  I *thought* Provision should be #1 and Production should be #2, but we apply our LanAdapter ID’s via GPP so these values are correct for our other systems.  I know both RDP and ICA need to be locked to the Production NIC and the order is *correct* but I’m a bit confused to why the numbers are different between RDP-TCP and ICA-TCP.

So I started on getting the issues resolved.  First, I was going to resolve RDP.  If it is targeting 0x1 that means that I need the Production NIC (VMWare Network Adapter #2) needs to be set as 0x1.  So I edit the LanaId of the Production NIC to 0x1 and the Provision NIC to 0x2.  I rebooted the box and I could RDP into it without issue.  I then checked the Remote Desktop Session Host Configuration:

This is the correct value.

OK, so RDP is set correctly and works.  I then tried to launch a Citrix application.

It failed.

It generated an event on the server:

Unable to connect to the CGP tunnel destination

Looking at the ICA Listener configuration showed me the following:

The value is wrong.  It should be #2

So the ICA-TCP listener was set to the ‘Provision’ NIC.  So our production traffic was not getting to it. This is the wrong value.  My first thought was the LANATABLE would make sense here…  We have the LANADAPTER key is set to 0x2 which would equal the ‘Provision’ NIC under this configuraiton…   So I changed the LANATABLE to be the reverse.  0x1 = Provision NIC and 0x2 = Production NIC.

The results:

Now both are wrong!

Jeez.
I reverted the LANATABLE to 0x1 = Production and 0x2 = Provision.  Again, only the RDP-TCP connection changed.  At this point the ICA Listener *must* be looking at another place…  I used Procmon to trace the registry when I opened the ICA Listener Configuration and noticed it did NOT query LANATABLE but did go through and look and query this key:
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesTcpipLinkage
The BIND order is set as Production (#2 NIC) first and Provision second in this order…  To change this order you need to go to Advanced Settings and modify the connection order:
Provision *should* be the top NIC here…
So I used the arrows on the side and moved Production to be second in this order:
Rebooted and I checked the ICA Listener:
Hurray!
I then tried both RDP and ICA traffic and both now worked correctly.
So, the lesson learned here is that this registry key:
Targets the BINDING order of the NIC’s.
And this registry key:

Targets the LANATABLE and the values specified within.

Read More

Automatically Defrag your Citrix PVS vDisks

2016-04-19
/ /
in Blog
/

Here is a script to automate ‘offline defragging’ your PVS vDisks.  Why Defrag?  It’s explained nicely here.

Effects of Fragmentation on Write Cache Size
The culprit above in excessive write cache size is fragmentation. This is caused because PVS redirection is block based and not file based. The redirection to the write cache is based on the logical block level structure of the vDisk. To illustrate this, consider the scenario below.

And the script:

 

Read More

Silent Install of Citrix PVS Target Device software

2016-04-12
/ /
in Blog
/

Installing the Citrix PVS Target Device software via a silent, non-interactive method fails.  Various solutions involve copying files to various locations after the install but I’m curious what it’s doing when it fails.  I started to automate one of my previous posts and got to the stage where I needed to silently install the target device software and my install was hanging.  It was hanging because I was getting prompted to run a custom action for the MSI install.

The logging, when we are ‘stuck’ is at this point:

Logging into the server and we get the notification that a something requires an action ‘Interactive Services Detection’

 

If you run the install in a interactive user session the custom action appears to be this message:

For some reason, Citrix found it necessary to include a dialog in the installer that forces you to answer it.  When this dialog is presented in a ‘silent’ install it requires interactivity to continue so the install stops until it is dealt with.

So, why is this dialog appearing?  It is appearing because CFSDEP2 service was not uninstalled cleanly and the installer requires it removed for a clean install.

When uninstalling the software silently you also get a ‘Interactive Services Detection’ dialog.  For whatever reason though, the uninstall dialog doesn’t halt the process.

What is triggering the ‘interactive services detection’ on uninstall and is it the cause of our CFSDEP2 not being removed?

When doing an interactive uninstall this is a screenshot of all the actions that it executes:

And you can very clearly see it deletes the CFSDEP2 file system filter driver.

What does it look like when run silently?

 

The interactive services detected prompts on uninstall (UI0Detect.exe) appears to be caused by ‘runonce.exe’ and the ‘grpconv.exe’ programs.

Is our CFSDEP2 service deleted when run silently?

It was not deleted.  Is the runonce.exe utility the cause of our CFSDEP2 service not being removed?

It is not.  The CFSDEP2 service is removed *before* the runonce.exe utility it executed.  So something else is triggering it’s removal.

Examining the cfsdep2.sys file in the C:Windowssystem32drivers reveals that the driver was not uninstalled with the Citrix PVS Target Device software.

It turns out that file system filter drivers can be installed and uninstalled using the command line.
Uninstall:

Install:

So, we should just be able to add a ‘Invoke-command’ and execute it, right?
I tried installing and uninstalling with both WMI and Invoke-Command:

Neither command worked remotely.  I could see rundll32.exe executing and exiting with status “0” which implies success.  But the commands themselves didn’t *actually* work.  When I executed the uninstall remotely neither the CFSDEP2.SYS file was deleted nor was the service uninstalled.  Doing a procmon.exe I could see that the supplemental ‘runonce.exe’ and ‘grpconv.exe’ were not run.  The reverse was true for the install as well, the CFSDEP2.SYS was not present and the service was not installed, but the exit code was “0”.  So we can’t trust the exit code, we need to manually check to see if the files and service are present.

So what’s happening?  It turns out, for some reason, that the file system filter driver install/uninstall *requires* an interactive session to complete successfully.  What is an interactive session you could use?  The SYSTEM account.  I was hoping to create a purely native script with no outside dependencies but PSEXEC will be required.  Elevating permissions in a remote powershell session is very difficult and maybe one day I’ll spend some time figuring it out and documenting it, but at this point I cheaped out and decided to use psexec.exe.  To get the install/uninstall to execute remotely, you can use psexec.exe with the following lines:

Or, since we are now running under the interactive SYSTEM context, we can just use the uninstall command:

And the install executes successfully as well under the interactive SYSTEM context:
 

Read More

ConvertFrom-MCLI for PVS 7.6 POSH module

2016-04-06
/ /
in Blog
/

Martin Zugec made a POSH script to convert output from mcli.exe to a PowerShell object, but his script fails with the new PVS 7.6 POSH module.  From looking at his script he is trying to key on ‘spaces’ to set the properties of the objects.  MCLI.exe outputs the properties with spaces.  PVS 7.6 POSH module fails because it doesn’t format it’s output as spaces.

MCLI.EXE output:

POSH module output:

Notice the subtle difference? No spaces.

To get Martin’s original script working we need to change the script to look for another character we can key on. Fortunately(?) it *appears* that the PVS POSH module outputs properties to start with a lower case letter. If we modify his script here:

To this:

It now outputs properties correctly from the PVS POSH module:

The full modified script is here:

 

Read More

Citrix Provisioning Services – Updating VMWare Tools and Target Device software — with all native tools

2016-04-05
/ /
in Blog
/
  1. Prerequistes

Step
Detail
2 servers:
  1. Staging server with Windows 2008 R2 SP1
  2. VMWare machine configured identically to your PVS target devices (BLD server)
‘Windows Server Backup’ feature installed on the staging server
The Citrix utility ‘CVHDMOUNT.EXE’ is installed on the staging server
‘Backup’ space (approx. 100GB)
VMWare VMDK hard disk of a greater size than the PVS disk:
<

 

Copy and install required drivers on staging server

 

Step
Detail
From a Windows Server, browse to a server with Citrix Provisioning Services installed on it.

 

Copy the drivers folder and CVhdMount.exe to the server

 

Open the drivers folder, right-click on cfsdep2.inf and select “Install”

 

Open Device Manager, right-click the computer name and choose “Add Legacy Hardware…”
Select “Next”
Select “Install the hardware that I manually select from a list (Advanced)” and click “Next”
Click “Next”
Click “Have Disk…”
Click “Browse” and go to the “drivers” folder and select “cvhdbusp6.inf” and click “Open”
Click “OK”
Ensure “Citrix Virtual Hard Disk Enumerator PVS” is listed and click “Next”
Click “Finish”

 

Setup VMWare virtual disk

Step
Detail
Open the “VMWare vSphere Client”

 

Right-click on the server you want to do the cloning on and click ‘Edit Settings…’
Under the ‘Hardware’ tab, click ‘Add…’
Select ‘Hard Disk’ and click ‘Next’
Select ‘Create a new virtual disk’ and click ‘Next’

 

Set the ‘Disk Size’ to be greater than the size of the VHD file, select the ‘Disk Provisioning’ options you require, select the ‘Location’ you want to store the disk and remember where it is stored.  You will need this location soon.  Click ‘Next’
Click ‘Next’
Click ‘OK’
Format the disk and set it as active

 

  1. Clone VHD to VMDK

      1. Backup Citrix vDisk

Step
Detail
RDP into the staging server and mount the VHD file you want to update:
Cvhdmount.exe –p 1 \serversharevDisks-XenAppXenApp65Tn01.13.avhd

 

Open Disk Management and confirm your Citrix VHD is mounted and the new VMWare disk is present

 

Open ‘Windows Server Backup’
Click ‘Backup Once…’
Select ‘Different options’ then ‘Next’
Select ‘Custom’ than ‘Next’
Select ‘Add Items’
Select the PVS disk and click ‘OK’
Click ‘Next’
Select ‘Local drives’ and click ‘Next’
Select the ‘Backup Destination’ and click ‘Next’
Click ‘Backup’
Wait for the backup to complete
Click ‘Close’

 

      1. Recover backup to VMDK

Step
Detail
Click ‘Recover’
Select ‘This server’ and click ‘Next’
Click ‘Next’
Select ‘’Volumes’ and click ‘Next’
Select the checkbox beside the volume and choose the ‘VMDK’ for the destination volume and click ‘Next’
Click ‘Yes’
Click ‘Recover’
Wait for the Recovery to finish
Click ‘Close’

 

      1. Fix BCD file for VMDK

Step
Detail
Unmount the Citrix vDisk.  Cvhdmount -U 0
In the command prompt, switch to the ‘Destination’ drive and check the BCD file:

 

Notice there are 3 entries that need to be corrected.
Execute the following commands, substituting the ‘E:’ for the proper drive letter:

 

bcdedit /store bcd /set {bootmgr} device partition=E:
bcdedit /store bcd /set {default} device partition=E:
bcdedit /store bcd /set {default} osdevice partition=E:

 

Confirm the BCD file now contains the correct entries:

 

      1. Configure BLD Virtual Machine and attach the VMDK

Step
Detail
Open the vCenter console, select the staging server and ‘Right-click’ and select ‘Edit Settings…’
Select the VMDK file, note the path of the Disk File and click ‘Remove’
Under ‘Removal Options’ select ‘Remove form virtual machine’ and click ‘OK’
Select the associated BLD server of this vDisk and right-click and select ‘Edit Settings…’  In this example, the vDisk I am modifying is XenApp65Tn01 which is associated with BLD server WSCTXBLD351T
Click ‘Add…’
Select ‘Hard Disk’ and click ‘Next’
Select ‘Use an existing virtual disk’ and click ‘Next’
Select ‘Browse’
Navigate to the path noted earlier, select the disk and click ‘Open’
Click ‘Next’
Click ‘Next’
Click ‘Finish’
Click ‘OK’
      1. Disable CDROM attachment on bootup

Step
Detail
Select the associated BLD server of this vDisk and right-click and select ‘Edit Settings…’  In this example, the vDisk I am modifying is XenApp65Tn01 which is associated with BLD server WSCTXBLD351T
Select the ‘CD/DVD drive 1’ and ‘Uncheck’ the ‘Connect at power on’ and click OK

 

Start the VM and uninstall the target device software

Step
Detail
Right-click on the VM and select “Power > Power On”
Right-click on the VM and select “Open Console”
Login to the VM once it boots
Click “Start”
Click “Control Panel”
Click “Program and Features”
Click on the “Citrix Provisioning Services Target Device x64”
Right-click and choose “Uninstall”
Click “Yes”
Click “OK”
Wait for the uninstall to complete then restart the computer

 

Upgrade VMWare Tools

Step
Detail
Login to the VM once it boots
Browse to the VMWare Tools install and open ‘setup64.exe’

 

Select ‘Next’
Select ‘Custom’ and click ‘Next’
Ensure the ‘NSX’ options are set to ‘Entire feature will be unavailable’ and click ‘Next’
Select ‘Close the applications and attempt to restart them’ and click ‘OK’
Click ‘Finish’
Click ‘Yes’ to restart

 

Install new Citrix Provisioning Services Target Device software

 

Step
Detail
Login to the VM once it boots
Browse to the share that holds the updated software and open ‘PVS_Device_x64.exe’
Click ‘Install’
Click “Next”
Select ‘Acknowledged’ and click ‘Next’
Choose “I accept the terms in the license agreement” and click “Next”
Click “Next”
Click “Next”
Select ‘Complete’ and click ‘Next’
Click “Install”
Uncheck “Launch Imaging Wizard” and click “Finish”
Click “Yes” To restart the computer.
      1. Remove VMWare Disk from BLD server

Step
Detail
Right-click the VM and select ‘Edit Settings…’
Select the VMDK disk, note the ‘Disk File’ path and click ‘Remove’
Ensure ‘Remove from virtual machine’ is selected and click ‘OK’
Select the CD/DVD Drive and check the ‘Connect at power on’ box and click ‘OK’

 

  1. Clone VMDK to VHD

      1. Backup VMWare Disk

Step
Detail
Right-click on the staging server and select ‘Edit Settings…’
Select ‘Add…’
Select ‘Hard Disk’ and click ‘Next’
Select ‘Use an existing virtual disk’ and click ‘Next’
Click ‘Browse’ and select the disk you noted earlier
Click OK
Click ‘Next’
Click ‘Next’
Click ‘Finish’
Click ‘OK’
RDP into the staging server, browse the backup drive and delete the contents of ‘WindowsImageBackup’

 

Open Disk Management and confirm your VMDK is mounted

 

Right-click on the VMDK and select ‘Shrink Volume…’
Enter a number to shrink the partition so it is *smaller* then your Citrix VHD disk size

 

NOTE If you do NOT shrink the partition you will be unable to restore the partition to the smaller Citrix VHD file.
Confirm the shrink worked successfully
Open ‘Windows Server Backup’
Click ‘Backup Once…’
Select ‘Different options’ then ‘Next’
Select ‘Custom’ than ‘Next’
Select ‘Add Items’
Select the VMDK disk and click ‘OK’
Click ‘Next’
Select ‘Local drives’ and click ‘Next’
Select the ‘Backup Destination’ and click ‘Next’
Click ‘Backup’
Wait for the backup to complete
Click ‘Close’
Go to the vCenter console and Right-click on the staging server and select ‘Edit Settings…’
Select the VMDK used for updating VMWare Tools/Target Device software ‘’ and click ‘Remove’
Select ‘Remove from virtual machine and delete files from disk’

 

      1. Recover backup to VMDK

Step
Detail
RDP into the staging server and mount the VHD file you want to update:
Cvhdmount.exe –p 1 \serversharevDisks-XenAppXenApp65Tn01.13.avhd

 

Open ‘Windows Server Backup’
Click ‘Recover’
Select ‘This server’ and click ‘Next’
Click ‘Next’
Select ‘’Volumes’ and click ‘Next’
Select the checkbox beside the volume and choose the ‘Citrix vDisk’ for the destination volume and click ‘Next’
Click ‘Yes’
Click ‘Recover’
Wait for the Recovery to finish
Click ‘Close’

 

      1. Fix BCD file for PVS vDisk

Step
Detail
In the command prompt, switch to the ‘Destination’ drive and check the BCD file:

 

Notice there are 3 entries that need to be corrected.
Execute the following commands, substituting the ‘E:’ for the proper drive letter:

 

bcdedit /store bcd /set {bootmgr} device partition=E:
bcdedit /store bcd /set {default} device partition=E:
bcdedit /store bcd /set {default} osdevice partition=E:

 

Confirm the BCD file now contains the correct entries:

This process could be scripted to make it less manual, faster, and less error prone, but because of the frequency we actually do these type of updates, I have just created a manual document for now.

Read More

AppV5 – Package {GUID} version {GUID} failed configuration in folder ‘%packageinstallationroot% with error 0x4C40310C-0x12

2016-03-30
/ /
in Blog
/

We experienced this error on a package on one of our Citrix servers in the AppVClientAdmin event logs.  Attempting to procmon this error didn’t reveal anything substantial for what could be causing this issue.  I then tried my last post to enable AppV debug logs to see if we can see what’s going on.

Because this server was being used by other users the log generated a lot of noise.  I stopped the log as soon as I got the error message though, which meant the error should be at the end of the generated log.  Fortunately, you can search for the error message:

[2]06B0.2DE4::‎2016‎-‎03‎-‎30 09:41:30.817 [Microsoft-AppV-Client]Package {499ed340-c809-47dc-a533-2cdeab537e93} version {3589c28b-edb7-41d6-865d-e01c4fdd4318} failed configuration in folder ‘D:AppVDataPackageInstallationRoot’ with error 0x4C40310C-0x12. 

I suspect the first bit of hexadecimal code (06B0.2DE4) are probably an identifier for a thread or some such so I suspect if I search for just this code I can be shown all events that lead up to this error:

Looks like it.

The log up to the error:

With the error time code, I can compare what Procmon says is going on.  And procmon reports back the following (at exactly 09:41:30.815):

Procmon has the AppVClient.exe doing the following action “QuerySecurityFile” and it’s looking for “Information:Owner”.  So what is this the value of this property?

Hmmm…  This does not look correct to me.  From my experience, I think the current owner ends up being ‘SYSTEM’ because that’s the account the AppVClient.exe runs under when it creates this folder.  As a hunch, I looked at a system that is operating correctly and compared it’s Owner attribute:

Sure enough, on a working system the Owner is SYSTEM.  So I’ll attempt to modify the non-working system and retry adding the package:

Success!  So it appears this error “0x4C40310C-0x12″ is DIRECTLY related to the ownership attribute on your PackageInstallationRoot folder.  If it is anything but SYSTEM it looks like it fails.  I do not know why that attribute changed, but changing it back to SYSTEM resolves this error code.

Read More