Citrix Provisioning Server – PXE requests stop working

/ /
in Blog

We use a bootable ISO in our environment to boot our VM’s to a specific set of PVS servers.  This ISO will vary by region enforcing that each target device that boots will be directed to their closest PVS server.

However, we have 1 region that does not leverage this capability and this region was designed to utilize the PXE services of the Citrix PVS server’s.  Occasionally, we encountered VM’s that will not boot and instead the console shows “PXE-E53: no boot filename received”


When I logged onto the Citrix PVS servers, I checked their services.  Both services were reported as “Running”:

When I checked the event logs I did not see any errors in either the application log or the system log.  Administrative events showed nothing out the usual either.

In order to confirm that the PVS service was actually listening, I executed

this showed me all the open ports the server was listening for and the processes tied to those ports.  Since PXE is a UDP operation, I examined the UDP portion of the netstat output.  

Port 69 is used by TFTP to transfer files, and port 67 is used by PXE.  However, I only saw port 69, port 67 was no where to be found.  I restarted the “Citrix PVS PXE Service”, reran netstat and confirmed that the PXE port was not listening and matched up the process ID to the proper services.

Restarting the failed target devices and they began to boot properly.

However, why did this fail in the first place?  I read on the Citrix forums that the Citrix services can become unbound if the network is not available when the services are started.  To test this I rebooted one the affected Citrix PVS servers.  Sure enough, it came back up with port 67 not being monitored but the service in a ‘Running’ state.  I wanted to see if I could capture the flow of communication from the network and when the service started so I used procmon and enabled “Boot Logging”.

Lo and behold, the procmon monitoring on startup added enough of a delay that the PXE service was bound consistently.  Stopping the boot logging and the PXE service would start but fail to bind to the port.

So now this leads to a bit of a quandary.  The delay seems to be in the milliseconds.  I’ve considered a couple solutions for this issue.

  1. A startup script that checks to ensure both ports and restarts the proper service if one of the ports is not found.
  2. Change the service startup type to be “Automatic (Delayed Start)”.  This delays the service by up to 2 minutes.  This does mean that the PVS server will NOT be able to service target device boot requests during this window.

I think we’re going to go with option 2.  The reason is we can apply this setting change via Group Policy Preferences.  This ensures that if we any removal/upgrade of the PVS software this setting will get reapplied.  And then we don’t have to worry about upgrading the OS and losing the startup script either or maintaining a script.

We’ve been affected by this a few times in the past, the fix has always been to restart the PVS server, but I managed to hit a window where the failure was happening consistently and managed to get this information.  🙂


Read More

Citrix Provisioning Services Reverse Imaging – 2017 edition

/ /
in Blog

We’ve been trying to upgrade our Citrix PVS tools to version 7.13 and had some issues.  I need to reverse image so I can remove the software without dealing with the in-place upgrade procedure. So I decided this would be a good time to update my reverse imaging process.  This time I will only be using native tools built into the OS.  With that, this is my new procedure for reverse imaging.  It’s much faster and easier than my last process, but it does have a dependency.  You must be running Windows 2012 or greater so that dism.exe is a part of the OS.  This reverse imaging process is VMWare specific but the principals should apply to all hypervisors.  The overview of the process is as follows:
Create a local hard drive on a ‘build’ server that your vDisk will be based off of.
Attach that disk to the PVS server
Image vDisk to local disk
Boot local disk, manipulate as needed
Image local disk back

  1. Create a new hard disk on your ‘build’ VM that we will use as the staging for the vDisk you want to reimage
  2. Attach the newly created disk to your PVS server
  3. Setup the disks on the PVS server.  In my example, the G: drive is a mounted VHDX of the Citrix vDisk.  I’m using DISM to capture an image file of this disk to a wim that I will apply to the local disk created in step 1.  You will need to create a partition on the disk you made in step 1 and set it as active (if this is a MBR disk).  Unfortunately, DISM does not do disk to disk imaging so this intermediary step is required.  In my video here, the E: drive is the local disk that belongs to my target ‘build’ server.  In addition, you must fix the BCD store as it will point to the original partition, not your new target partition.
    The BCD commands are:

    You will need to replace “partition=E:” with your drive letter of the LOCAL disk.

  4. Remove your hard disk from the PVS virtual machine without deleting it.  Ensure the disk is attached to your BUILD target device.  If you are booting up your BUILD target device from an ISO or PXE you may need to disable those features so that it boots from the hard disk.  I also disconnect the network from the virtual machine because we will need to reset the computer account with AD.
  5. To reset the computer account with AD, you need to login to the system.  The easiest method is if you know the local administrator password, log in with that and “rejoin” the domain.  If not, disconnect the network then logon with an account that has recently logged onto the server.  Cached credentials should be able to pass you through.  Once you are in, connect the network and rejoin the domain.  You can typically “rejoin” by changing the domain name to the NETBIOS name, or the reverse if the NETBIOS name is present

    At this point you can do whatever work is required.

  6. Once you are done with whatever work you need to do, set the target device to boot from hard disk, but keep your vDisk set as an option.  This will attach the vDisk on boot as a secondary drive.  This will allow us to image back to it.  You need to ensure this vDisk is in PRIVATE mode OR has a Maintenance version.  This will set the vDisk to a Read/Write mode.

    If you have your WRITE drive attached, you will need to ensure your local system disk is the LAST disk in the order for your VM.  The PVS boot loader, when set to boot from hard disk, seems to try and boot the LAST disk it detects.

    Checking the PVS Status Tray shows you’ve booted from a local hard drive.

    Something you will need to check and validate is that the Hard Disk of the Citrix vDisk is NOT the drive letter of your write cache (if you find you cannot attach a write cache).  The reason is you have probably redirected the page file, event logs or whatever and if the Citrix vDisk occupies that drive letter you will not be able to image to it because it will be in use.
  7. To image back open the Imaging Wizard:

    And go through the imaging process.

    I prefer to image to the vDisk volume (as seen in this process).

  8. Done!  Set the device to boot via the vDisk as opposed to Hard Disk, delete the local OS disk, boot up your vDisk and run any scripts to ‘finalize’ your image.
Read More

Citrix Storefront – Adventures in customization – Change any ICA parameter

/ /
in Blog

When I was originally exploring using the Citrix SDK to manipulate some ICA parameters, I found I couldn’t do it.  Using Powershell and passing a header parameter I could manipulate the ICA file to how I wanted, but when I attempted to do so with Internet Explorer, I found it wasn’t passing the header parameter.  I assumed this was because of how browsers were launching the ICA file.  However, I attempted to use Sam Jacob’s Citrix Storefront clientname manipulation extension, I discovered I had the same problem.  I originally thought I was doing something wrong, but on a whim I tried Chrome and it worked…  Perfectly!

When I investigated further I found it appears that when Storefront detects IE it doesn’t use the GetLaunchStatus API, which passes the headers to Storefront for ICA manipulation.  I found a way to fix this was the add the IE detection to the script.js and add the missing call to the GetLaunchStatus API.

But now that I have headers being passed properly, I thought I should revisit my ICA parameter manipulation.  Sam’s customization works great for manipulating the client name, but we have a few applications with various different requirements:

  1. Pass parameters to some specific applications
  2. Modify clientname based on the specific application that is launched

Can we replace Sam’s customization with one that can manipulate any ICA parameter we so choose?

Yes.  Yes we can.  Storefront is very nice, flexible and powerful 🙂

I’ve designed this customization to look for an additional app setting tag.  This needs to be added to the web.config at the “”C:\inetpub\wwwroot\Citrix\Store\web.config”” level:

Any parameter you want passed through needs to be enabled as a forwarded header in the “”C:\inetpub\wwwroot\Citrix\StoreWeb\web.config”” file:


Lastly, you need to edit your \custom\script.js file to include your application and whatever parameters you want passed to it:

One of the cool things about this is you can still use tags to change the behaviour of these applications.  For instance, if you want a certain subset of your applications to be 8bit you can do set check for that tag on app launch and add the header to set the desired color.  You can set certain applications to keep their ICA file by setting “RemoveICAFile” to Off.  You can individually modify any setting for a specific application or subset of applications.

Here is the StoreCustomization_Launch.dll and source code.

Here is the source code for prosperity:



Read More

Citrix Storefront – Adventures in customization – Customization breaks in internet explorer

/ /
in Blog

I ran into a frustrating issue with my last post (changing the clientname based on the application).  The issue was, it wasn’t working!  Sometimes!  On some versions of IE!  On some Operating Systems!


On Windows 10, IE11 or Edge it worked fine.  On Windows 7 or Server 2008 R2 with IE11 it worked…  SOMETIMES.

I started to dig into what was happening.  Why wasn’t the ClientName being set consistently?

It turns out that the code path for the “StoreFront Customization SDK” (at least the ICA file modification portion) operates when a request is made to “GetLaunchStatus”.  A working application launch from StoreFront looks like this:

Notice when I click the application icon, TWO requests are made.  The first one, GetLaunchStatus contains the headers that is passed to the SDK customization plugin, the second is a hidden iFrame to pull the ICA file.

Now, if I login to a server or PC that I’ve never logged into before (so that I have a fresh profile) and launch IE and immediately go to Storefront, this is the result:

Notice it only goes to “LaunchIca”.  Because of this my ClientName is not modified.

To validate the theory a bit further that something in Storefront maybe causing my issue, I opened up Chrome and changed it’s UserAgent to be identical of IE11 on Server 2008 R2:

And what happened?

Only LaunchIca is called.  We have a reproducible failure.

As I was digging into the Citrix code to launch the “GetLaunchStatus” URL I found it would “skip” going to URL under a series of conditions, one of them relates to checking for the presence of IE (CTXS.Device.isIE).  I found two candidate functions that maybe causing this failure, and I think this one is the most likely:

Specifically, this line under the switch statement:

CTXS.Device.isIE() && CTXS.ClientManager.usingNativeClientInInternetZone() ? (g(a, CTXS.LAUNCH_SUCCESS), g(a, CTXS.LAUNCH_READY), a.isLaunchReady = !0) : (CTXS.ClientManager.getLaunchMethod() == CTXS.LaunchMethod.PROTOCOL_HANDLER && (e.ticket = f.fileFetchTicket, e.staTicket = f.fileFetchStaTicket, e.fileFetchUrl = f.fileFetchUrl, e.serverProtocolVersion = f.serverProtocolVersion), d(a, c, e));

I found if I modified the UserAgent at that exact point so CTXS.Device.isIE() did not return TRUE that it would execute the GetLaunchStatus.

Now that I suspect I have my culprit I examined the code that calls GetLaunchStatus URL and added it to my custom.js file:

The end result now looks like this:

And now my customizations are called correctly and work with IE11.  I haven’t done extensive testing so “your mileage may vary” with this fix, but if you are finding your customizations aren’t operating correctly you may want to try this fix.


Read More

Citrix Storefront – Adventures in customization – Assign a custom clientname to an application

/ /
in Blog

I have set a personal goal to avoid creating more than a single store for Storefront.  With that, I’ve been working on customizing Storefront for our organization and occasionally a problem comes up that requires a unwieldy solution…  Like creating multiple stores.

An example of this is we have an application that configures itself based on the CLIENTNAME variable.  The current solution for  this problem is to provide ICA files with CLIENTNAME preconfigured.  The application is launched, detects the client name variable then displays the relevant data.

Really, this is a pretty terrible way for the app to configure itself.

You would hope the application would have been coded to accept a switch or something for configuring it and then just publish out the application(s) instead.

But, I’m not the developer of this application and have to come up with a solution that works within this framework.  The initial thought was we can enable ‘overrideicaclientname’ and then modify the default.ica files with the hard coded client name.  Then we’d create as many stores as is required to support the different configurations…  Then direct users to the different URL based on their needed configuration…

If this is starting to sound ugly, it’s because it is.

Citrix provides some guidance for manipulating the CLIENTNAME.  Unfortunately, the manipulation of the CLIENTNAME in the example would not be sufficient.  We need to configure it to be a specific string and all the examples within are dynamically generated.

Fortunately, Citrix CTP Sam Jacobs has come to the rescue and extended the work of Simon Frost. Sam’s addition was to accept a custom HEADER and configure the CLIENTNAME on that value.  This sounds perfect.  I think I can configure a custom header during the launch of a specific application, and StoreFront will take that name, configure the CLIENTNAME and voila!

So how would you do this?  Jason Samuel goes over the configuration of the necessary steps (it’s in both Sam and Simon’s posts as well, but I found Jason’s post more clean and clear).  The exception is (step 6 in Jason’s post) the clientNameRewriteRule needs to be set with the custom header we are going to look for.

My instructions are:

  1. Edit “C:\inetpub\wwwroot\Citrix\Store\web.config” (note: this is the “STORE” web.config and not the STOREWEB)
    Configure a section “appSettings” to be like so:

  2. Find the “overrideIcaClientName” and ensure it’s set to “on”:

  3. Edit “C:\inetpub\wwwroot\Citrix\StoreWeb\web.config”  (note: this is the “STOREWEB” web.config)
    Add a ‘forwardedHeaders’ section under “communication”

  4. Edit the “C:\inetpub\wwwroot\Citrix\StoreWeb\custom\script.js” file and add the following:
  5. Download Sam’s “StoreCustomization_Input.dll” from here and overwrite the placeholder located here:

And?  Did it work?!

Look at that!  It works!  Splendid!

And now I can configure ClientName based on the application that is launched!  So now we do not need to create multiple stores, we can keep this to a single store and it’s more dynamic and cleaner (IMHO).  The only caveat is upgrading Storefront these steps may need to be done again as the web.config files may be cleaned or reset.

UDPATE! I found that customizations were being inconsistently applied!  Check this post for further details and a possible fix!

Read More

Check for NUMA within a VM guest with Powershell

/ /
in Blog

I have only tested this in VMWare, but it seems to work.  I have shamelessly stole the main code from here:


I was going to use this to detect NUMA and configure a network adapter RSS based upon NUMA configuration…   But I’m getting lazy and am going to ignore it.  But I don’t want this code to go away.  So here it is.  A way to detect if you are on a NUMA system in a guest VM in Powershell.

If you have greater than “1” for the NumaNode count then NUMA is present.

CoreInfo.exe result on the same system:


Read More

Remove Ghost devices natively with Powershell

/ /
in Blog

We’ve been looking at using Base Image Scripting Framework (BISF) as our new preparation and personalization platform for our PVS farm.  In our current world we have a bunch of features and tweaks that are not apart of BISF so I’ve been writing some additions so that we achieve feature parity.

One of the features I wanted to take a good hard look at was removing ‘Ghost’ devices — or devices that aren’t present in your system.  Ghost devices look like this:

Our current world accomplishes this task by using a script written by Simon Price and devcon.exe.  There is nothing wrong with this method per se but BISF is native Powershell and I want to stick to that without having outside dependencies.  Can this requirement be achieved with nothing but PowerShell?  Fortunately, google came to rescue and pointed me here.  A script written by Alexander Boersch got me 80% of the way there (whoo hoo!).  He wrote the method and ability to access setupapi.dll which gives us the functions and methods necessary to query and manipulate devices in Device Manager.  PowerShell is supposed to have the ability to do C# code natively and his example was perfect for taking me where I needed to go.

How does it work?  What’s the output?



notice the “filter match” text



And a brief video of it in action:


Lastly, the script: