Provisioning Services

Powershell script to manipulate “Register this connection’s addresses in DNS”

2013-07-08
/ / /

We run a multihome NIC setup with our Citrix PVS servers and the “Provisioning” Network is a seperate VLAN that is only used by the PVS servers and goes no where.  Unfortunately, however, the “Provision” NIC can register itself in the DNS, causing devices outside of the Provisioning network (everyone) to resolve to the incorrect address.  To resolve this I ran this script across all my vDisks to remove the ability of the provision network to register itself as available in DNS.:

As you can see above, we have two NICs, “Provision” and “Production”.  We do not want “Provision” registerted so it gets the $false,$false set, whereas “Production” gets $true,$false.

Read More

Manually add Windows startup scripts (or inject startup scripts into an offline image)

2013-05-29
/ / /

Due to the CGP issue, our solution is to add a startup script to each vDisk.  Since I don’t want to make a version of each vDisk than attach it to a server than boot it up than gpedit.msc…  We have around 10 vDisks and that process would be annoying and take a while.  So I decided to investigate doing it offline as mounting a VHD using cvhdmount.exe and then injecting the startup script would be a lot easier.

To do that, one simply needs to browse to:
“C:\windows\system32\GroupPolicy\Machine\Scripts\Startup” (for machine startup script, aka, a script that starts when your computer starts up) or “C:\windows\system32\GroupPolicyUsers\Machine\Scripts\Startup” (for all users startup script [I’m assuming since I actually didn’t go through and test the user portion]) and copy your script file there.
Then, back out one level to C:\windows\system32\GroupPolicy\Machine\Scripts and edit Scripts.ini to include your new script file; incrementing the last line.
To:
Read More

(OS 10061)No connection could be made because the target machine actively refused it. : Unable to connect to the CGP tunnel destination (127.0.0.1:1494)

2013-05-27
/ / /

 

This has been an ongoing problem for us (Unable to connect to the CGP tunnel destination (127.0.0.1:1494)

I may have found out why it was happening in our environment.  We are using Provisioning Services and with it we are using two NIC’s, one for the Provisioning Services and one for Standard networking.

It appears the XTE service became configured to use the Provisioning Services NIC.  This was verified in the httpd.conf in the C:\Program Files (x86)\Citrix\XTE\conf folder.

Provisioning NIC and Production (network) NIC

 

httpd.conf as was when the system booted (and non-functional)

When I traced the XTE service using procmon.exe and wireshark with this non-functional conf this is what I saw when I attempted to launch the application:

You can see it attempt to connect to itself via 1494 but then nothing else happens
Wireshark shows virtually nothing on the network and nothing related to IMA
When I edited the file to have the Production NIC…

 
then restarted the XTE service and retraced via Procmon and Wireshark…
We now see tons of activity and the application now launches without issues.

================EDIT===============

We have now found why we are getting this error, and why we are getting it intermittently.  The issue is we are using PVS with multi-homed NIC’s.  One NIC (LanAdapter 1) is the “Provisioning” network, and the second NIC (LanAdapter 2) is the “Production” network.  The Provisioning network is on a completely seperate vLan and sees no traffic outside of it’s little network.  The ICA Listener was attaching itself to the Provisioning network instead of the production network, so when we tried to connect to the server it would fail with the CGP tunnel error because the outside network cannot talk to the Provisioning network.  To attempt to resolve this issue one of our techs (Saman) created a group policy preference registry key that set the following value (HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Terminal Server\WinStations\ICA-TCP – LanAdapter):

By setting it to “2” we could ensure the ICA listener is always listening on LanAdapter 2, our production network.  Unfortunately, a Windows Update appears to have caused either Group Policy Registry Preferences to execute (sometimes) *after* the IMAService service started, or allowed the IMAService service to start *before* Group Policy Registry Preferences.  IMAService will recreate that file every second restart.  To resolve this issue I created a startup script that executes after 65 seconds, deleting the httpd.conf file and restarting the appropriate services until the httpd.conf file is recreated.

In my testing it appears you need to restart the “IMAService” service twice to get it to recreate the httpd.conf file.  Because of this, I created the script to retry up to 3 times to try and regenerate the file.

Read More

Enable advanced logging on the XTE service of a XenApp server

2013-05-23
/ / /

Back in February or March we did Windows updates on our PVS XenApp servers and then sometime after the servers do not allow anyone to login.

The issue can crop up as a CGP Tunnel error message, a protocol driver error message or something along those lines.  We do not know why it’s happening or why it only happens after we apply Windows updates from that time period.  The odd thing is it’s intermittent as well, we can launch 20 systems from 1 vDisk that has the updates applied and everything will be fine for 2 weeks then, suddenly, 5 of the systems won’t allow logins via ICA with the errors.  Rebooting the servers sometimes fixes it, sometimes not.  Very intermittent and very weird.  So I attempted to troubleshoot this issue again by adding:

to the httpd.conf in the XTE folder of a server that was exhibiting these issues.  The log levels are listed here:
http://support.citrix.com/article/CTX114680

After adding those lines and restarting the XTE service the issue resolved itself!  Frustrating to be sure, and I will look at adding this line to our vDisk image so that when it crops up we’ll have more diagnostic data to look at.

Read More

Citrix Provisioning Services (PVS) 6.1 – Automatic vDisk Update “Update device failed to shutdown within the timeout period.”

2013-05-22
/ / /

I have setup our PVS environment to execute the vDisk Automatic Update feature utilizing a custom script (update.bat).  This script does a bunch of things, resync’s the time with NTP (to avoid daylight savings issues), refreshs GPO’s, executes Windows Update, cleans up temp files, runs the PVS optimizer, etc.

Unfortunately this can take longer than 30 minutes.  For some reason, when executing the ESD client as “None” (aka, so a script runs) the “Update timeout” doesn’t seem to take effect, instead the 30 minute shutdown timeout is on the clock.

ESD client is set to none
You cannot increase the shutdown timeout beyond 30 minutes
2013-05-22 10:10:59,690 [10] INFO  Mapi.MapiIPC - [xipProcessor] Starting an update on (System.Collections.Generic.Dictionary`2[System.String,System.Object]) 2013-05-22 11:02:07,763 [10] ERROR Mapi.MapiIPC - [xipProcessor] [WSCTXBLD303T] Update device failed to shutdown within the timeout period. 2013-05-22 11:02:07,763 [10] INFO  Mapi.MapiIPC - [xipProcessor] [WSCTXBLD303T] Update device failed to shutdown within the timeout period. 2013-05-22 11:02:56,186 [10] ERROR Mapi.MapiIPC - [xipProcessor] [WSCTXBLD303T] Submit image failed (VM: WSCTXBLD303T, Image: XenApp65Tn02, Update device failed to shutdown within the timeout period.) 2013-05-22 11:02:56,186 [10] INFO  Mapi.MapiIPC - [xipProcessor] [WSCTXBLD303T] Submit image failed (VM: WSCTXBLD303T, Image: XenApp65Tn02, Update device failed to shutdown within the timeout period.)
Total time for the updates was 49 minutes (10:11AM to 11:02AM)

The only solution I have been able to come up with so far is set to run the updates in less than 30 minutes.  I think I’ll attempt changing the “Update.bat” to “Preupdate.bat” and see if that avoids the “Shutdown timeout”.

Unfortunately, I do not know when the shutdown timeout period starts or why it starts.  I was hoping the “Update timeout” was started when running the “Update.bat” file.  This does not appear to be so, sadly.

Citrix documentation implies that you should work hard to keeping the timeout below 30 minutes.

“Citrix recommends to only apply updates that can be downloaded and installed in 30 minutes or less.”

======================EDIT==================
Preupdate.bat does not appear to operate under the Update Timeout either.

======================EDIT 2=================
To increase the limit you need to create a registry key called “DiskProvider” and create a dword with the decimal value of the length of time you want the *total* time called “deviceShutdownTimeout”.

NOTE: This does NOT change the value in the GUI and will override the  value in the GUI, regardless of what it is set to.  You need to restart the SOAP service after making this change.  This registry key must exist on the PVS server that the site designates as the “vDisk Update Server”

 

Read More

Corrupted VHD files

2013-05-14
/ / /

Corrupt VHD files…  We utilize PVS and when we were attempting to update the target device software utilizing Hyper-V we got several error messages saying the VHD is corrupted.  The actual root of the issue is that PVS makes 16MB block size VHD files (thanks SAMAN!) and Windows 2008 only reads VHD files that are 512KB or 2MB block sized files.

Attempting to mount the VHD file via disk management fails as well.  The only utility that would mount the VHD file that was generating the error messages at the end of this post is the Citrix CVHDMount utility available via PVS (C:\Program Files\Citrix\Provisioning Services\CVHDMount.exe)

Since our Hyper-V server was separate from the PVS server, we needed to install the drivers to allow CVHDMount.exe to work.  In order to do this you need to install the drivers in this folder:
C:\Program Files\Citrix\Provisioning Services\drivers

You can right-click “Install” on the cfsdep2.inf file.

For the other driver you need to open Device Manager, go through “Add new hardware…”, “Add legacy hardware” and then browse to the drivers folder and add the cvhdbusp6.inf.

Now we would mount the disk as a drive letter using this command line:

At this point you need to set the disk to “Offline” in Disk Management.

From here we can now add the disk as a physical disk to Hyper-V.

And we can now do the Target Device Update.  Error messages seen during this troubleshooting:

 

Read More

Citrix PVS bootup failure with the Boot Device Management ISO

2013-05-01
/ / /

We had an issue recently with some Citrix Provisioning Servers (PVS) that were not getting a DHCP address when booting off of the Boot Device Management ISO (BDM.ISO).  What was happening is the server would just show this:

DHCP Discover ._._._._.

DHCP maximum number of retries reached.

When booting off of PXE or a WinPE CD I was getting DHCP without issues.

When I did a Wireshark of the line I saw the BDM.ISO boot send out DHCPDISCOVER packets but it never received a response.  I then Wiresharked the WinPE and PXE boots and saw the DHCPDISCOVER packet, followed by a DHCPOFFER, and so on.  When I examined the two packets I saw the BDM.ISO DHCPDISCOVER packet actually was a BOOTP unicast whereas the PXE and WinPE packets were BOOTP broadcast.  Thinking we had a DHCP Relay issue we checked our DHCP server (an InfoBlox server) and checked the logs for the MAC Address and here is what we saw when we booted with the BDM.ISO:

BDM.ISO DHCP traffic

The DHCP server was not responding to the DHCPDISCOVER.  This only occurred with the unicast packet and for some reason was “load balance to peer”.  However it’s setup, it appears UNICAST BOOTP packets are setup for load balancing but not sending a response.

PXE/WinPE DHCP traffic

The DHCP server is responding to a BROADCAST BOOTP packet in a very different way.  There is no load balancing going on and the server responds to the DISCOVER packet.

Unfortunately, we did not resolve this issue when I wrote this.  We got our farm to work by pointing the DHCP Relay to the previous DHCP server that is configured in such a way to resolve this DHCP request and present an DHCP OFFER.  Hopefully our network guys will get the DHCP fixed on the proper server.  If you are experiencing similar issues you may have a similar issue where the DHCP at your site is handling unicast DHCPDISCOVER packets differently then broadcast packets.

Read More

Reboot Server and PVS Streaming Service is started but the console shows the service is down

2013-04-25
/ / /

Upon rebooting a server we found the Citrix PVS Console showed the server as down.  When we investigated the server we found the service was started and their were no errors in the logs that we could see.  Restarting the service brought the server as up in the console.  We did see one particular error though, the date was suddenly incorrect in the event viewer.

 

Further investigation showed EventID 52, the time service resync’ed a offset.

Since this was a virtual machine we checked the VMWare settings to confirm that the time was not being sync’ed

But the time was still getting offset.  Further investigation showed the VMWare hosts time was not set correctly and the server was having it’s time set to the hosts time; even though the above check box was not set.

It appears VMWare has additional time synchronization settings that are enabled by default and must be set to explicitly deny to not have the time synchronize from different scenarios.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1189

Upon VMWare Tools starting on reboot, a “resume”, or the tools being restarted or other scenarios.  To prevent it from happening you must edit the VMX file and set the values as stated in the kb article above.

Read More

(OS 10061)No connection could be made because the target machine actively refused it. : Unable to connect to the CGP tunnel destination (127.0.0.1:1494)

2013-04-15
/ / /

(OS 10061)No connection could be made because the target machine actively refused it.  : Unable to connect to the CGP tunnel destination (127.0.0.1:1494)

We got this error on a XenApp 6.5 provisioning server

We utilize Citrix PVS for our XenApp servers.  After updating our XenApp vDisk using our usual method and rebooting the server we got the above error.  Since we versioned it we had our usual servers working without issue, and a couple of “test” versions failing.  I started troubleshooting this using the google method of shotgun trying the first 5 error/solutions.  This included, disabling re-enabling ICA listener, setting the listener on only one NIC interface (which it was, I toggled it for all then just one), deleting the ICA listener and recreating it.  None of them worked so I started troubleshooting.

I encountered this error today and started troubleshooting.  The first step I did was trace the path to the error using Procmon.exe.  This, unfortunately, did not reveal anything of substance.  What I found with this error is the network process is almost exactly the same for both.  I can see XTE.exe communicating via 2598, I can see, what appears to be it handing it off to IMASrv.exe.  At this point is when it causes that error message to be displayed on the bad machine, but IMASrv.exe continues on the good machine.

At this point I’m thinking that maybe the IMA settings for session reliability have become corrupted.  It was suggested to me to disable session reliability and see if it works by a colleague of mine.

The odd thing about resetting the IMA policy is that it isn’t even set in the first place.

“Add” means it is not set

But I disabled it anyways in the IMA policy in AppCenter, NOT in group policy.  After setting session reliability to disabled I restarted the IMA Service on the server and the XTE service.  The XTE service refused to start, I assume because we disabled it, and the IMA service came up.  I then connected to the application via 1494 without issue.  At this point I removed the session reliability setting again.

Click “Remove”

And then waited a minute or so and restarted the IMA Service and XTE Service.  The XTE service came up cleanly, running netstat -a only showed 2598 was listening and I retried the application hosted on the troublesome server.

BANG, it worked like a charm.  I wish I knew why it was happening with more detail, unfortunately, this is as far as I got and I cannot make the error occur again.

The next time this occurs I would like to try stopping the IMA Service and removing this file:
C:\Program Files (x86)\Citrix\FarmGpo\FarmGpo.bin

I think that contains the local IMA policy that may have been corrupted and causing my issue.

=======================EDIT===================

I believe I have fixed the issue here.  It appears it may be a NIC binding issue.

Read More

Click the PVS Target Device Optimizer button with AutoIt

2013-02-27
/ / /

Short but sweet script.  Disappointing that Citrix doesn’t have a silent run switch for this application.

 

Read More