Uncategorized

Query a bunch of Windows 2003 event logs

2013-08-20
/ / /
This will find event ID 3001 in the Application log file with a list of computers from “systems.txt”

Read More

Windows Server 2012 R2 cache drive size for parity drives

2013-08-19
/ / /

It turns out the maximum a parity drive write cache size can be is 100GB.  I have a 500GB SSD (~480GB real capacity) so the maximum write cache size I can make for a volume is 100GB.  I suspect I maybe able to create multiple volumes and have each of them with a write cache of 100GB.  Until then this is the biggest it seems you can make for a single volume, so MS solves that issue of having a too large write cache.

Read More

Testing Windows Storage Spaces Performance on Windows 2012 R2

2013-08-08
/ / /

Windows Storage Spaces parity performance on Windows Server 2012 is terrible.  Microsoft’s justification for it is that it’s not meant to be used for anything except “workloads that are almost exclusively read-based, highly sequential, and require resiliency, or workloads that write data in large sequential append blocks (such as bulk backups).”

I find this statement to be a bit amusing because trying to back up anything @ 20MB/sec takes forever.  If you setup a Storage Spaces parity volume at 12TB (available space) and you have 10TB of data to copy to it just to get it going it will take you 8738 seconds, or 145 hours, or 6 straight days.  I have no idea who thought anything like that would be acceptable.  Maybe they want to adjust their use case to volumes under 1GB?

Anyways, with 2012R2 there maybe some feature enhancements including a new feature for storage spaces; ‘tiered storage’ and write back caching.  This allows you to use fast media like flash to be  a staging ground so writes complete faster and then the writes to the fast media can transfer that data to the slower storage at a time that is more convient.  Does this fix the performance issues in 2012?  How does the new 2-disk parity perform?

To test I made two VM’s.  One a generic 2012 and one a 2012R2.  They have the exact same volumes, 6x10GB volumes in total.  The volumes are broken down into 4x10GB volumes on a 4x4TB RAID-10 array, 1x10GB volume on a 256GB Samsung 840 Pro SSD and 1x10GB volume on a RAMDisk (courtesy of DataRAM).  Performance for each set of volumes is:

4x4TB RAID-10 -> 220MB/s write, 300MB/s read
256MB Samsung 840 Pro SSD -> ~250MB/s write, 300MB/s read
DataRAM RAMDisk -> 4000MB/s write, 4000MB/s read

The Samsung SSD volume has a small sequential write advantage, it should have a significant seek advantage, as well since the volume is dedicated on the Samsung it should be significantly faster as you could probably divide by 6 to get the individual performance of the 4x10GB volumes on the single RAID.  The DataRAM RAMDisk drive should crush both of them for read and write performance under all situations.  For my weak testing, I only tested sequential performance.

First thing I did was create my storage pool with my 6 volumes that reside on the RAID-10.  I used this powershell script to create them:

The first thing I did was create a stripe disk to determine my maximum performance amoung my 6 volumes.  I mapped to my DataRAM Disk drive and copied a 1.5GB file from it using xcopy /j

Performance to the stripe seemed good.  About 1.2Gb/s (150MB/s)

I then deleted the volume and recreated it as a single parity drive.

Executing the same command xcopy /j I seemed to be averaging around 348Mb/s (43.5MB/s)

This is actually faster than what I remember getting previously (around 20MB/s) and this is through a VM.

I then deleted the volume and recreated it as a dual parity drive.  To get the dual parity drive to work I actually had to add a 7th disk.  5 nor 6 would work as it would tell me I lacked sufficient space.

Executing the same command xcopy /j I seemed to be averaging around 209Mb/s (26.1MB/s)

I added my SSD volume to the VM and deleted the storage spaces volume.  I then added my SSD volume to the pool and recreated it with “tiered” storage now.

When I specified to make use the SSD as the tiered storage it removed my ability to create a parity volume.  So I created a simple volume for this testing.

Performance was good.  I achieved 2.0Gb/s (250MB/s) to the volume.

With the RAMDisk as the SSD tier I achieved 3.2Gb/s (400MB/s).  My 1.5GB file may not be big enough to ramp up to see the maximum speed, but it works.  Tiered storage make a difference, but I didn’t try to “overfill” the tiered storage section.

I wanted to try the write-back cache with the parity to see if that helps.  I found this page that tells me it can only be enabled through PowerShell at this time.

I enabled the writecache with both my SSD and RAMDisk as being a part of the pool and the performance I got for copying the 1.5GB file was 1.8Gb/s (225MB/s)

And this is on a single parity drive!  Even though the copy completed quickly I could see in Resource Manager the copy to the E: drive did not stop, after hitting the cache at ~200MB/s it dropped down to ~45-30MB/s for several seconds afterwards.

You can see xcopy.exe is still going but there is no more network activity.  The total is in Bytes per second and you can see it’s writing to the E: drive at about 34.13MB/s

I imagine this is the ‘Microsoft Magic’ going on where the SSD/write cache is now purging out to the slower disks.

I removed the RAMDisk SSD to see what impact it may have if it’s just hitting the stock SSD.

Removing the RAMDisk SSD and leaving the stock SSD I hit about 800Mb/s (100MB/s).

This is very good!  I reduced the writecache size to see what would happen if the copy exceeded the cache…  I recreated the volume with the writecachesize at 100MB

As soon as the writecache filled up it was actually a little slower then before, 209Mb/s (26.1MB/s).  100MB just isn’t enough to help.

100MB of cache is just not enough to help

Here I am now at the end.  It appears tiered storage only helps mirrored or stripe volumes.  Since they are the fastest volumes anyways, it appears the benefits aren’t as high as they could be.  With parity drives though, the writecachesetting has a profound impact in the initial performance of the system.  As long as whatever fills the cache as enough time to purge to disk in the inbetweens you’ll be ok.  By that I mean without a SSD present and write cache at default a 1GB file will copy over at 25MB/s in 40 seconds.  With a 100MB SSD cache present it will take 36 seconds because once the cache is full it will be bottlenecked by how fast it can empty itself.  Even worse, in my small scale test, it hurt performance by about 50%.  A large enough cache probably won’t encounter this issue as long as there is sufficient time for it to clear.  Might be worthwhile to invest in a good UPS as well.  If you have a 100GB cache that is near full and the power goes out it will take about 68 minutes for the cache to finish dumping itself to disk.  At 1TB worth of cache you could be looking at 11.37 hours.  I’m not sure how Server 2012R2 deals with a power outage on the write cache, but since it’s a part of the pool I imagine on reboot it will just pick up where it left off…?

Anyways, with storage spaces I do have to give Microsoft kudos.  It appears they were able to come close to doubling the performance on the single parity to ~46MB/s.  On the dual-parity it’s at about 26MB/s under my test environment.  With the write cache everything is exteremely fast until the write cache becomes full.  After that it’s painful.  So it’s very important to size up your cache appropriately.  I have a second system with 4x4TB drives in a storage pool mirrored configuration.  Once 2012 R2 comes out I suspect I’ll update to it and change my mirror into a single parity with a 500GB SSD cache drive.  Once that happens I’ll try to remember to retest these performance numbers and we’ll see what happens 🙂

Read More

Aside post format. Pellentesque hendrerit interdum dolor. Maecenas eget lectus rhoncus, congue risus in, sodales nisi. Aliquam erat volutpat. Vestibulum scelerisque, elit vitae lacinia fringilla, lorem erat convallis sem, et pharetra lacus sapien at lorem. Pellentesque habitant morbi tristique senectus et netus et malesuada fames ac turpis egestas.

“An error occurred while making the requested connection” – Citrix Web Interface

2013-07-22
/ / /

So I’m getting the dreaded “An error occurred while making the requested connection” while trying to launch some applications from our Citrix Web Interface.  It started happening suddenly but I’m tasked with figuring out why.  First thing I did was go to the Web Interface and check the event logs.  I found the following:

 

This wasn’t much help, but I was able to narrow down that this was happening on one set of our servers that are split across two DC’s.  One set of servers at BDC was fine, the other set of servers at ADC had a subset of servers that were not.  Doing a qfarm /load showed the problematic servers had no users on them at all, and no load evaluators were applied that would be causing our issue.

Logging into the server it was deteremined that it’s DNS was registered to the wrong NIC (it was a PVS server that was multi-homed) and even worse for some of the servers, the NIC IP address was an old address and the new address wasn’t even resolving!

For some reason it now appears our Windows 2008 servers are not registering their DNS on startup.  To resolve this issue for us we added a startup script with the simple command “ipconfig /registerdns” and within a few seconds the IP address is registered within DNS correctly and with the correct NIC.  We suspect that something is misconfigured at ADC as BDC does not have this issue nor does it need this tweak, but this is our work around until that is resolved.

Read More

Powershell script to compare Citrix Webinterface files

2013-07-17
/ / /

I’ve created a powershell script that will compare Citrix Webinterface files then export them out to a csv file.

 

Read More

Import-Csv -header $variable

2013-07-17
/ / /

I ran into an issue with Import-CSV where I was trying to pass a variable to the -header and it was failing pretty horribly, creating a single field instead of multiple fields.

I created a script to generate a CSV from Citrix WebInterface .conf files to compare the various web interfaces we have so that when we migrate users from the older interface to the newer interfaces we can properly communicate to them the changes they will experience.  In the course of developing this script I wanted to do a “dir *.conf” command and use that output as the header in the CSV.  Here is what I did originally:

This failed.  Our $header variable became a single header in the CSV.  The help for “Import-CSV” says the -header should have a string value.  I tried $header.ToString() but it didn’t work either.  I found you need to do the following:

This makes an array then adds the appropriate values to the $header variable and the import-csv now has multiple columns.

Hurray!

Read More

Utilizing MCLI.exe to comment the newest version of a vDisk on Citrix Provisioning Services (PVS)

2013-07-16
/ / /

I’ve written this script to utilize MCLI.exe to add a comment to the newest version of a vDisk and have marked which fields correspond to what.

 

 

This script can now be added to the “PVS Automatic” update feature to automatically comment the latest vDisk when it is updated.

Read More

Troubleshooting Audio issues in Citrix XenApp

2013-07-10
/ / /

We recently ran across an issue with XenApp 6.5 where we were publishing an application that required the “Beep” but it wasn’t working.  The following is the troubleshooting steps I did to enable audio to work on that application.

First we created a Citrix policy to enable audio.  This policy looked like so:

We filtered on a user security group to enable the client audio redirection and added that filter group to the application.  From the original appearance of things, this should have been sufficient to enable client redirection.  But it did not.  So I wanted to verify that the policy was actually applying to the user account.  To do that, you login to the system with a user account and check in Regedit for the value “AllowAudioRedirection”.  If it’s set to 0x1 then the Citrix group policy has evaluated that client redirection should be enabled for your session.

Unfortunately, I still did not have audio redirection working…

Citrix advises that you can use dbgview.exe to further troubleshoot the Citrix Receiver to assist.  I launched dbgview.exe and started the trace and launched the application from the Webinterface.

“00000043 0.60113800 [7752] CAMSetAudioSecurity: Wd_FindVdByName failed”

CAM is a virtual channel (Virtual Channel Priority for Audio) and we can see it’s failing.  I then used the Citrix ICA creator and launched the application using that.  The dbgview for that output looks like so:

00000013 0.44386363 [4496] CAMSetAudioSecurity: success

We can see that the audio virtual channel was able to successfully latch and I confirmed I had audio in the application.

From here the issue appeared to be when I launched the application from the webinterface or desktop shortcut.  I then compared the two ICA files, the one from the web interface and the one I created separately to see what was different.  The difference was glaringly obvious.  The working ICA file had “ClientAudio=On” and the broken one had “ClientAudio=Off”.

Curious, I launched AppCenter and clicked through the applications settings and saw the following:

“Enable legacy audio” was unchecked.  I checked it and then logged off and logged back on the web interface and when I downloaded the ICA file, “ClientAudio=On” and I had audio.  I then unchecked that setting and confirmed it manipulated the ICA file as with it unchecked the ICA file generated had “ClientAudio=Off”

Who knows why it’s called “legacy audio”.  May as well just call that option “Enable audio” as that would be more accurate.  The Citrix documents on this setting says the following:

http://support.citrix.com/proddocs/topic/xenapp6-w2k8-admin/ps-sessions-en-dis-aud-pubapp-v2.html
To enable or disable audio for published applications

If you disable audio for a published application, audio is not available within the application under any condition. If you enable audio for an application, you can use policy settings and filters to further define under what conditions audio is available within the application.

  1. In the Delivery Services Console, select the published application for which you want to enable or disable audio, and select Action > Application properties. 
  2. In the Application Properties dialog box, click Advanced > Client options. Select or clear the Enable legacy audio check box.

Emphasis is mine.

Anyways, and now we have our applications with working audio and everything seems to be good again 🙂

To summarize the enable audio for a XenApp application you must:
1) Enable “legacy” audio
2) Enable a Citrix policy to configure audio redirection
3) Done.

Read More