Application virtualization, IoT and Cloud Computing, Blog of Sacha Thomet

Troubleshooting

Citrix CVAD and MTU Discovery

Update November 2022:

Martin Latteier wrote me that Citrix introduces a new feature which addresses the issue. Fact is, on certain cable connections with DS-Lite (IPv6 only) the MTU discovery with EDT did not work properly, because the cable modem did not process the “DF flag”, which caused the MTU discovery to detect a too high MTU, the datagrams had to be fragmented and the performance was bad. The solution was to switch off EDT on the client side or to change the MTU discovery to a static MTU (edtMSS=13xx).

If you set this flag the issue should be solved:

Documentation: https://docs.citrix.com/en-us/citrix-gateway/current-release/hdx-enlightened-data-transport-support/pmtud-support-on-citrix-gateway.html

Update January 2022:
Only update the default.ica how it’s described below, referencing to https://support.citrix.com/article/CTX231821 if you also are using Citrix Workspace App for Mac in Version 2201 or later!

Recently I saw an increased number of issues by connecting to Citrix virtual desktops and apps. Desktops were connected from private devices over a Citrix ADC (aka Netscaler) but virtual apps were connected from our internal network from different branch offices.
I wrote a blog article some weeks ago: Issues by accessing Citrix Virtual Apps and Desktops on a pure IPv6 Internet Provider with enabled EDT

Our first workaround for this issue was to disable the UDP based EDT protocol which is now the default for Citrix Virtual Apps and Desktops. But this is not really our intended solution because all the benefits of EDT get lost, so we just did it on client-side with a reg hack, but so we have to identify all the clients with issues. (see the recent blog article)

Further analysis of this issue showed that the root cause of this issue is on the MTU, which means the Maximum Transmission Unit or simplified said Package size.
I found out that our branch offices has a smaller MTU than 1500 what is the default, the reason is a crypto-tunnel which is configured to all our branches. So there will be a fragmentation of the packages, in some cases the fragmentation causes problems because the payload is no more enough that a session launch will work. It looks like EDT is much more sensible on that than the classic ICA protocol on TCP.

Solution 1: How to configure MSS when using EDT on networks with non-standard MTU
In the recent blog article, I called Citrix “please fix that” but they already started on this at this point. In this article, you can see how to configure EDT for non-standard MTUs https://support.citrix.com/article/CTX231821 e.g. put the MTU to 1480 with the default.ica
The disadvantage of this solution is that the MTU is lowered for all connections, also it’s needed to figure out which MTU fits best. The benefit of this solution that it works on every platform with the exception of Android.

Solution 2: enable MTU Discovery
An even smarter option Citrix introduces on the background with CVAD 19.12, this is MTU Discovery for EDT. This feature gives the possibility to have the best MTU for each session. MTU discovery is not a invention by Citrix, just new on EDT. You can read more about MTU discovery here:
https://en.wikipedia.org/wiki/Path_MTU_Discovery

I tried to visualize these 3 scenarios:

standard, MTU set to 1500

Default configuration, MTU to 1500

The default is that MTU of 1500 bytes are sent, if the MTU on the path is somewhere smaller the package gets fragmented, what is not a big issue but in real life we see that it can be a problem.

MTU reduced to 1380 bytes according to the mentioned Citrix article
(Change in default.ica on StoreFront)

Now packaged of 1380 bytes are transmitted.
Is the possible MTU somewhere on the path smaller than 1380, the packages are still fragmented. Example on this picture, the BYOD client from the Internet.
Also, the MTU is lowered when not needed, in this example the Client in the HQ
+ This solution works on all platforms except Android.

MTU with MTU Discovery (Registry Setting on VDA)

MTU Discovery alway use best MTU

+ The MTU is figured out optimal and individually for every session. First, the Session will start with 1024 bytes and the MTU will be increased during the Session.
– Needs Citrix Workspace App for Windows 19.11 or newer.
– If the connection goes over an ADC its a minimum required version 13.0.52.24 or 12.1.56.22. If you are on a lower version the MTU perhaps remains on 1024 bytes!

You can check your EDT MTU with this command:
ctxsession -v

MTU Discovery is today not active as a default, you need to activate it on the VDA with a Registry Key, the VDA must be at minimum 19.12, details here: https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/technical-overview/hdx/adaptive-transport.html#edt-mtu-discovery

Update 5.6.2020 – Combine solutions 1 and 2 and have the best config!
For our case, we have separated Storefront Servers for access via ADC, so I decided to implement a solution with the best of both worlds… I set the MTU in the default.ica to 1300 and I enable MTU Discovery – so Apple macOS can use the setting from the default.ica and Windows computers have the optimal value with MTU discovery

Technical:
on VDA, add this DWORD: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\Wds\icawd
Name: MtuDiscovery
Type: DWORD
Data: 00000001

default.ica, add this lines:
OutBufLength=1300
udtMSS=1300
edtMSS=1300

If MTU Discovery is active, means enabled and accessed from a Citrix Workplace App the MTU Discovery override the value set in the default.ica, what is very helpful!

How do you configure MTU on your Citrix with EDT environment? Leave a comment!

Issues by accessing Citrix Virtual Apps and Desktops on a pure IPv6 Internet Provider with enabled EDT

Some weeks ago first Incidents reached my team with the problem description that Users having a problem accessing their pooled Windows 10 desktops trough Citrix ADC (aka Netscaler).

I noticed that all affected users have one thing in common, they all have UPC Cablecom as an Internet Provider. As soon they connect their device to another internet connection, for instance to the Wifi hotspot created with their mobile phone all works fine.

Further troubleshooting steps showed that if I publish a dedicated VDI with disabled EDT also all works fine. If you don’t know what EDT is, Google EDT and Citrix 🙂

Usually, EDT is enabled and uses UDP for communication if the port is open and it’s possible. Otherwise, it should fall back to TCP. It seems to fall back just works if the ports are closed.

Now my problem is, I can’t disable EDT for all pooled VDI as we have some users who really need it. I also won’t give all the users dedicated desktops.
Luckily Julian Jakob gave me the hint to disable EDT on client-side. This is the best workaround.

Guidance for the User
(Because of BYOD the user needs to do that … )

On Windows it’s just a Registry key which the users have to create:

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Citrix\ICA Client\Engine\Lockdown Profiles\All Regions\Lockdown\Network\UDT]
"HDXoverUDP"="Off"

On a Mac this command should be typed in the terminal:

defaults write com.citrix.receiver.nomas HDXOverUDPAllowed -bool NO

On iOS devices you can disable EDT in the Workpace App:

Settings => Advanced => Adaptive Transport Settings => EDT (set on inactive)

Please Citrix fix that! All that I want is a fall back to TCP if UDP is not working, I don’t care if the port is closed or if it’s another issue. If EDT doesn’t work don’t use it. For users, this manual config on the client-side is not just annoying, some are not able to do it.

Update 26.5.2020

Update 4.6.2020

I’ve enabled MTUDiscovery and it really looks promising, please be aware that MTU Discovery currently is just for the WorkspaceApp for Windows available. (CWA 19.12 an newer)

Fatal error during installation (1603) on StoreFront upgrade to 3.12

Today I’ve upgraded Citrix StoreFront Server from 3.9 to 3.12

as every time first I shut down all of the following services:

net stop W3SVC
net stop CitrixConfigurationReplication
net stop CitrixCredentialWallet
net stop CitrixDefaultDomainService
net stop “Citrix Subscriptions Store”
net stop “Citrix Peer Resolution Service”
net stop CitrixServiceMonitor
net stop CitrixTelemetryService

then I run CitrixStoreFront-x64.msi, reboot the server and after that I do that on the second node. Most time this goes pretty flawless.

But this time, my upgrade failed with an error:

CitrixStoreFront-x64.msi’ failed with error code 1603. Fatal error during installation”

I remember I had this already one time before, but what the hell was the solution … a short search with Google showed me:

https://discussions.citrix.com/topic/371535-storefront-upgrade-to-301-from-300-fails

Well I’m on StoreFront 3.9 and when I have a look into “C:\Program Files\Citrix\Receiver StoreFront\Services\ProtocolTransitionService\Citrix.DeliveryServices.ProtocolTransition.ServiceHost.exe.config” I see in some lines “Version=3.8.0.0” – but I have 3.9, so I replace all “Version=3.8.0.0” to “Version=3.9.0.0”

Result: StoreFront upgrade to 3.12 is successful – All’s well that ends well.

No more able to start SOAP on PVS

After the last monthly Microsoft Security Updates one of my PVS Servers was no more able to start the SOAP service. I received an Event 7000 with the message:

The Citrix PVS Soap Server service failed to start due to the following error: The service did not respond the the start or control request in a timely fashion.

I live in Bern, and we are known as slow-paced people here in Bern, probably because of our slow sounding accent. So my idea is if the service need more time to start, I’ll give him more time.

I’ve created a new DWORD called ServicesPipeTimeout  with the value 120000 in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control which means the service has 2 minutes time to start. After a reboot my SOAP was again up and running.

By the way and additional tip according this service… SOAP is sometimes bitchy … it’s a good idea to set the service to auto restart after a crash.

Avoid blanks and dots in StoreFront 3.5 farm names!

Last day’s I upgraded my existing Storefront 3.01 to StoreFront 3.5 and in some farms I had a very bad issue after the upgrade, it seemed that all is ok but users are no more able to start any application or desktop.

On the StoreFront server I saw an Warning Event 28 from Citrix Store Service “ Failed to launch the resource “Farm Name.ApplicationName” as it was not found.

SF35_blanks_error-event28

The issue was that a blank and a dot (.) was in my Farmname, seems this is a bug in SF3.5 that you can configure that but It won’t work!

SF35_blanks_error-config

after I changed the display name to something like xa65farm without blank and point all works well.

Client to Server Content Redirection no more working because of License Error

Recently I faced a really strange error with a weird solution. Because I didn’t found this on the web I post this here, maybe it can help someone else too.

We had the issue that users cannot use anymore the Client to Server Content Redirection, means they can’t double-click anymore on *.vsd files to open the file with the XenApp server installed MS Visio.

After some minutes of investigation and checking the Logs on Client, XenApp server and so on we’ve finally checked the EventLog on the Webinterface server and found the following error:

Event 31007
 
eventlog-contentredirectionerror
Site path: C:\inetpub\wwwroot\Citrix\PNAgent. 

The Citrix servers are not licensed to support workspace control. This message was reported from the XML Service at address 
http://myserver.ch:8080/scripts/wpnbr.dll [com.citrix.xml.NFuseProtocol.RequestReconnectSessionData]. 
[Unique Log ID: a5e760c4] 

For specific information about this message, see the Web Interface documentation at 
http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.

Now we remembered that we consolidated some Citrix license server’s and changed the license server for this farm some days ago. Of course we rebooted all XenApp servers, but not the Webinterface servers because they don’t use a special license server. A reboot of the Citrix Webinterface server solved this issue.

“Cannot complete your request‘” on Netscaler Gateway VPX

In my lab environment I was using a Citrix Webinterface 5.x which was accessible  from Internet over a Access Gateway 5 VPX. Since Citrix Store Front is in a fairly usable release (> Version 2.x), I intended to update my lab environment to the current software releases and update my skills to Store Front and Netscaler Gateway VPX.

You can find a step by step Netscaler Gateway intro here http://blogs.citrix.com/2013/07/03/citrix-netscaler-gateway-10-1-118-7-quick-configuration-wizard
Also a very nice guide you can find here, this guide also contains information about how to configure StoreFront for Netscaler Gateway VPX: http://benjamin.eavey.com/2013/07/netscaler-vpx-as-secure-gateway-replacement

Cannot complete your request

After completion of the configuration I was not able to access the my environment from outside. The login to the Netscaler Gateway, the black window, was working fine, but as soon I hit the StoreFront I get this Error:

cannot-complete

Because StoreFront is working fine from internal, I assumed that’s not a completely wrong StoreFront configuration. After i had a look into the event viewer on the StoreFront server I can see that something is wrong here:

eventlog_error_callback

 

The crucial indication that’s a problem between the Store Front server and the the Netscaler Gateway in role of Authentication Callback Server I found here:

eventlog_error_callback_event3

when I browse to the address https://192.168.x.x/CitrixAuthService/AuthService.asmx you can see a certificate error, so I need to have here a FQDN that match to the installed certificate but I wont communicate outside, so first I’ve defined the internal IP as Callback URL:

general-settings

 

Now I’ve changed the Callback URL to the FQDN appropriate to the certificate:

general-settings-ok-with-fqdn

But because the DNS resolve this URL as the external IP which is not accessible over the necessary TCP ports, I was constrained to do a dirty hack … I have edited my host file :

hosts

 

Citrix DSCHECK doesen’t work … “No resource module ImaMsgsUI.dll found.”

When you try to make a Citrix Datastore validaiton with DSCHECK you receive the error message :

” No resource module ImaMsgsUI.dll found. “

dscheck_doesent_work

Cause:
Probably you are on a Citrix server with UAC,

Solution:
Launch the command line with administrative privileges, “run as administrator” before you make the dscheck.

more details about dscheck.exe here: http://support.citrix.com/article/CTX124406