Troubleshooting
Citrix CVAD and MTU Discovery
Update November 2022:
Martin Latteier wrote me that Citrix introduces a new feature which addresses the issue. Fact is, on certain cable connections with DS-Lite (IPv6 only) the MTU discovery with EDT did not work properly, because the cable modem did not process the “DF flag”, which caused the MTU discovery to detect a too high MTU, the datagrams had to be fragmented and the performance was bad. The solution was to switch off EDT on the client side or to change the MTU discovery to a static MTU (edtMSS=13xx).
If you set this flag the issue should be solved:
Update January 2022:
Only update the default.ica how it’s described below, referencing to https://support.citrix.com/article/CTX231821 if you also are using Citrix Workspace App for Mac in Version 2201 or later!
Recently I saw an increased number of issues by connecting to Citrix virtual desktops and apps. Desktops were connected from private devices over a Citrix ADC (aka Netscaler) but virtual apps were connected from our internal network from different branch offices.
I wrote a blog article some weeks ago: Issues by accessing Citrix Virtual Apps and Desktops on a pure IPv6 Internet Provider with enabled EDT
Our first workaround for this issue was to disable the UDP based EDT protocol which is now the default for Citrix Virtual Apps and Desktops. But this is not really our intended solution because all the benefits of EDT get lost, so we just did it on client-side with a reg hack, but so we have to identify all the clients with issues. (see the recent blog article)
Further analysis of this issue showed that the root cause of this issue is on the MTU, which means the Maximum Transmission Unit or simplified said Package size.
I found out that our branch offices has a smaller MTU than 1500 what is the default, the reason is a crypto-tunnel which is configured to all our branches. So there will be a fragmentation of the packages, in some cases the fragmentation causes problems because the payload is no more enough that a session launch will work. It looks like EDT is much more sensible on that than the classic ICA protocol on TCP.
Solution 1: How to configure MSS when using EDT on networks with non-standard MTU
In the recent blog article, I called Citrix “please fix that” but they already started on this at this point. In this article, you can see how to configure EDT for non-standard MTUs https://support.citrix.com/article/CTX231821 e.g. put the MTU to 1480 with the default.ica
The disadvantage of this solution is that the MTU is lowered for all connections, also it’s needed to figure out which MTU fits best. The benefit of this solution that it works on every platform with the exception of Android.
Solution 2: enable MTU Discovery
An even smarter option Citrix introduces on the background with CVAD 19.12, this is MTU Discovery for EDT. This feature gives the possibility to have the best MTU for each session. MTU discovery is not a invention by Citrix, just new on EDT. You can read more about MTU discovery here:
https://en.wikipedia.org/wiki/Path_MTU_Discovery
I tried to visualize these 3 scenarios:
standard, MTU set to 1500
The default is that MTU of 1500 bytes are sent, if the MTU on the path is somewhere smaller the package gets fragmented, what is not a big issue but in real life we see that it can be a problem.
MTU reduced to 1380 bytes according to the mentioned Citrix article
(Change in default.ica on StoreFront)
Now packaged of 1380 bytes are transmitted.
– Is the possible MTU somewhere on the path smaller than 1380, the packages are still fragmented. Example on this picture, the BYOD client from the Internet.
– Also, the MTU is lowered when not needed, in this example the Client in the HQ
+ This solution works on all platforms except Android.
MTU with MTU Discovery (Registry Setting on VDA)
+ The MTU is figured out optimal and individually for every session. First, the Session will start with 1024 bytes and the MTU will be increased during the Session.
– Needs Citrix Workspace App for Windows 19.11 or newer.
– If the connection goes over an ADC its a minimum required version 13.0.52.24 or 12.1.56.22. If you are on a lower version the MTU perhaps remains on 1024 bytes!
You can check your EDT MTU with this command:
ctxsession -v
MTU Discovery is today not active as a default, you need to activate it on the VDA with a Registry Key, the VDA must be at minimum 19.12, details here: https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/technical-overview/hdx/adaptive-transport.html#edt-mtu-discovery
Update 5.6.2020 – Combine solutions 1 and 2 and have the best config!
For our case, we have separated Storefront Servers for access via ADC, so I decided to implement a solution with the best of both worlds… I set the MTU in the default.ica to 1300 and I enable MTU Discovery – so Apple macOS can use the setting from the default.ica and Windows computers have the optimal value with MTU discovery
Technical:
on VDA, add this DWORD: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\Wds\icawd
Name: MtuDiscovery
Type: DWORD
Data: 00000001
default.ica, add this lines:
OutBufLength=1300
udtMSS=1300
edtMSS=1300
If MTU Discovery is active, means enabled and accessed from a Citrix Workplace App the MTU Discovery override the value set in the default.ica, what is very helpful!
How do you configure MTU on your Citrix with EDT environment? Leave a comment!
Issues by accessing Citrix Virtual Apps and Desktops on a pure IPv6 Internet Provider with enabled EDT
Some weeks ago first Incidents reached my team with the problem description that Users having a problem accessing their pooled Windows 10 desktops trough Citrix ADC (aka Netscaler).
I noticed that all affected users have one thing in common, they all have UPC Cablecom as an Internet Provider. As soon they connect their device to another internet connection, for instance to the Wifi hotspot created with their mobile phone all works fine.
Further troubleshooting steps showed that if I publish a dedicated VDI with disabled EDT also all works fine. If you don’t know what EDT is, Google EDT and Citrix 🙂
Usually, EDT is enabled and uses UDP for communication if the port is open and it’s possible. Otherwise, it should fall back to TCP. It seems to fall back just works if the ports are closed.
Now my problem is, I can’t disable EDT for all pooled VDI as we have some users who really need it. I also won’t give all the users dedicated desktops.
Luckily Julian Jakob gave me the hint to disable EDT on client-side. This is the best workaround.
Guidance for the User
(Because of BYOD the user needs to do that … )
On Windows it’s just a Registry key which the users have to create:
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Citrix\ICA Client\Engine\Lockdown Profiles\All Regions\Lockdown\Network\UDT]
"HDXoverUDP"="Off"
On a Mac this command should be typed in the terminal:
defaults write com.citrix.receiver.nomas HDXOverUDPAllowed -bool NO
On iOS devices you can disable EDT in the Workpace App:
Settings => Advanced => Adaptive Transport Settings => EDT (set on inactive)
Please Citrix fix that! All that I want is a fall back to TCP if UDP is not working, I don’t care if the port is closed or if it’s another issue. If EDT doesn’t work don’t use it. For users, this manual config on the client-side is not just annoying, some are not able to do it.
Update 26.5.2020
- If you have a non-standard MTU, see this article: https://support.citrix.com/article/CTX231821
- Background why this is happening, it seems UPC uses CarrierGradeNAT * and its a problem with IPv4/IPv6 Encapsulation, at the end there is too little payload available (MTU issue). Kudos to Markus Löffler for this hint and the link.
- Citrix has introduced in the background a feature called MTU Discovery. It’s not yet officially announced but it’s in the documentation since 1912: https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/technical-overview/hdx/adaptive-transport.html#edt-mtu-discovery maybe this also can help solving this issue. I was not yet able to test.
* Explanation in German: https://www.elektronik-kompendium.de/sites/net/2010221.htm
Update 4.6.2020
I’ve enabled MTUDiscovery and it really looks promising, please be aware that MTU Discovery currently is just for the WorkspaceApp for Windows available. (CWA 19.12 an newer)
Fatal error during installation (1603) on StoreFront upgrade to 3.12
Today I’ve upgraded Citrix StoreFront Server from 3.9 to 3.12
as every time first I shut down all of the following services:
net stop W3SVC
net stop CitrixConfigurationReplication
net stop CitrixCredentialWallet
net stop CitrixDefaultDomainService
net stop “Citrix Subscriptions Store”
net stop “Citrix Peer Resolution Service”
net stop CitrixServiceMonitor
net stop CitrixTelemetryService
then I run CitrixStoreFront-x64.msi, reboot the server and after that I do that on the second node. Most time this goes pretty flawless.
But this time, my upgrade failed with an error:
CitrixStoreFront-x64.msi’ failed with error code 1603. Fatal error during installation”
I remember I had this already one time before, but what the hell was the solution … a short search with Google showed me:
https://discussions.citrix.com/topic/371535-storefront-upgrade-to-301-from-300-fails
Well I’m on StoreFront 3.9 and when I have a look into “C:\Program Files\Citrix\Receiver StoreFront\Services\ProtocolTransitionService\Citrix.DeliveryServices.ProtocolTransition.ServiceHost.exe.config” I see in some lines “Version=3.8.0.0” – but I have 3.9, so I replace all “Version=3.8.0.0” to “Version=3.9.0.0”
Result: StoreFront upgrade to 3.12 is successful – All’s well that ends well.
No more able to start SOAP on PVS
After the last monthly Microsoft Security Updates one of my PVS Servers was no more able to start the SOAP service. I received an Event 7000 with the message:
The Citrix PVS Soap Server service failed to start due to the following error: The service did not respond the the start or control request in a timely fashion.
I live in Bern, and we are known as slow-paced people here in Bern, probably because of our slow sounding accent. So my idea is if the service need more time to start, I’ll give him more time.
I’ve created a new DWORD called ServicesPipeTimeout with the value 120000 in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control which means the service has 2 minutes time to start. After a reboot my SOAP was again up and running.
By the way and additional tip according this service… SOAP is sometimes bitchy … it’s a good idea to set the service to auto restart after a crash.
Avoid blanks and dots in StoreFront 3.5 farm names!
Last day’s I upgraded my existing Storefront 3.01 to StoreFront 3.5 and in some farms I had a very bad issue after the upgrade, it seemed that all is ok but users are no more able to start any application or desktop.
On the StoreFront server I saw an Warning Event 28 from Citrix Store Service “ Failed to launch the resource “Farm Name.ApplicationName” as it was not found.
The issue was that a blank and a dot (.) was in my Farmname, seems this is a bug in SF3.5 that you can configure that but It won’t work!
after I changed the display name to something like xa65farm without blank and point all works well.
Client to Server Content Redirection no more working because of License Error
Recently I faced a really strange error with a weird solution. Because I didn’t found this on the web I post this here, maybe it can help someone else too.
We had the issue that users cannot use anymore the Client to Server Content Redirection, means they can’t double-click anymore on *.vsd files to open the file with the XenApp server installed MS Visio.
After some minutes of investigation and checking the Logs on Client, XenApp server and so on we’ve finally checked the EventLog on the Webinterface server and found the following error:
Event 31007
Site path: C:\inetpub\wwwroot\Citrix\PNAgent. The Citrix servers are not licensed to support workspace control. This message was reported from the XML Service at address http://myserver.ch:8080/scripts/wpnbr.dll [com.citrix.xml.NFuseProtocol.RequestReconnectSessionData]. [Unique Log ID: a5e760c4] For specific information about this message, see the Web Interface documentation at http://support.citrix.com/proddocs/topic/web-interface-impington/wi-log-messages-event-ids-hardwick.html.
Now we remembered that we consolidated some Citrix license server’s and changed the license server for this farm some days ago. Of course we rebooted all XenApp servers, but not the Webinterface servers because they don’t use a special license server. A reboot of the Citrix Webinterface server solved this issue.
“Cannot complete your request‘” on Netscaler Gateway VPX
In my lab environment I was using a Citrix Webinterface 5.x which was accessible from Internet over a Access Gateway 5 VPX. Since Citrix Store Front is in a fairly usable release (> Version 2.x), I intended to update my lab environment to the current software releases and update my skills to Store Front and Netscaler Gateway VPX.
You can find a step by step Netscaler Gateway intro here http://blogs.citrix.com/2013/07/03/citrix-netscaler-gateway-10-1-118-7-quick-configuration-wizard
Also a very nice guide you can find here, this guide also contains information about how to configure StoreFront for Netscaler Gateway VPX: http://benjamin.eavey.com/2013/07/netscaler-vpx-as-secure-gateway-replacement
Cannot complete your request
After completion of the configuration I was not able to access the my environment from outside. The login to the Netscaler Gateway, the black window, was working fine, but as soon I hit the StoreFront I get this Error:
Because StoreFront is working fine from internal, I assumed that’s not a completely wrong StoreFront configuration. After i had a look into the event viewer on the StoreFront server I can see that something is wrong here:
The crucial indication that’s a problem between the Store Front server and the the Netscaler Gateway in role of Authentication Callback Server I found here:
when I browse to the address https://192.168.x.x/CitrixAuthService/AuthService.asmx you can see a certificate error, so I need to have here a FQDN that match to the installed certificate but I wont communicate outside, so first I’ve defined the internal IP as Callback URL:
Now I’ve changed the Callback URL to the FQDN appropriate to the certificate:
But because the DNS resolve this URL as the external IP which is not accessible over the necessary TCP ports, I was constrained to do a dirty hack … I have edited my host file :
Issue by creating a PVS 7.1 farm
If you try to create a new Farm and the Provisioning Services Configuration Wizard stuck on a “Not responding” during the Database Server step maybe you have to many databases on the Server:
Workaround: Use the DbScript.exe to get the Database creation script and create the database with this script directly on the DB server:
Citrix DSCHECK doesen’t work … “No resource module ImaMsgsUI.dll found.”
When you try to make a Citrix Datastore validaiton with DSCHECK you receive the error message :
” No resource module ImaMsgsUI.dll found. “
Cause:
Probably you are on a Citrix server with UAC,
Solution:
Launch the command line with administrative privileges, “run as administrator” before you make the dscheck.
more details about dscheck.exe here: http://support.citrix.com/article/CTX124406
