On a day-to-day basis I assist with the operations of a Citrix environment with 100+ individual XenDesktop sites (small offices). With Director, only a single site is visible at a time. I would have to select each site individually to find out if there are any failure events at a location. For 100+ sites this would be extremely tedious and time consuming. Wouldn’t it be great if there was a way to look through all unique sites and find out if there’s a failure? Heck yeah it would!
Our Solution
What I did was create a Python script that does just that. The script consumes a text file with a list of all of the XML Brokers and asks each broker “What is your current your failure count?”. If the failure count is greater than 0 (zero), which means there’s a failure, then open up IE and navigate to that site for further investigation. The way the script is written, it will iterate and re-iterate through the list until the script is manually stopped. This way I can leave it running all day and if there’s an issue IE will pop up prompting me to login, and pause for 30 seconds.
If no issues are found at the queried XML Broker, then the script will wait 5 seconds and move to the next XML Broker in the list. If you need to manually stop the script, then use CTRL+C or just close the window where the application is being executed. When an error is found, then I can logon on to the Director server and begin troubleshooting.
Each query is targeted at the URL “/Citrix/Monitor/OData/v2/Data/FailureLogSummaries”. The first XML tag is what the script is looking for because it contains the current failure count. For documentation on what information is contained in this field, please visit eDocs http://support.citrix.com/proddocs/topic/xendesktop-7/cds-ms-odata-wrapper.html.
The location of the text file for me is “D:\temp\ddcFile.txt”. You may modify the variable “ddcFile” to your specific file location. The file lists DDC as such.
ddc1.mycompany.net
ddc2.mycompany.net
ddc3.mycompany.net
Here is the Python code is below.
import requests
import time
import xml.etree.ElementTree as ET
import requests.auth
from requests_ntlm import HttpNtlmAuth
import getpass
import webbrowser
#use this for username\password
username = raw_input("Enter Domain\\Username :")
password = getpass.getpass("Enter Password :")
#xml namespaces
ns = {'default': "http://www.w3.org/2005/Atom",
'base': "http://192.168.0.112/Citrix/Monitor/OData/v2/Data/",
'd': "http://schemas.microsoft.com/ado/2007/08/dataservices",
'm': "http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"}
#Used for automatically launching IE when a failure is detected.
ie = webbrowser.get(webbrowser.iexplore)
class DdcMonitor:
def main(self):
while True:
#opens the file with the list of DDCs
ddcFile = open('D:\\temp\\ddcFile.txt', 'r')
for ddcFQDN in ddcFile:
#uses HTTP. HTTPS could be added if needed.
directorURL = "http://" + ddcFQDN.rstrip("\n") + "/Citrix/Monitor/OData/v2/Data/FailureLogSummaries"
print("Now probing : " + str(directorURL))
#Connection information
#here is an example of a constructed query
#directorURL = "http://192.168.0.112/Citrix/Monitor/OData/v2/Data/FailureLogSummaries"
directorSession = requests.session()
directorSession.auth = HttpNtlmAuth(username,password)
directorReqData = directorSession.get(directorURL)
#XML information
root = ET.fromstring(directorReqData._content)
entry = root.find('default:entry', ns)
sub_1 = entry.find("default:content", ns)
for sub_2 in sub_1.find("m:properties", ns):
if "FailureCount" in str(sub_2.tag):
if int(sub_2.text) > 0:
print("")
print("The Failure Count is increasing at " + directorURL)
print("The error count is currently : " + sub_2.text)
print("Waiting 30 seconds")
ie.open('http://' + ddcFQDN + "/director")
print("")
time.sleep(30)
else:
print("The Failure Count is not increasing at " + directorURL)
print("The probe will run again in 5 seconds")
print("")
print("")
time.sleep(5)
ddcFile.close()
time.sleep(1)
try:
DdcMonitor().main()
print("the program is no longer running")
except:
print("Something caused the program to stop. Please restart the program")
Is there any information that you would like to monitor from your Citrix deployment on an hourly, daily, or weekly basis? Let me know in the comments below.
Thanks for reading!
BC
This software / sample code is provided to you “AS IS” with no representations, warranties or conditions of any kind. You may use, modify and distribute it at your own risk.
Abstract
The purpose of this blog is to assist with troubleshooting the “Internal Error” error when installing or replacing certificate key pairs.
Let’s dive in
No matter how many times you click the “install” button the certificate key pair just will not install. Sound familiar? I’ve encounters this issue, many, many, many times. The root cause is always one of the following:
The certificate or private key are formatted improperly.
The certificate does not correspond to the correct private key.
Let’s go into each of these.
1. The certificate or private key are formatted improperly.
Sometimes extra spaces can throw off the import of a certificate/key pair. Personally, I try to put all certificates into a PEM format. NetScaler OpenSSL can be used to accomplish this. For keys use the following command. Use the “–inform DER” option for DER encoded keys.
Openssl rsa –in -out
Along the same lines, use the following command to reformat the certificate file:
Openssl x509 –in -out
Try installing the certificate/key pair again. If that doesn’t work then…
2. The certificate does not correspond to the correct private key.
I encounter this much more than people would think. If the certificate was generate from a CSR that was not signed by the correct private key, then the certificate will never install. Ever. How can we check if the certificate was generated from a CSR that was signed by the correct key? By checking the modulus of the private key, CSR, and certificate of course :-)! The modulus of the private key and certificate must be 100% exact matches. For an explanation as to why this is the case, please seehttps://en.wikipedia.org/wiki/RSA_(cryptosystem).
To view the modulus of a private key, use the following command.
Openssl rsa – modulus –noout –in
To view the modulus of a csr:
Openssl req – modulus –noout –in
To view the modulus of a certificate:
Openssl x509 –modulus –noout –in
Below is a sample output for what the modulus will look like for my public certificate.
FIPS NetScaler Appliances
So what happens if you can’t access your private key to run OpenSSL commands (FIPS NetScaler appliances)? Remember that the Key, CSR, and Cert MUST all use the same modulus if they are related. With this theory in mind, you can generate a bogus CSR off of the FIPS key to see what the modulus should be for the public certificate. If you generate a CSR off of a FIPS box and the modulus for that CSR does not match the modulus for the Public Certificate that was returned to you, then that certificate did not use a CSR that was generated off of that FIPS key.
I hope this information has helped. The OpenSSL commands I have listed are only a handful of my favorites. For a really good cheat sheet on useful commands (I have it bookmarked), check out the link below.
https://www.sslshopper.com/article-most-common-openssl-commands.html
Let me know if you have questions in the comments below!
BC
Before we begin let me first say…
All NetScaler Gateway landing page customizations are unsupported. Changing the NetScaler Gateway landing page will cause you to have an unsupported environment. I do not condone malicious attempts to lockout user accounts. The purpose of this article is to highlight a current risk and mitigation steps.
Now that the disclaimer is out of the way. Let’s start with the customizations :-).
The current recommended configuration for two-factor authentication at NetScaler is available here.
http://support.citrix.com/article/CTX125364
With the configuration highlighted in the article above. Web based users that authenticate are hitting AD first. Ideally, we would want to follow the authentication workflow that is configured for the Native Receiver. The Native Receiver evaluates RADIUS first, and if this is successful, then the LDAP policy is invoked.
What is the risk of leaving the configuration exactly how the article has outlined the configuration?
If Bob, a malicious user, knows Alice’s username, then Bob could enter a bogus password 3 times and lock Alice’s account. Bob could do this as often as he liked until some measure went into place to stop Bob. If Bob knew a lot of usernames and had some knowledge of scripting tools such as JMeter, then he could lockout a large number of user accounts effectively acting as a DOS. This would be bad, and I again, I would not condone such an attack. So what can we do to mitigate such a risk?
The quick and easy way to do it is to reverse the web authentication policies so that they match up with the Native Receiver (RADIUS as primary, LDAP as secondary). However, this will force users to enter their RADIUS passcode before entering their AD username. Most organizations want to have the dynamic pin as the 2nd password for users to enter.
So how can we mitigate the risk AND have the dynamic token as the second password users need to enter?
Like in the quick and easy method, we would need to make the RADIUS authentication primary and the LDAP authentication secondary. Now we need to customize some JavaScript on the NetScaler. The file /vpn/login.js is what we need to customize. This file can be found under“/netscaler/ns_gui/vpn/login.js”. What we will do is change the ordering of the POST values.
The JavaScript below has the original values in red that we will change.
function ns_showpwd_default()
{
var pwc = ns_getcookie(“pwcount”);
document.write(‘
’);
}
UnsetCookie(“pwcount”);
}
The JavaScript below contains the revised fields so that when a user POSTs their credentials, NetScaler will can evaluate RADIUS before attempting to contact AD. The values passwd1 andpasswd are swapped.
function ns_showpwd_default()
{
var pwc = ns_getcookie(“pwcount”);
document.write(‘
The purpose of this post is to show how users can be educated to use a single URL, while still using having a StoreFront base URL that is different from the NetScaler Gateway URL.
Please keep in mind this solution works best for Receiver for Web. This solution does work with the Native Receiver, but the Provisioning file would be the easiest way to configure the Native Receiver in my opinion.
For this scenario, I will use connect.example.com for external access to the Citrix environment. Int-connect.example.com will be used for internal access to the Citrix environment. Below are overview of the requirements for the scenario to get us started.
1. SAN certificate for int-connect.example.com and connect.example.com.
2. Connect.example.com will resolve to the publicly accessible NetScaler Gateway VIPs.
3. Int-connect.example.com will resolve to the internal StoreFront Load Balanced VIPs.
4. CNAME on the internal DNS. connect.example.com –> int-connect.example.com.
5. Responder Policy to redirect from int-connect.example.com to connect.example.com.
Now for the magic of creating the single FQDN that users need to know. In this example, the “single URL” for users is connect.example.com. On the internal DNS infrastructure, create a CNAME for connect.example.com to point to int-connect.example.com. Then, on the NetScaler appliance, create a Responder Policy that redirects traffic with the HTTP Hosts header of “connect.example.com” to “int-connect.example.com”. Bind this policy to the StoreFront LB VIP on NetScaler.
So what is the expected user behavior?
A user on the internal network types connect.example.com into their browser. Connect.example.com resolves as a CNAME for int-connect.example.com. The user will resolve int-connect.example.com. After obtaining the IP address for int-connect.example.com, the user connects to the SF LB VIP using the IP address and the HTTP host header connect.example.com. The Responder policy redirects the user to int-connect.example.com. The user’s browser follows the redirect and is able to access the StoreFront LB VIP. By using a SAN certificate with the names we need, the user will not receive a certificate warning.
The workflow above is all seamless to the user. From their perspective, they type connect.example.com, and that takes them to the resources they need to focus on their job.
Please keep in mind that this workflow is unique to Receiver for Web. Users that manually configure Receiver on the internal network will need to type out “int-connect.example.com” to connect to the StoreFront VIP and avoid a redirect. Again, I recommend using the provisioning file from StoreFront to configure the Native Receiver.
Let me know if you have questions in the comments below!
BC
If no issues are found at the queried XML Broker, then the script will wait 5 seconds and move to the next XML Broker in the list. If you need to manually stop the script, then use CTRL+C or just close the window where the application is being executed. When an error is found, then I can logon on to the Director server and begin troubleshooting.
Each query is targeted at the URL “/Citrix/Monitor/OData/v2/Data/FailureLogSummaries”. The first 
FIPS NetScaler Appliances
So what happens if you can’t access your private key to run OpenSSL commands (FIPS NetScaler appliances)? Remember that the Key, CSR, and Cert MUST all use the same modulus if they are related. With this theory in mind, you can generate a bogus CSR off of the FIPS key to see what the modulus should be for the public certificate. If you generate a CSR off of a FIPS box and the modulus for that CSR does not match the modulus for the Public Certificate that was returned to you, then that certificate did not use a CSR that was generated off of that FIPS key.
I hope this information has helped. The OpenSSL commands I have listed are only a handful of my favorites. For a really good cheat sheet on useful commands (I have it bookmarked), check out the link below.
The workflow above is all seamless to the user. From their perspective, they type connect.example.com, and that takes them to the resources they need to focus on their job.
Please keep in mind that this workflow is unique to Receiver for Web. Users that manually configure Receiver on the internal network will need to type out “int-connect.example.com” to connect to the StoreFront VIP and avoid a redirect. Again, I recommend using the provisioning file from StoreFront to configure the Native Receiver.
Let me know if you have questions in the comments below!
BC