Saturday, September 15, 2018

Troubleshooting TCP Resets (RSTs)

Inconsistent issues are by far the most difficult to track down. Network inconsistencies are particularly problematic because there can often be many different devices that must be looked into in order to identify the root cause. The following troubleshooting goes through a couple of steps. The first part is to start a tcpdump process that will record TCP RSTs. Then you can send a lot of HTTP requests. Below is the command to issue the tcpdump and fork the process to the background. However, the output will still be sent to the active terminal session because of the trailing &.

sudo tcpdump -i any -n -c 9999 -v 'tcp[tcpflags] & (tcp-rst) != 0 and host www.somesite.com' &

Below is the command to issue lots of HTTP requests. The important part to understand about the below command is to go through the TCP build up and tear down that happens during the HTTP request process.

for i in {1..10000}; do curl -ks https://www.somesite.com/robots.txt > /dev/null ; done

Below is an example of what a potential output could be.

17:16:56.916510 IP (tos 0x0, ttl 62, id 53247, offset 0, flags [none], proto TCP (6), length 40)
10.1.1.1.443 > 192.168.5.5.41015: Flags [R], cksum 0x56b8 (correct), seq 3221469453, win 4425, length 0
17:17:19.683782 IP (tos 0x0, ttl 252, id 59425, offset 0, flags [DF], proto TCP (6), length 101)
10.1.1.1.443 > 192.168.5.5.41015: Flags [R.], cksum 0x564b (correct), seq 3221469453:3221469514, ack 424160941, win 0, length 61 [RST+ BIG-IP: [0x2409a71:704] Flow e]
17:18:54.484701 IP (tos 0x0, ttl 62, id 53247, offset 0, flags [none], proto TCP (6), length 40)
10.1.1.1.443 > 192.168.5.5.41127: Flags [R], cksum 0x46f7 (correct), seq 4198665759, win 4425, length 0

While it may be unclear exactly why the TCP RSTs are happening this does provide a mechanism to reproduce TCP RSTs behaviors to investigate on other devices in the Network traffic flow. Below is documentation on how to troubleshoot TCP RSTs for the F5. 

https://support.f5.com/csp/article/K13223

 Happy troubleshooting!

Grabbing AWS CloudFront IPs with curl and jq

There's times when you want to restrict access to your infrastructure behind CloudFront so that requests must go through the CloudFront CDN instead of your origin directly. Fortunately, AWS lists their public IP ranges in a JSON format in the following link, https://ip-ranges.amazonaws.com/ip-ranges.json. However, there are a lot of services in the above link and it would be very tedious to take the entire JSON and read through it to grab specific CloudFront IP's. Using the combination of command line tools curl and jq we can easily grab just the CloudFront IP ranges to lock down whatever origin that exists. Below is the command that I've used to grab just the CloudFront IP's. Enjoy!

curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes | .[] | select(.service == "CLOUDFRONT") | .ip_prefix'

BC

Decrypt all PFX files in a directory

I recently received a whole bunch of different PFXs where I needed to decrypt the files, extract the keys, and extract the server certificate. Below is Bash script to do just that. Replace the bolded somepass with the real password used to decrypt the PFX and execute the script in the directory with all of the PFX files. Note, the script would only work if the PFX's all have the same password. Enjoy!

for f in *.pfx; 
do 
 pemout="${f}.pem"; 
 keyout="${pemout}.key";
 crtout="${pemout}.crt";
 openssl pkcs12 -in $f -out $pemout -nodes -password pass:somepass; 
 openssl rsa -in $pemout -out $keyout;
 openssl x509 -in $pemout -out $crtout;
done

BC

Demystifying the NGINX Real IP Module

The Real IP module within NGINX is very strict. The purpose of this post is to go over how the NGINX's real_ip_from works by walking through a few examples. Below is the official NGINX document. http://nginx.org/en/docs/http/ngx_http_realip_module.html

Example 1

NGINX configuration.
 set_real_ip_from 0.0.0.0/0 ;
 real_ip_recursive on ;
 real_ip_header x-forwarded-for ;

Source and Destination IP
 src IP = 10.0.0.2
 dst IP = 10.0.0.3

Request
 GET /someurl.html HTTP/1.1
 host: brookscunningham.com
 X-Forwarded-For: 1.1.1.1, 2.2.2.2, 3.3.3.3, 4.4.4.4

The IP 1.1.1.1 would be utilized for the real client IP based on the above request.

Example 2

NGINX configuration. Same configuration as above. Source and Destination IP
 src IP = 10.0.0.2
 dst IP = 10.0.0.3

Request
 GET /someurl.html HTTP/1.1
 host: brookscunnningham.com

The X-Forwarded-For header is missing. NGINX will utilize the layer 3 source IP as the client IP. In this case NGINX will utilize the IP 10.0.0.2 as the real IP.

Example 3

Now lets get tricky and lock down the Real IP Module to a subset of IP's. NGINX Config
set_real_ip_from 10.0.0.0/8 ;
real_ip_recursive on ;
real_ip_header x-forwarded-for ;

Source and Destination IP
src IP = 10.0.0.2
dst IP = 10.0.0.3

HTTP Request
 GET /someurl.html HTTP/1.1
 host: brookscunningham.com
 X-Forwarded-For: 1.1.1.1, 2.2.2.2, 3.3.3.3, 4.4.4.4

NGINX would use the IP 4.4.4.4 as the real client IP in the above request. The reason for this is that NGINX will trust the last IP in the chain of trusted IP's in the designated real IP header.

Example 4

NGINX Config
set_real_ip_from 10.0.0.0/8 ;
set_real_ip_from 4.4.4.4 ;
real_ip_recursive on ;
real_ip_header x-forwarded-for ;

Source and Destination IP
src IP = 10.0.0.2
dst IP = 10.0.0.3

HTTP Request
 GET /someurl.html HTTP/1.1
 host: brookscunningham.com
 X-Forwarded-For: 1.1.1.1, 2.2.2.2, 3.3.3.3, 4.4.4.4

NGINX would use the IP 3.3.3.3 as the real IP since that is 10.0.0.0/8 is trusted and 4.4.4.4 is the last IP in the chain of trusted IP's.

Example 5

NGINX Config
set_real_ip_from 10.0.0.0/8 ;
set_real_ip_from 4.4.4.4 ;
real_ip_recursive off ;
real_ip_header x-forwarded-for ;
Source and Destination IP
src IP = 10.0.0.2
dst IP = 10.0.0.3

HTTP Request
 GET /someurl.html HTTP/1.1
 host: brookscunningham.com
 X-Forwarded-For: 1.1.1.1, 2.2.2.2, 3.3.3.3, 4.4.4.4

NGINX would use the IP 4.4.4.4 as the real IP since the real_ip_recursive is set to off. Only the last IP in the chain of X-Forwarded-For would be utilized for the client IP.  

Example 6 - Internet IPs as source IP

NGINX Config
set_real_ip_from 10.0.0.0/8 ;
set_real_ip_from 4.4.4.4 ;
real_ip_recursive off ;
real_ip_header x-forwarded-for ;

Source and Destination IP
src IP = 55.55.55.55
dst IP = 10.0.0.3

HTTP Request
 GET /someurl.html HTTP/1.1
 host: brookscunningham.com
 X-Forwarded-For: 1.1.1.1, 2.2.2.2, 3.3.3.3, 4.4.4.4

NGINX would 55.55.55.55 as the real client IP. The reason for this is because the source IP address is not defined as trusted within the set_real_ip_from.

Example 7

NGINX Config
set_real_ip_from 10.0.0.0/8 ;
set_real_ip_from 4.4.4.4 ;
set_real_ip_from 55.55.55.55 ;
real_ip_recursive on ;
real_ip_header x-forwarded-for ;

Source and Destination IP
src IP = 55.55.55.55
dst IP = 10.0.0.3

HTTP Request
 GET /someurl.html HTTP/1.1
 host: brookscunningham.com
 X-Forwarded-For: 1.1.1.1, 2.2.2.2, 3.3.3.3, 4.4.4.4

NGINX would 3.3.3.3 as the real client IP. The reason for this is because real_ip_recursive is set to on and the source IP address is now defined as trusted within the set_real_ip_from up to 4.4.4.4.