The past was easy peasy, not the case anymore these days. HTTPS was rarely used and only where it really needed to be. Anyone on the wire could see what’s going on. And yeah, it meant literally everyone.
Was it good? No, not at all. Everyone could track anyone, collect information, behavior, etc. Result? Everyone started to move towards HTTPS (ssl then tls). This basically means couple of good and bad outcomes. Squid cache ration went down, and I mean very, very low these days. Most connections handled by Squid these days are tunnels (CONNECT) and anything can be sent through such tunnel, not only private data, credit card numbers, etc. but also ads and virus (sic!).
This created specific need to be able to check the content of the stream, especially in enterprise environment. All the DLP systems can only work on streams if they can check the payload, decrypted traffic which means they need to be so well known man-in-the-middle. This means to be tunnel end-point from client side and initiation of new tunnel to server. Only this allows to inspect the traffic.
At the same time one could ask, hey how about certificates? The long winded answer can be found elsewhere, but short answer is that in enterprise environment there is at least one Root Certificate Authority (CA) and Intermediate Certificate Authority can be created. The root or intermediate CA (called CA afterwards) together with key file is uploaded to the DLP system allowing it to generate new certificates on the fly for terminated tunnels.
The CA is already trusted in enterprise environment as Root CA certificate is added to Trusted CA key ring on client host as part of operating system deployment package or domain connection.
Since client trusts Root CA it automatically trusts certificates signed by the given CA – this is how Public Key Infrastructure works. Additionally the DLP system usually tries to mimic all original certificate parameters and only CA related details are different. But people rarely check details of certificate if everything is in green and no popup/error is raised.
private DLP
With all above being said one can have their own DLP system based on Squid. This “little piece of software” is great in handling huge loads, caching data and calling others for help. This is what we’re going to do today.
Squid to terminate SSL connection uses
ssl_bump functionality. The Ubunty 16.04 LTS default package does not allow to use this great functionality hence we need to start with little configuration.
Let’s get sources and all needed libraries (for all below any sudo call is skipped as it just makes the output longer and you, dear reader certainly know how to use sudo).
|
apt-get source squid apt-get build-dep squid gnutls-bin devscripts build-essential fakeroot |
then we need to apply patch to enable ssl as needed
|
--- build/squid3/squid3-3.5.15/debian/rules 2016-02-17 01:13:33.000000000 +0100 +++ build/squid3/squid3-3.5.15/debian/rules.new 2016-02-22 22:50:04.079470555 +0100 @@ -45,7 +45,10 @@ --with-pidfile=/var/run/squid.pid \ --with-filedescriptors=65536 \ --with-large-files \ - --with-default-user=proxy + --with-default-user=proxy \ + --with-openssl \ + --enable-ssl \ + --enable-ssl-crtd BUILDINFO := $(shell lsb_release -si 2>/dev/null) |
To apply patch, use your standard methodology
patch -p<level> < diff-file.patch
Afterward squid package and any missing packages need to be installed
|
dpkg -i squid-package-version-#-.deb squid-common-package-version-#.deb apt-get install -f # and prevent squid from being upgraded apt-mark hold squid3 squid3-common |
Certificate generation
First of all we need to have folder where we will store it all
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
|
mkdir /etc/squid/CA-certificates cd /etc/squid/CA-certificates #Create the cnf configuration file x509v3ca.cnf cat > /etc/squid/CA-certificates/x509v3ca.cnf << EOF [ req ] default_bits = 4096 default_md = sha1 default_keyfile = rootca.key distinguished_name = req_distinguished_name x509_extensions = v3_ca string_mask = nombstr [ req_distinguished_name ] [ v3_ca ] basicConstraints = critical,CA:true nsCertType = critical,sslCA extendedKeyUsage = critical,serverAuth,clientAuth,emailProtection,timeStamping,msCodeInd,msCodeCom,msCTLSign,msSGC,msEFS,nsSGC keyUsage = keyCertSign,cRLSign subjectKeyIdentifier = hash EOF |
The certificates mea:
|
#Generate RSA CA private key: openssl genrsa -out ./rootca.key 4096 #Create CA certificate openssl req -new -nodes -x509 -sha1 -out ./rootca.crt -key ./rootca.key -config ../x509v3ca.cnf -extensions v3_ca -subj '/O=High Security DKLP/OU=DKLP RootCA/CN=DKLP/' -days 9999 #The certificate in PEM format. Use this to distribute to most non-Windows platforms. e.g. *nix systems. openssl x509 -in ./rootca.crt -out ./rootca.pem -outform PEM #The certificate in pkcs12 format. Use this to distribute to Windows platforms. openssl pkcs12 -export -in ./rootca.crt -inkey ./rootca.key -out ./rootca.p12 -name "RootCA" -password pass: #The certificate in DER binary format format. Use this to distribute to Android Devices. openssl x509 -inform PEM -outform DER -in ./rootca.crt -out /./rootca.crt |
Depending on client system certificate import will look differently. The good idea might be to place it on some easily accessible server, i.e. local wpad system.
Once certificate is downloaded it should be installed. On windows this can be done with:
|
certutil -addstore -enterprise -f "Root" rootca.pem Root Signature matches Public Key Certificate "CN=DKLP" added to store. CertUtil: -addstore command completed successfully. |
Some apps do not respect Operating System level certificates, but most will. Some apps might need to be restarted it shouldn’t be required to restart full system, but who knows what type of ancient system one can be using?
All the new certificates generated on the fly need to be stored somewhere. The folder needs to be created:
|
mkdir /mnt/sda6/Squid_ssldb chown proxy:proxy /mnt/sda6/Squid_ssldb |
And update
squid.conf (not all SSL cert ERROR related flags should be as below, tune it up to your needs).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42
|
############################################################################ # https proxy #sslproxy_foreign_intermediate_certs /etc/squid/extra-intermediate-CA.pem acl step1 at_step SslBump1 acl step2 at_step SslBump2 acl step3 at_step SslBump3 acl sslBumpnet src "/etc/squid/ssl_bump/sslBumpnet" acl sslnoBumpnet src "/etc/squid/ssl_bump/sslnoBumpnet" acl sslnoBumpnetdst dst "/etc/squid/ssl_bump/sslnoBumpnetdst" acl sslnoBumpSites ssl::server_name "/etc/squid/ssl_bump/sslnoBumpSites" #ssl_bump client-first ssl_bump peek step1 ssl_bump splice sslnoBumpnetdst ssl_bump splice sslnoBumpSites ssl_bump splice sslnoBumpnet !sslBumpnet ssl_bump bump all # The default port with ssl-bump (this is the one used in client proxy settings) http_port 3127 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_bump/certs/rootca.crt key=/etc/squid/ssl_bump/certs/rootca.key # Let's silently intercept connections (including SSL) for stubborn customers https_port 3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_bump/certs/rootca.crt key=/etc/squid/ssl_bump/certs/rootca.key # self signed # comment out the sslproxy_flags acl SSLERR ssl_error X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN acl BrokenButTrustedServers2 dstdomain "/etc/squid/ssl_bump/dstdom2.broken" acl UnableGetIssuer ssl_error X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE sslproxy_cert_error allow BrokenButTrustedServers2 UnableGetIssuer sslproxy_cert_error allow SSLERR #sslproxy_cert_error deny all sslproxy_flags DONT_VERIFY_PEER sslproxy_cert_error allow all sslproxy_options NO_SSLv2,NO_SSLv3,SINGLE_DH_USE sslproxy_cipher EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS # Cert generator sslcrtd_program /usr/lib/squid/ssl_crtd -s /mnt/sda6/Squid_ssldb -M 4MB sslcrtd_children 20 startup=10 idle=5 |
Create set of files under /etc/squid/ssl_bump:
- sslBumpnet – subnets/hosts which we will bump, can be selective if needed
- sslnoBumpnet – these subnets/hosts we won’t bump (see logical construction above and tune it to your needs
- sslnoBumpnetdst – these domains/servers we won’t bump
- sslnoBumpSites
Example sslnoBumpnetdst as it might be tricky to set it up at the beginning. Some apps have built-in certificates and verify connection against them, i.e. Google Play store and couple of others.
Interesting was to find out what banks are doing with data and identify as in one case at least that for stats reasons were sending out GET request with bank account details, including saldo and transaction details to on-line stats agency. This was against any bank standards and bank should be prosecuted due to this. This was only possible after terminating tunnels as otherwise GET wasn’t visible. This can only be acceptable for tests and for your own private use at home in the isolated lab (as always).
Below list was fine tuned based on tests and was longer earlier.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
|
## # .banks Hosts/domains ## connectivitycheck.gstatic.com play.google.com .googleusercontent.com play.googleapis.com .sync.app.asana.com ###### # company SSL # Juniper Pulse will require this #.company domain .com ssl.company.com.external.domain ###### ###### # Company Skype # seen but down to domain # sippoolbn11a05.infra.lync.com # webpoolbn11a05.infra.lync.com # lyncdiscoverinternal.acme.com # lyncdiscover.acme.com # sip.acme.com # pipe.skype.com - not needed #sip.acme.com #lyncdiscover.acme.com #lyncdiscoverinternal.acme.com #.online.lync.com .infra.lync.com #.lync.com #.skype.com ###### ###### # WU (Squid 3.5.x and above with SSL Bump) # Only this sites must be spliced. .update.microsoft.com .update.microsoft.com.akadns.net ###### |
The last point is to restart squid
To test it, select port 3127 as proxy. Once all tested transparent interception (don’t forget to have CA trusted on client side).
|
IPT=/sbin/iptables INS=inside # your internal interface (can be result of ip addr ls... function SQUID=1.2.3.4 # your squid IP address SQUIDP=3128 # squid intercept port SQUIDPSSL=3129 # squid intercept port ssl $IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 80 -j LOG --log-level info --log-prefix "IP PROXY-redirect: DNAT " $IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 80 -j DNAT --to-destination $SQUID:$SQUIDP $IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 443 -j LOG --log-level info --log-prefix "IP PROXY-redirect: DNAT " $IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 443 -j DNAT --to-destination $SQUID:$SQUIDPSSL |
For non-Android based systems proxy.pac/wpad.dat file could be created
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
|
function FindProxyForURL(url, host) { if (shExpMatch(url,"*://192.168.*/*")) return "DIRECT"; else if (shExpMatch(url,"*://172.16.*.*/*")) return "DIRECT"; else if (shExpMatch(url,"*://127.0.0.1/*")) return "DIRECT"; else if (shExpMatch(url,"*://localhost/*")) return "DIRECT"; else if (shExpMatch(url,"*.our.domain/*")) return "DIRECT"; else { return "PROXY our.proxy:3127"; } } |
To get that working the wpad.yourdomain and wpad host needs to be resolved to your web server and thw wpad.dat/proxy.pac file needs to be in root folder.
Android platform is prone to not process this file (at least as for April 2017 and manual proxy settings in WiFi section need to be set.
This should all work… but hey, problems are expected.
- The usual problem is that CA is not trusted.
- Client/app has it’s own CA list and does not trust operating system level list. This required adding CA to this trusted app ring.
- Perl/Python based apps might have their local ssl and root cert trusted ring, see: Kodi and ssl_bump Squid.
- To troubleshoot full logging is often useful, see Log all request details on Squid
Links and reads
- Diladele non-free:
https://docs.diladele.com/administrator_guide_4_0/installation_and_removal/install_on_ubuntu.html
https://www.diladele.com/solution.html
- ClamAV & SquidClamAV
http://terminal28.com/how-to-install-and-configure-squid-proxy-server-clamav-squidclamav-c-icap-server-debian-linux/
http://squidclamav.darold.net/install.html
- http://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit
- http://wiki.squid-cache.org/Features/SslPeekAndSplice
- http://www.squid-cache.org/Doc/config/ssl_bump/
- http://wiki.squid-cache.org/Features/DynamicSslCert
- http://ubuntuserverguide.com/2013/12/how-to-filter-https-traffic-with-squid-3-on-ubuntu-server-13-10.htmlhttps://forums.kali.org/showthread.php?23036-SSL-Interception-with-Squid3-(MITM)
- http://marek.helion.pl/install/squid.html
- http://thejimmahknows.com/squid-3-1-caching-proxy-with-ssl/
- http://www.squid-cache.org/Doc/config/acl/https://docs.diladele.com/administrator_guide_4_0/system_configuration/https_filtering/recompile_squid.htmlhttps://smoothnet.org/squid-v3-5-proxy-with-ssl-bump/