Network folder synced to OneDrive/SharePoint

SharePoint synchronization mechanism using in background groove.exe does a lot to block synchronization to a network folder.

Whilst this might be well explained by why would one sync a network location such as SharePoint to another network folder, there are situations where it is required. One of such requirements is when system is running in VM and the data needs to be synchronized to VM shared folder which is then visible to operating system as network drive, i.e. then available to different VMs to avoid waste of space if that folder would be put on “C:” drive.

The workaround which works pretty well is to us symbolic link. Before doing so, make sure to close any instance of groove.exe or any other software using data in the synchronized folder.

Should you have your folders synchronized already the steps are:

Stop all groove.exe processes.
Rename existing folder, in example below <Company Name Team>. If OneDrive folder was selected to be c:\OneDrive it will be c:\OneDrive\<Company Name Team> and there will be another personal OneDrive folder.
If this worked fine it means that all programs were closed properly, otherwise error would be raised.
Open CLI with Administrator privileges

MS DOS

cd c:\OneDrive mklink /d <Company Name Team> o:\SharePoint\<Company Name Team>

1
2

cd c:\OneDrive
mklink /d <Company Name Team> o:\SharePoint\<Company Name Team>
Re-run OneDrive and/or Groove to verify that all has been recognized properly.

This should be as simple as that.

Important: synchronization of private folder on SharePoint requires newer OneDrive version than the one for shared folders and this one has additional check which does not seem to accept above trick.

April 17, 2017November 25, 2017

Kodi and ssl_bump Squid

UPDATE: 2017-11-25 Kodi changed modules and how Certs are checked (certifi and schism)

Friend of mine, happy user of freshly baked private DLP based on Squid and ssl_bump, have quickly realized that to update his add-ons he has to connect to avoid ssl_bump based proxy.

Throughout checking shown that Kodi uses Python libraries with local certificates and trusted Certificate Authorities (at least on Windows). Troubleshooting did lead to:

OLD: before 2017-11-25

C:\Users\<user>\AppData\Roaming\Kodi\addons\script.module.requests\lib\requests\cacert.pem

1	C:\Users\<user>\AppData\Roaming\Kodi\addons\script.module.requests\lib\requests\cacert.pem

NEW:

C:\Users\<user>\AppData\Roaming\Kodi\addons\script.module.certifi\lib\certifi\cacert.pem

1	C:\Users\<user>\AppData\Roaming\Kodi\addons\script.module.certifi\lib\certifi\cacert.pem

AND

<user>\AppData\Roaming\Kodi\addons\script.module.schism.common\lib\requests\cacert.pem

1	<user>\AppData\Roaming\Kodi\addons\script.module.schism.common\lib\requests\cacert.pem

All what was needed was to add his Root CA cert content at the end of the file (crt format):

[other certs]
-----END CERTIFICATE-----
#
# My RootCA
#

-----BEGIN CERTIFICATE-----
[cut]
-----END CERTIFICATE-----

[other certs]

-----END CERTIFICATE-----

# My RootCA

-----BEGIN CERTIFICATE-----

[cut]

-----END CERTIFICATE-----

Afterwards everything came back to normal.

April 17, 2017April 17, 2017

Log all request details on Squid

Sometimes it is required to log all request details on Squid, i.e. when you need to figure out details to write your own URL rewriter to optimize videocache or other statistics

By defailt squid strips details after “?” and all what we need is to tern it of.

strip_query_terms off

More details at http://www.squid-cache.org/Doc/config/strip_query_terms/

By default, Squid strips query terms from requested URLs before
	logging.  This protects your user's privacy and reduces log size.

	When investigating HIT/MISS or other caching behaviour you
	will need to disable this to see the full URL used by Squid.

By default, Squid strips query terms from requested URLs before

logging. This protects your user's privacy and reduces log size.

When investigating HIT/MISS or other caching behaviour you

will need to disable this to see the full URL used by Squid.

April 17, 2017April 18, 2017

Chaining Squid URL rewriters – custom URL rewriter chained with SquidGuard

Below describes how to get URL filtering based on SquidGuard and URL rewrite for quality optimization/video caching, etc. Article talks about basics and how to set up URL rewrite and later how to chain multiple URL rewriters,

Basic URL rewrite

Basic URL rewrite has been covered here https://blob.mypn.eu/get-the-resolution-right-squid-basic-url-rewrite-script/.

SquidGuard URL filtering

SquidGuard URL filtering, how to set it up and keep alive has been covered here https://blob.mypn.eu/squidguard-url-filtering/

Chain multiple URL rewriters

Squid, at least on v3.5 does allow to define only single url_rewrite_program which causes set of implications. The main disadvantage is that without use of external program it is impossible to chain multiple URL rewriters as needed in our case.

The aim is to have chained above URL bitrate rewrite as well as to use SquidGuard to filter unwanted content, i.e. advertisements (SquidGuard category adv / ads). The perfect solution would be to be able to use ACLs to i.e. direct video related domains for URL rewrite whilst the rest to SquidGuard for filtering.

Given today limitations the simplest way (read lazy) way is to use chaining script which will do the work for us. Checking online one will find http://adzapper.sourceforge.net/#download which provides two scripts: wrapzap and zapchain. These scripts were created by Cameron Simpson back in 2000/2001, so quite some time ago and are often referenced with multiple examples how to get them working.

wrapzap & zapchain

Wrapzap is used to set all environment variables however I can’t find any of them being required for our example. At the bottom of the script it calls the real zapchain with selected URL filters

zapper="/usr/local/bin/zapchain"
tvnplayer="/usr/local/bin/squid-tvnplayer.pl"
squidguard="/usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf"
exec "${zapper}" "${squidguard}" "$tvnplayer"

zapper="/usr/local/bin/zapchain"

tvnplayer="/usr/local/bin/squid-tvnplayer.pl"

squidguard="/usr/bin/squidGuard -c /etc/squidguard/squidGuard.conf"

exec "${zapper}" "${squidguard}" "$tvnplayer"

Regardless of number of tries I could not get this working despite online reports suggesting that it should just work. Other tested option was to run zapchain directly from squid.conf with selected filters.

The main difference was probably due to the way URL rewriters were supposed to work vs nowadays. In the past it seems like rewriter would output just the new URL whilst modern implementation expects syntax:

OK rewrite-url="..."
		Rewrite the URL to the one supplied in 'rewrite-url='.
		The new URL is fetched directly by Squid and returned to
		the client as the response to its request.

OK rewrite-url="..."

Rewrite the URL to the one supplied in 'rewrite-url='.

The new URL is fetched directly by Squid and returned to

the client as the response to its request.

This required additional parsing and rewrite of original zapchain script to deal with modified output.

#!/usr/bin/perl
#
# Chain multiple redirectors together.
# - Rewrite covering SqudiGuard output and proper logic (pass output from one Rewriter to another
#     Dawid Kowalski <blob@mypn.eu> - 2017.04
# - Original author
#     Cameron Simpson <cs@zip.com.au> - 2000/2001
#

use strict vars;

use IO::File;
use IPC::Open2;

die "Usage: $0 redirectors...\n" if ! @ARGV;

## 
## Note that the ZAP_CHAINING variable is obsolete these days.
## - Cameron Simpson <cs@zip.com.au> 17jul2001
##
##$::Chaining = ( length $ENV{ZAP_CHAINING}
##	      ? $ENV{ZAP_CHAINING} eq 'FULL'
##		? 2
##		: 1
##	      : 0
##	      );
$::Chaining = 1;

my @sub=();
my $nsubs=0;

for my $sub (@ARGV)
{ 
  ++$nsubs;
  my $rd = "RD$nsubs";
  my $wr = "WR$nsubs";
  my $pid = open2($rd, $wr, $sub);
  die "$0: can't open2($sub): $!\n" if ! defined $pid;
  autoflush $wr 1;
  push(@sub,[$sub,$pid,$rd,$wr]);
}

autoflush STDOUT 1;

my @words;
my $o_;
my $redir;
my($sub,$pid,$rd,$wr);

while (defined($_=<STDIN>))
{ chomp;
  @words = split;
  $o_ = $_;
  my @nwords;
  # pass through every redirector
  for my $s (@sub)
  { ($sub,$pid,$rd,$wr)=@$s;
    print $wr $_, "\n";
    $redir=<$rd>;
    die "$0: unexpected EOF from [$sub]" if ! defined $redir;
    chomp($redir); #warn "DEBUG0: @words -> $redir";
    if (length($redir))
    # redirected
    { my @nwords=split(/\s+/,$redir);
      if (@nwords == 1)
      # plain URL - do nothing
      { #	warn "DEBUG1: @nwords";
	if (@nwords[0] == "ERR" ){
	    @nwords = @words;
	}
	$words[0]=$nwords[0];
      }
      else
      { # full redirector input line
	my @passover;
	if (@nwords != 4)
	{ #	  warn "$0: @words -> @nwords";
	}
	if (@nwords == 2 )
	{ # prepare subsitution in case of URL is modified to submit to enother rewriter
	    my $url=$words[1];
	    my @nurl=split(/^OK rewrite-url="(.*?)"/,$redir);
	    my @nurl2 = $nurl[1]; # " workaround for poor syntax highlight in editor
	    my @twords=@words;
	    @twords[0] = @nurl2[0];
	    @nwords = @twords;
	}
	@words=@nwords;
      }
      $_="@words";
    }
  }
  if ($::Chaining == 0)
  # pure redirector
  { print STDOUT (($_ eq $o_) ? '' : $words[0]), "\n";
  }
  elsif ($::Chaining == 1)
  # print new URL;
  { print STDOUT $words[0], "\n";
  }
  else
  { print STDOUT $_, "\n";
  }
}

exit 0;

100

101

102

103

104

105

106

#!/usr/bin/perl

# Chain multiple redirectors together.

# - Rewrite covering SqudiGuard output and proper logic (pass output from one Rewriter to another

# Dawid Kowalski <blob@mypn.eu> - 2017.04

# - Original author

# Cameron Simpson <cs@zip.com.au> - 2000/2001

use strict vars;

use IO::File;

use IPC::Open2;

die "Usage: $0 redirectors...\n" if ! @ARGV;

## Note that the ZAP_CHAINING variable is obsolete these days.

## - Cameron Simpson <cs@zip.com.au> 17jul2001

##$::Chaining = ( length $ENV{ZAP_CHAINING}

## ? $ENV{ZAP_CHAINING} eq 'FULL'

## ? 2

## : 1

## : 0

## );

$::Chaining = 1;

my @sub=();

my $nsubs=0;

for my $sub (@ARGV)

{

++$nsubs;

my $rd = "RD$nsubs";

my $wr = "WR$nsubs";

my $pid = open2($rd, $wr, $sub);

die "$0: can't open2($sub): $!\n" if ! defined $pid;

autoflush $wr 1;

push(@sub,[$sub,$pid,$rd,$wr]);

}

autoflush STDOUT 1;

my @words;

my $o_;

my $redir;

my($sub,$pid,$rd,$wr);

while (defined($_=<STDIN>))

{ chomp;

@words = split;

$o_ = $_;

my @nwords;

# pass through every redirector

for my $s (@sub)

{ ($sub,$pid,$rd,$wr)=@$s;

print $wr $_, "\n";

$redir=<$rd>;

die "$0: unexpected EOF from [$sub]" if ! defined $redir;

chomp($redir); #warn "DEBUG0: @words -> $redir";

if (length($redir))

# redirected

{ my @nwords=split(/\s+/,$redir);

if (@nwords == 1)

# plain URL - do nothing

{ # warn "DEBUG1: @nwords";

if (@nwords[0] == "ERR" ){

@nwords = @words;

}

$words[0]=$nwords[0];

}

else

{ # full redirector input line

my @passover;

if (@nwords != 4)

{ # warn "$0: @words -> @nwords";

}

if (@nwords == 2 )

{ # prepare subsitution in case of URL is modified to submit to enother rewriter

my $url=$words[1];

my @nurl=split(/^OK rewrite-url="(.*?)"/,$redir);

my @nurl2 = $nurl[1]; # " workaround for poor syntax highlight in editor

my @twords=@words;

@twords[0] = @nurl2[0];

@nwords = @twords;

}

@words=@nwords;

}

$_="@words";

}

if ($::Chaining == 0)

# pure redirector

{ print STDOUT (($_ eq $o_) ? '' : $words[0]), "\n";

}

elsif ($::Chaining == 1)

# print new URL;

{ print STDOUT $words[0], "\n";

}

else

{ print STDOUT $_, "\n";

}

exit 0;

This provided expected results and submitting output of one URL rewriter to another. My use case would rarely result in double modification of the output. The order in which rewriters are called represents the hit rate of the both. Adblocker implemented with use of SquidGuard gets much more traffic and will shorten all calls to ads effectively providing less load on the second one.

April 17, 2017April 18, 2017

Get the resolution right – Squid basic URL rewrite script

Squid allows to use URL rewrite program to alter URL silently (rewrite) or preferred method to redirect to other URL. Mobile apps often rely on data retrieved from URL whilst at the same time not supporting re-directions (i.e. web TV/Movie platforms).

Int the simplest form the rewrite configuration could look like:

### Simple rewrite
acl rewrite_quality dstdomain .some.cdn.network.inexistent
url_rewrite_program /path/to/rewrite-script.pl
url_rewrite_children 5
url_rewrite_access allow rewrite_quality
url_rewrite_access deny all
###

### Simple rewrite

acl rewrite_quality dstdomain .some.cdn.network.inexistent

url_rewrite_program /path/to/rewrite-script.pl

url_rewrite_children 5

url_rewrite_access allow rewrite_quality

url_rewrite_access deny all

###

What it does it that for all calls for domains defined as part of rewrite_quality acl, in this case .some.cdn.network.inexstent it would pass the URL through the rewrite-script.pl.

Squid launches the defined script upon start (depending on number of children – url_rewrite_children ) and passes requests to script STDIN.

The full description of request as passed by Squid is:

[channel-ID] URL [key-extras]

1	[channel-ID] URL [key-extras]

Further described at http://wiki.squid-cache.org/Features/AddonHelpers. The example request passed to the script looks like:

https://test.url/ 1.2.3.4/client.hostname.domain - GET myip=- myport=3128"

1	https://test.url/ 1.2.3.4/client.hostname.domain - GET myip=- myport=3128"

Additional detals around url_rewrite_program can be found at http://www.squid-cache.org/Doc/config/url_rewrite_program/

Custom URL rewrite script

Simple URL rewrite script could be as following to rewrite bitrate part of URL for TVN Player / Player (aka player.pl):

#!/usr/bin/perl
use strict;
# Turn off buffering to STDOUT
$| = 1;
# selected bitrate
my $bitrate = "2800000";
# other tested bitrates
#240000 - poorest ,420000, 600000, 800000, 1200000, 1800000, 2800000 - best

while (<>) { # Read from STDIN
    my @elems = split; # splits $_ on whitespace by default
    # The URL is the first whitespace-separated element.
    my $url = $elems[0];
    if ($url =~ m#^(http://.*\.dcs\.redcdn\.pl/.*/tvnplayer/(cutv/)?vod/.*\?type=video&bitrate=).*(&id=.*)#i) {
        $url = "${1}${bitrate}${3}";
         # previous, simple rewrite - different from latest squidGuard implementation
#        print "$url\n";
        # modern type of answer - Squid 3.5
        print "OK rewrite-url=\"$url\"\n";
    }
# another test?
#    elsif ($url =~ m#^https?://(?:[^.]*\.)?another-example\.com(/.*)?#i) {
#        print "http://redirect/blocked.html\n";
#    }
    else {
        # Unmodified URL
        print "ERR\n";
        # previous implementation
        # print "$url\n";
    }
}

#!/usr/bin/perl

use strict;

# Turn off buffering to STDOUT

$| = 1;

# selected bitrate

my $bitrate = "2800000";

# other tested bitrates

#240000 - poorest ,420000, 600000, 800000, 1200000, 1800000, 2800000 - best

while (<>) { # Read from STDIN

my @elems = split; # splits $_ on whitespace by default

# The URL is the first whitespace-separated element.

my $url = $elems[0];

if ($url =~ m#^(http://.*\.dcs\.redcdn\.pl/.*/tvnplayer/(cutv/)?vod/.*\?type=video&bitrate=).*(&id=.*)#i) {

$url = "${1}${bitrate}${3}";

# previous, simple rewrite - different from latest squidGuard implementation

# print "$url\n";

# modern type of answer - Squid 3.5

print "OK rewrite-url=\"$url\"\n";

}

# another test?

# elsif ($url =~ m#^https?://(?:[^.]*\.)?another-example\.com(/.*)?#i) {

# print "http://redirect/blocked.html\n";

# }

else {

# Unmodified URL

print "ERR\n";

# previous implementation

# print "$url\n";

}

The script as above needs to be then pointed within squid.conf

url_rewrite_program /path/to/rewrite-script.pl

1	url_rewrite_program /path/to/rewrite-script.pl

Motivation for above was that for unknown reasons web and mobile players were behaving differently and very bad quality was selected on some players regardless of the available bandwidth. Above forced the proper quality and has certainly a lot of drawbacks due to silent URL rewrite as it forces other clients to the same selected quality. Note that above example sets very low quality for tests.

April 17, 2017April 18, 2017

SquidGuard – URL filtering

SquidGuard is one of very well known URL filtering solutions. Paired together with some good URL/domain list is very powerful and fast solution.

SquidGuard installation is very simple and well described on internet.

sudo apt-get install squidguard

1	sudo apt-get install squidguard

Example squidGuard.conf could look like:

dbhome /path/to/squidguard/db
logdir /var/log/squidguard

src admin {
        ip              192.168.100.10
        user            user1 user2
        within          workhours
}
src     local {
        ip              192.168.100.0/24 127.0.0.0/8
}
src     user1 {
        ip              192.168.100.10-192.168.100.19
}
# Destinations classes
dest adv {
domainlist      BL/adv/domains
urllist         BL/adv/urls
}
# ...
dest webtv {
domainlist      BL/webtv/domains
urllist         BL/webtv/urls
}
acl {
        default {
                pass !adv !spyware !tracker all
                redirect http://wpad/blank.gif
#               redirect http://admin.foo.bar.de/cgi-bin/blocked.cgi?clientaddr=%a&clientname=%n&clientuser=%i&clientgroup=%s&targetgroup=%t&url=%u
                log default.log
}

dbhome /path/to/squidguard/db

logdir /var/log/squidguard

src admin {

ip 192.168.100.10

user user1 user2

within workhours

}

src local {

ip 192.168.100.0/24 127.0.0.0/8

}

src user1 {

ip 192.168.100.10-192.168.100.19

}

# Destinations classes

dest adv {

domainlist BL/adv/domains

urllist BL/adv/urls

}

# ...

dest webtv {

domainlist BL/webtv/domains

urllist BL/webtv/urls

}

acl {

default {

pass !adv !spyware !tracker all

redirect http://wpad/blank.gif

# redirect http://admin.foo.bar.de/cgi-bin/blocked.cgi?clientaddr=%a&clientname=%n&clientuser=%i&clientgroup=%s&targetgroup=%t&url=%u

log default.log

}

By default config does not include the dest sections.

To generate one as no other list/script could be easily found, below was quickly written:

#!/bin/bash
DBDIR="/path/to/db/"
OD=${PWD}; cd "${DBDIR}"
for i in `find BL/* -type d`; do
    NAME=`echo ${i}|cut -d"/" -f2`
    echo "dest ${NAME} {
domainlist      ${i}/domains
urls            ${i}/urls
}"
done
cd "${OD}"

#!/bin/bash

DBDIR="/path/to/db/"

OD=${PWD}; cd "${DBDIR}"

for i in `find BL/* -type d`; do

NAME=`echo ${i}|cut -d"/" -f2`

echo "dest ${NAME} {

domainlist ${i}/domains

urls ${i}/urls

done

cd "${OD}"

The minor problem with the script is that it generates incorrect lines generating error at SqudiGuard level for parent folders with subcategories. But you’ll need to run this script once only. Should one find better way to get it done, let me know pls.

One of well known, updated and free to use for private purpose is Shalla list.

Automated list update process could be described with below. Please note to not run it more often than every 24h as per request from Shalla guys as the list is not updated more often.

#!/bin/bash

OD="$PWD"
SQGD="/mnt/sda6/squidguard/"
FL="$SQGD/shallalist.tar.gz"
TMP="$SQGD/tmp/"
SQGDB="$SQGD/db/"

SRC="http://www.shallalist.de/Downloads/shallalist.tar.gz"
http_proxy="" # let's turn off proxy as file could be cached, surprise surprise.
CRLO=`/usr/bin/curl -o $FL $SRC -s`
retval=$?

if [ $retval -ne 0 ];then
    echo "Something went wrong with download:
${CRLO}"
    exit $retval
fi

FLSZ=`/usr/bin/find $FL -size +100k 2>/dev/null`
if [[ -s "$FL" ]] && [[ -n "$FLSZ" ]]; then
#if [[ -n "$FLSZ" ]]; then
    echo "good"
    exit 0
    cd ${TMP}
    tar xzf $FL
    cd ${SQGDB}
    mv BL BL_
    mv ${TMP}/BL ${SQGDB}
    rm -rf ./BL_
    update-squidguard 2>&1 > /dev/null
    cd ${OD}
else
    echo "ERROR: file $FL is way too small or does not exist"
    exit 1
fi
exit 0

#!/bin/bash

OD="$PWD"

SQGD="/mnt/sda6/squidguard/"

FL="$SQGD/shallalist.tar.gz"

TMP="$SQGD/tmp/"

SQGDB="$SQGD/db/"

SRC="http://www.shallalist.de/Downloads/shallalist.tar.gz"

http_proxy="" # let's turn off proxy as file could be cached, surprise surprise.

CRLO=`/usr/bin/curl -o $FL $SRC -s`

retval=$?

if [ $retval -ne 0 ];then

echo "Something went wrong with download:

${CRLO}"

exit $retval

FLSZ=`/usr/bin/find $FL -size +100k 2>/dev/null`

if [[ -s "$FL" ]] && [[ -n "$FLSZ" ]]; then

#if [[ -n "$FLSZ" ]]; then

echo "good"

exit 0

cd ${TMP}

tar xzf $FL

cd ${SQGDB}

mv BL BL_

mv ${TMP}/BL ${SQGDB}

rm -rf ./BL_

update-squidguard 2>&1 > /dev/null

cd ${OD}

else

echo "ERROR: file $FL is way too small or does not exist"

exit 1

exit 0

Script can be then linked to /etc/cron.daily folder:

ln -s /usr/local/sbin/update-shalla-squidguard.sh /etc/cron.daily/

1	ln -s /usr/local/sbin/update-shalla-squidguard.sh /etc/cron.daily/

Links:

April 11, 2017November 12, 2017

Thumbs on tunnels – terminate tunnel and check content – DLP

The past was easy peasy, not the case anymore these days. HTTPS was rarely used and only where it really needed to be. Anyone on the wire could see what’s going on. And yeah, it meant literally everyone.

Was it good? No, not at all. Everyone could track anyone, collect information, behavior, etc. Result? Everyone started to move towards HTTPS (ssl then tls). This basically means couple of good and bad outcomes. Squid cache ration went down, and I mean very, very low these days. Most connections handled by Squid these days are tunnels (CONNECT) and anything can be sent through such tunnel, not only private data, credit card numbers, etc. but also ads and virus (sic!).

This created specific need to be able to check the content of the stream, especially in enterprise environment. All the DLP systems can only work on streams if they can check the payload, decrypted traffic which means they need to be so well known man-in-the-middle. This means to be tunnel end-point from client side and initiation of new tunnel to server. Only this allows to inspect the traffic.

At the same time one could ask, hey how about certificates? The long winded answer can be found elsewhere, but short answer is that in enterprise environment there is at least one Root Certificate Authority (CA) and Intermediate Certificate Authority can be created. The root or intermediate CA (called CA afterwards) together with key file is uploaded to the DLP system allowing it to generate new certificates on the fly for terminated tunnels.

The CA is already trusted in enterprise environment as Root CA certificate is added to Trusted CA key ring on client host as part of operating system deployment package or domain connection.

Since client trusts Root CA it automatically trusts certificates signed by the given CA – this is how Public Key Infrastructure works. Additionally the DLP system usually tries to mimic all original certificate parameters and only CA related details are different. But people rarely check details of certificate if everything is in green and no popup/error is raised.

private DLP

With all above being said one can have their own DLP system based on Squid. This “little piece of software” is great in handling huge loads, caching data and calling others for help. This is what we’re going to do today.

Squid to terminate SSL connection uses ssl_bump functionality. The Ubunty 16.04 LTS default package does not allow to use this great functionality hence we need to start with little configuration.

Let’s get sources and all needed libraries (for all below any sudo call is skipped as it just makes the output longer and you, dear reader certainly know how to use sudo).

apt-get source squid
apt-get build-dep squid gnutls-bin devscripts build-essential fakeroot

1 2	apt-get source squid apt-get build-dep squid gnutls-bin devscripts build-essential fakeroot

then we need to apply patch to enable ssl as needed

--- build/squid3/squid3-3.5.15/debian/rules     2016-02-17 01:13:33.000000000 +0100
+++ build/squid3/squid3-3.5.15/debian/rules.new 2016-02-22 22:50:04.079470555 +0100
@@ -45,7 +45,10 @@
                --with-pidfile=/var/run/squid.pid \
                --with-filedescriptors=65536 \
                --with-large-files \
-               --with-default-user=proxy
+               --with-default-user=proxy \
+               --with-openssl \
+               --enable-ssl \
+               --enable-ssl-crtd

 BUILDINFO := $(shell lsb_release -si 2>/dev/null)

--- build/squid3/squid3-3.5.15/debian/rules 2016-02-17 01:13:33.000000000 +0100

+++ build/squid3/squid3-3.5.15/debian/rules.new 2016-02-22 22:50:04.079470555 +0100

@@ -45,7 +45,10 @@

--with-pidfile=/var/run/squid.pid \

--with-filedescriptors=65536 \

--with-large-files \

- --with-default-user=proxy

+ --with-default-user=proxy \

+ --with-openssl \

+ --enable-ssl \

+ --enable-ssl-crtd

BUILDINFO := $(shell lsb_release -si 2>/dev/null)

To apply patch, use your standard methodology patch -p<level> < diff-file.patch

Afterward squid package and any missing packages need to be installed

dpkg -i squid-package-version-#-.deb squid-common-package-version-#.deb
apt-get install -f
# and prevent squid from being upgraded
apt-mark hold squid3 squid3-common

dpkg -i squid-package-version-#-.deb squid-common-package-version-#.deb

apt-get install -f

# and prevent squid from being upgraded

apt-mark hold squid3 squid3-common

Certificate generation

First of all we need to have folder where we will store it all

mkdir /etc/squid/CA-certificates
cd /etc/squid/CA-certificates
#Create the cnf configuration file x509v3ca.cnf
cat > /etc/squid/CA-certificates/x509v3ca.cnf << EOF
[ req ]
default_bits              = 4096
default_md               = sha1
default_keyfile          = rootca.key
distinguished_name  = req_distinguished_name
x509_extensions      = v3_ca
string_mask             = nombstr

[ req_distinguished_name ]

[ v3_ca ]
basicConstraints        = critical,CA:true
nsCertType  		= critical,sslCA
extendedKeyUsage  	= critical,serverAuth,clientAuth,emailProtection,timeStamping,msCodeInd,msCodeCom,msCTLSign,msSGC,msEFS,nsSGC
keyUsage  		= keyCertSign,cRLSign
subjectKeyIdentifier    = hash
EOF

mkdir /etc/squid/CA-certificates

cd /etc/squid/CA-certificates

#Create the cnf configuration file x509v3ca.cnf

cat > /etc/squid/CA-certificates/x509v3ca.cnf << EOF

[ req ]

default_bits = 4096

default_md = sha1

default_keyfile = rootca.key

distinguished_name = req_distinguished_name

x509_extensions = v3_ca

string_mask = nombstr

[ req_distinguished_name ]

[ v3_ca ]

basicConstraints = critical,CA:true

nsCertType = critical,sslCA

extendedKeyUsage = critical,serverAuth,clientAuth,emailProtection,timeStamping,msCodeInd,msCodeCom,msCTLSign,msSGC,msEFS,nsSGC

keyUsage = keyCertSign,cRLSign

subjectKeyIdentifier = hash

EOF

The certificates mea:

#Generate RSA CA private key:
openssl genrsa -out ./rootca.key 4096
#Create CA certificate
openssl req -new -nodes -x509 -sha1 -out ./rootca.crt -key ./rootca.key -config ../x509v3ca.cnf -extensions v3_ca -subj '/O=High Security DKLP/OU=DKLP RootCA/CN=DKLP/' -days 9999
#The certificate in PEM format. Use this to distribute to most non-Windows platforms. e.g. *nix systems.
openssl x509 -in ./rootca.crt -out ./rootca.pem -outform PEM
#The certificate in pkcs12 format. Use this to distribute to Windows platforms.
openssl pkcs12 -export -in ./rootca.crt -inkey ./rootca.key -out ./rootca.p12 -name "RootCA" -password pass:
#The certificate in DER binary format format. Use this to distribute to Android Devices.
openssl x509 -inform PEM -outform DER -in ./rootca.crt -out /./rootca.crt

#Generate RSA CA private key:

openssl genrsa -out ./rootca.key 4096

#Create CA certificate

openssl req -new -nodes -x509 -sha1 -out ./rootca.crt -key ./rootca.key -config ../x509v3ca.cnf -extensions v3_ca -subj '/O=High Security DKLP/OU=DKLP RootCA/CN=DKLP/' -days 9999

#The certificate in PEM format. Use this to distribute to most non-Windows platforms. e.g. *nix systems.

openssl x509 -in ./rootca.crt -out ./rootca.pem -outform PEM

#The certificate in pkcs12 format. Use this to distribute to Windows platforms.

openssl pkcs12 -export -in ./rootca.crt -inkey ./rootca.key -out ./rootca.p12 -name "RootCA" -password pass:

#The certificate in DER binary format format. Use this to distribute to Android Devices.

openssl x509 -inform PEM -outform DER -in ./rootca.crt -out /./rootca.crt

Depending on client system certificate import will look differently. The good idea might be to place it on some easily accessible server, i.e. local wpad system.

Once certificate is downloaded it should be installed. On windows this can be done with:

certutil -addstore -enterprise -f "Root" rootca.pem
Root
Signature matches Public Key
Certificate "CN=DKLP" added to store.
CertUtil: -addstore command completed successfully.

certutil -addstore -enterprise -f "Root" rootca.pem

Root

Signature matches Public Key

Certificate "CN=DKLP" added to store.

CertUtil: -addstore command completed successfully.

Some apps do not respect Operating System level certificates, but most will. Some apps might need to be restarted it shouldn’t be required to restart full system, but who knows what type of ancient system one can be using?

All the new certificates generated on the fly need to be stored somewhere. The folder needs to be created:

mkdir /mnt/sda6/Squid_ssldb
chown proxy:proxy /mnt/sda6/Squid_ssldb

1 2	mkdir /mnt/sda6/Squid_ssldb chown proxy:proxy /mnt/sda6/Squid_ssldb

And update squid.conf (not all SSL cert ERROR related flags should be as below, tune it up to your needs).

############################################################################
# https proxy
#sslproxy_foreign_intermediate_certs /etc/squid/extra-intermediate-CA.pem

acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
acl sslBumpnet src "/etc/squid/ssl_bump/sslBumpnet"
acl sslnoBumpnet src "/etc/squid/ssl_bump/sslnoBumpnet"
acl sslnoBumpnetdst dst "/etc/squid/ssl_bump/sslnoBumpnetdst"
acl sslnoBumpSites ssl::server_name "/etc/squid/ssl_bump/sslnoBumpSites"

#ssl_bump client-first
ssl_bump peek step1
ssl_bump splice sslnoBumpnetdst
ssl_bump splice sslnoBumpSites
ssl_bump splice sslnoBumpnet !sslBumpnet
ssl_bump bump all

# The default port with ssl-bump (this is the one used in client proxy settings)
http_port  3127 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_bump/certs/rootca.crt key=/etc/squid/ssl_bump/certs/rootca.key
# Let's silently intercept connections (including SSL) for stubborn customers 
https_port 3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_bump/certs/rootca.crt key=/etc/squid/ssl_bump/certs/rootca.key

# self signed
# comment out the sslproxy_flags
acl SSLERR ssl_error X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN
acl BrokenButTrustedServers2 dstdomain "/etc/squid/ssl_bump/dstdom2.broken"
acl UnableGetIssuer ssl_error X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE
sslproxy_cert_error allow BrokenButTrustedServers2 UnableGetIssuer
sslproxy_cert_error allow SSLERR
#sslproxy_cert_error deny all

sslproxy_flags DONT_VERIFY_PEER
sslproxy_cert_error allow all

sslproxy_options NO_SSLv2,NO_SSLv3,SINGLE_DH_USE
sslproxy_cipher EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS

# Cert generator
sslcrtd_program /usr/lib/squid/ssl_crtd -s /mnt/sda6/Squid_ssldb -M 4MB
sslcrtd_children 20 startup=10 idle=5

############################################################################

# https proxy

#sslproxy_foreign_intermediate_certs /etc/squid/extra-intermediate-CA.pem

acl step1 at_step SslBump1

acl step2 at_step SslBump2

acl step3 at_step SslBump3

acl sslBumpnet src "/etc/squid/ssl_bump/sslBumpnet"

acl sslnoBumpnet src "/etc/squid/ssl_bump/sslnoBumpnet"

acl sslnoBumpnetdst dst "/etc/squid/ssl_bump/sslnoBumpnetdst"

acl sslnoBumpSites ssl::server_name "/etc/squid/ssl_bump/sslnoBumpSites"

#ssl_bump client-first

ssl_bump peek step1

ssl_bump splice sslnoBumpnetdst

ssl_bump splice sslnoBumpSites

ssl_bump splice sslnoBumpnet !sslBumpnet

ssl_bump bump all

# The default port with ssl-bump (this is the one used in client proxy settings)

http_port 3127 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_bump/certs/rootca.crt key=/etc/squid/ssl_bump/certs/rootca.key

# Let's silently intercept connections (including SSL) for stubborn customers

https_port 3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_bump/certs/rootca.crt key=/etc/squid/ssl_bump/certs/rootca.key

# self signed

# comment out the sslproxy_flags

acl SSLERR ssl_error X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN

acl BrokenButTrustedServers2 dstdomain "/etc/squid/ssl_bump/dstdom2.broken"

acl UnableGetIssuer ssl_error X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE

sslproxy_cert_error allow BrokenButTrustedServers2 UnableGetIssuer

sslproxy_cert_error allow SSLERR

#sslproxy_cert_error deny all

sslproxy_flags DONT_VERIFY_PEER

sslproxy_cert_error allow all

sslproxy_options NO_SSLv2,NO_SSLv3,SINGLE_DH_USE

sslproxy_cipher EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:HIGH:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS

# Cert generator

sslcrtd_program /usr/lib/squid/ssl_crtd -s /mnt/sda6/Squid_ssldb -M 4MB

sslcrtd_children 20 startup=10 idle=5

Create set of files under /etc/squid/ssl_bump:

sslBumpnet – subnets/hosts which we will bump, can be selective if needed
sslnoBumpnet – these subnets/hosts we won’t bump (see logical construction above and tune it to your needs
sslnoBumpnetdst – these domains/servers we won’t bump
sslnoBumpSites

Example sslnoBumpnetdst as it might be tricky to set it up at the beginning. Some apps have built-in certificates and verify connection against them, i.e. Google Play store and couple of others.

Interesting was to find out what banks are doing with data and identify as in one case at least that for stats reasons were sending out GET request with bank account details, including saldo and transaction details to on-line stats agency. This was against any bank standards and bank should be prosecuted due to this. This was only possible after terminating tunnels as otherwise GET wasn’t visible. This can only be acceptable for tests and for your own private use at home in the isolated lab (as always).

Below list was fine tuned based on tests and was longer earlier.

##
# .banks Hosts/domains
##
connectivitycheck.gstatic.com
play.google.com
.googleusercontent.com
play.googleapis.com
.sync.app.asana.com

######
# company SSL
# Juniper Pulse will require this
#.company domain .com
ssl.company.com.external.domain
######
######
# Company Skype
# seen but down to domain
# sippoolbn11a05.infra.lync.com
# webpoolbn11a05.infra.lync.com
# lyncdiscoverinternal.acme.com
# lyncdiscover.acme.com
# sip.acme.com
# pipe.skype.com - not needed

#sip.acme.com
#lyncdiscover.acme.com
#lyncdiscoverinternal.acme.com
#.online.lync.com
.infra.lync.com

#.lync.com
#.skype.com
######
######
# WU (Squid 3.5.x and above with SSL Bump)
# Only this sites must be spliced.
.update.microsoft.com
.update.microsoft.com.akadns.net
######

# .banks Hosts/domains

connectivitycheck.gstatic.com

play.google.com

.googleusercontent.com

play.googleapis.com

.sync.app.asana.com

######

# company SSL

# Juniper Pulse will require this

#.company domain .com

ssl.company.com.external.domain

######

# Company Skype

# seen but down to domain

# sippoolbn11a05.infra.lync.com

# webpoolbn11a05.infra.lync.com

# lyncdiscoverinternal.acme.com

# lyncdiscover.acme.com

# sip.acme.com

# pipe.skype.com - not needed

#sip.acme.com

#lyncdiscover.acme.com

#lyncdiscoverinternal.acme.com

#.online.lync.com

.infra.lync.com

#.lync.com

#.skype.com

######

# WU (Squid 3.5.x and above with SSL Bump)

# Only this sites must be spliced.

.update.microsoft.com

.update.microsoft.com.akadns.net

######

The last point is to restart squid

service squid reload

1	service squid reload

To test it, select port 3127 as proxy. Once all tested transparent interception (don’t forget to have CA trusted on client side).

IPT=/sbin/iptables
INS=inside          # your internal interface (can be result of ip addr ls... function
SQUID=1.2.3.4       # your squid IP address
SQUIDP=3128         # squid intercept port
SQUIDPSSL=3129      # squid intercept port ssl
$IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 80 -j LOG --log-level info --log-prefix "IP PROXY-redirect: DNAT "
$IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 80 -j DNAT --to-destination $SQUID:$SQUIDP
$IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 443 -j LOG --log-level info --log-prefix "IP PROXY-redirect: DNAT "
$IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 443 -j DNAT --to-destination $SQUID:$SQUIDPSSL

IPT=/sbin/iptables

INS=inside # your internal interface (can be result of ip addr ls... function

SQUID=1.2.3.4 # your squid IP address

SQUIDP=3128 # squid intercept port

SQUIDPSSL=3129 # squid intercept port ssl

$IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 80 -j LOG --log-level info --log-prefix "IP PROXY-redirect: DNAT "

$IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 80 -j DNAT --to-destination $SQUID:$SQUIDP

$IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 443 -j LOG --log-level info --log-prefix "IP PROXY-redirect: DNAT "

$IPT -t nat -A PROXY-redirect -i $INS ! -d $SQUID -p tcp --dport 443 -j DNAT --to-destination $SQUID:$SQUIDPSSL

For non-Android based systems proxy.pac/wpad.dat file could be created

function FindProxyForURL(url, host)
{
  if (shExpMatch(url,"*://192.168.*/*"))
    return "DIRECT";
  else if (shExpMatch(url,"*://172.16.*.*/*"))
    return "DIRECT";
  else if (shExpMatch(url,"*://127.0.0.1/*"))
    return "DIRECT";
  else if (shExpMatch(url,"*://localhost/*"))
    return "DIRECT";
  else if (shExpMatch(url,"*.our.domain/*"))
    return "DIRECT";
  else
  {
    return "PROXY our.proxy:3127";
  }
}

function FindProxyForURL(url, host)

{

if (shExpMatch(url,"*://192.168.*/*"))

return "DIRECT";

else if (shExpMatch(url,"*://172.16.*.*/*"))

return "DIRECT";

else if (shExpMatch(url,"*://127.0.0.1/*"))

return "DIRECT";

else if (shExpMatch(url,"*://localhost/*"))

return "DIRECT";

else if (shExpMatch(url,"*.our.domain/*"))

return "DIRECT";

else

{

return "PROXY our.proxy:3127";

}

To get that working the wpad.yourdomain and wpad host needs to be resolved to your web server and thw wpad.dat/proxy.pac file needs to be in root folder.

Android platform is prone to not process this file (at least as for April 2017 and manual proxy settings in WiFi section need to be set.

This should all work… but hey, problems are expected.

The usual problem is that CA is not trusted.
Client/app has it’s own CA list and does not trust operating system level list. This required adding CA to this trusted app ring.
Perl/Python based apps might have their local ssl and root cert trusted ring, see: Kodi and ssl_bump Squid.
To troubleshoot full logging is often useful, see Log all request details on Squid

Links and reads

April 10, 2017April 18, 2017

404 URL not found – permalinks and WordPress

Setting up WordPress one can run into very silly issues.

Not Found

The requested URL /test was not found on this server.

Apache/2.4.7 (Ubuntu) Server at blob.mypn.eu Port 443

The usual suspects are:

.htaccess – verified,
mod_rewrite/rewrite – verified too

Yet despite number of reviewed online recommendations nothing worked until silly mistake has been discovered the web server site definition:

Bad:

<Directory />
  Options FollowSymLinks
  AllowOverride FileInfo
</Directory>

Options FollowSymLinks

AllowOverride FileInfo

</Directory>

Good:

<Directory /folder/where/your/wordpress/is/>
  Options FollowSymLinks
  AllowOverride FileInfo
</Directory>

Options FollowSymLinks

AllowOverride FileInfo

</Directory>

The most important part is <Directory …> where it needs to point to directory where your WordPress is installed and doesn’t go well along with “/” set.