WireGuard – Linux Based VPN Server for iOS

Okay, I lied. WireGuard won’t just run on Linux for the server side but that is what it was originally designed for. Linux is the first class citizen as the WireGuard implementation there exists within the kernel.

I also lied about the clients – it’ll work on nearly any OS. We have been using OpenVPN with great success with many customers for years. We have our own management software and my macOS Viscosity client (highly recommended) has over 30 endpoints at this time. For various reasons, we use tap interfaces which just do not work for iOS.

I came across WireGuard a while ago and was intrigued by some of it’s design principles. Specifically:

  • UDP only (I remain, to this day, completely bewildered and baffled by any VPN running over TCP – yes, Mikrotik, I’m looking at your OpenVPN implementation);
  • how it presents as a simple network interface (and thus is configured via the normal iproute2 tools such as ip); and
  • its ssh-like public/private key exchange mechanism.

But I turned away as it stated that it was still a work in progress. It still states this but it looks pretty mature. Two gaps I have with OpenVPN right now seem to be filled by WireGuard: simple just works client for Apple iOS; and easy set-up mechanism for small deployments (e.g. I just want to get remote access to my home server without setting up a certificate authority or using static keys).

So, let’s look at setting up a server (Linux) / client (iOS) with WireGuard. As usual, I’m running the latest Ubuntu LTS on my server – in this case 18.04.

Important note about VPNs and dual-stack networks: many VPNs only work on IPv4. When using such VPNs on a foreign network with IPv6 support, you will only be protected for traffic that transit the IPv4 VPN. Any traffic that works over IPv6 will not go through your VPN – and today, this is a good chunk of traffic. The configuration below assumes your server is dual-stacked – which, today, it should be.

Note also in the examples below, I am using Google’s public DNS. You should install your own DNS resolver on your VPN server rather than using a third party one.

As WireGuard routes packets to and from its encrypted interface, you will need to ensure packet forward is enabled on your server:

sysctl net.ipv4.ip_forward=1
sysctl net.ipv6.conf.all.forwarding=1

Make this permanent by editing /etc/sysctl.conf.

Install WireGuard using its PPA via:

add-apt-repository ppa:wireguard/wireguard
apt-get update
apt-get install wireguard

WireGuard uses DKMS to build the module for the kernel you are running. It would be useful to do a dist-upgrade and reboot before installing this to put yourself on the latest kernel.

The installation of WireGuard above will install and build the kernel module, install the tools and create the /etc/wireguard directory. Let’s go there now and create keys for the server:

cd /etc/wireguard
# create a private server key:
wg genkey >server-private.key
chmod go-rwx server-private.key
# and create a public key from the private key:
cat server-private.key | wg pubkey >server-public.key

We may as well get ahead of ourselves and generate a key pair for our iOS client now also. When we’ve generated the configuration for the server and client, we can delete these key files from the server. In fact you should do this.

wg genkey >client1-private.key
cat client1-private.key | wg pubkey >client1-public.key

Now let’s create the server side configuration in /etc/wireguard/wg0.conf:

[Interface]
Address = 10.97.98.1/24, fd80:10:97:98::1/64
SaveConfig = false
DNS = 8.8.8.8, 2a00:1450:400b:c01::8b
ListenPort = 51820
PrivateKey = <contents of server-private.key>

# client1
[Peer]
PublicKey = <contents of client1-public.key>
AllowedIPs = 10.97.98.2/32, fd80:10:97:98::2/64

Again, chmod go-rwx wg0.conf.

The IPv6 addresses chosen above are unique local addresses (rfc4193) – similar to RFC1918 private addresses in IPv4. When choosing your IPv6 ULA, use a prefix generator such as this one. As we are using ULA addresses, we have to NAT IPv6. I hate doing this but it makes the example simple. If you have routable IPv6 addresses, try and use a real prefix without NAT.

You can now bring the tunnel up and down using the useful utility commands: wg-quick up wg0 and wg-quick down wg0. But you’ll probably want to enable them on systemd for auto-start on system boot:

systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0

When up and running, you can examine the interface with ifconfig wg0 and see the state of clients with just wg:

# ifconfig wg0
wg0: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1420
        inet 10.97.98.1  netmask 255.255.255.0  destination 10.97.98.1
        inet6 fd80:10:97:98::1  prefixlen 64  scopeid 0x0<global>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
# wg
interface: wg0
  public key: w29jZeurXAcABTTvA0V5pIOgK8jUZuYxNE9dCciN7Q8=
  private key: (hidden)
  listening port: 51821

peer: WrZzlF0fjMWKFqn/krqPrdyfYnlshLMwDNNiweEocRE=
  allowed ips: 10.97.98.2/32, fd80:10:97:98::2/128

WireGuard has an iOS client – download it from the AppStore here. One of its most useful features is the ability to add a configuration via a QR code (you will need to apt install qrencode on your server). Let’s create a client configuration in a text file on the server now:

[Interface]
PrivateKey = <contents of client1-private.key>
Address = 10.97.98.2/24, fd80:10:97:98::2/64
DNS = 8.8.8.8, 2a00:1450:400b:c01::8b

[Peer]
PublicKey = <contents of server-public.key>
Endpoint = <server-ip/hostname>:51820
AllowedIPs = 0.0.0.0/0, ::/0

Then generate the qrcode and display to screen with: qrencode -t ansiutf8 <client.conf. You’ll be able to import it by pointing your phone at the screen. Sample QR code:

There’s still a couple things you need to do to make this all work: allow UDP packets in your firewall and allow the forwarding and NATing of tunnelled traffic between the tunnel interface and the public internet facing interface(s). I don’t like to over-prescribe how to do this as there are different ways and different topologies. But let me give a basic example.

Start with allowing WireGuard traffic in your firewall – you need an iptables rules such as:

iptables  -A INPUT -p udp --dport 51820 -j ACCEPT
ip6tables -A INPUT -p udp --dport 51820 -j ACCEPT

For forwarding traffic, there are a number of options but the easiest is to use stateful rules to allow established / related traffic and assume everything coming in your encrypted tunnelled WireGuard interfaces is okay:

iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -i wg+ -j ACCEPT

ip6tables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
ip6tables -A FORWARD -i wg+ -j ACCEPT

Lastly, for NAT – and assuming eth0 is your public interface, use:

iptables  -t nat -A POSTROUTING -o eth+ -s 10.97.98.0/24 -j MASQUERADE
ip6tables -t nat -A POSTROUTING -o eth+ -s fd80:10:97:98::/64 -j MASQUERADE

Finally, test your set-up works for IPv4 and IPv6 using sites such as ipv6-test.com or ipleak.net.

You can add more peers by editing /etc/wireguard/wg0.conf and then restarting the tunnel interface via systemctl restart wg-guard@wg0. This will briefly disrupt existing tunnel traffic but it’s the simplest method. There are ways to add new tunnels on the command line but you need to remember to keep the configuration file in sync.

A Brief History of IXP Manager

For another INEX project, I was asked to put together a timeline for IXP Manager – an open source application for managing Internet eXchange Points. Reproduced here:

IXP Manager was originally a web portal written in PHP by Nick Hilliard in 2005. It was a basic database frontend that just did fairly simple CRUD (CReate, Update, Delete operations) and allowed our members to log in and view their traffic usage graphs.

Around this was a ton of Perl scripts that sucked that data out of the the database and created configuration files for route collectors, graphing, monitoring, etc.

The major achievement of Nick’s original system was the database design (the schema). The core of that schema is still the core of IXP Manager over 10 years later.

I started in INEX in 2007 and started to expand IXP Manager using what was becoming a more modern web development paradigm – Model/View/Controller with a framework called Zend Framework.

There wasn’t a grand plan here – it was just “as we needed” organic growth over the coming years.

In 2010 we decided what we had was actually pretty good and could be very useful for other IXPs. We got committee approval to open source the software and we released IXP Manager V2 in 2010 under the GPL2 license (GNU Public License v2).

This license essentially means anyone can use the software free of charge but also that they should contribute back improvements that they may make. The idea being that INEX would eventually benefit from other IXPs contributing to the project.

Open sourcing a project doesn’t mean it’ll be successful though! What we didn’t do in 2010 was put infrastructure around it such as: presentations at IXP conferences, mailing lists for user support, decent documentation, etc.

We corrected all that and re-released an updated version called IXP Manager v3 in 2012. This time it took off! We also started collaborating with ISOC (The Internet Society) around this time to help start-up IXPs (mainly eastern Europe and Africa) use IXP Manager.

Some established IXPs also contributed money towards development of missing features – most notably LONAP in the UK – and these new features fed back into INEX.

We’ve worked hard on v3 and it’s developed well since with many new features and improvements. Sometime in late 2016 – maybe even this month – we’ll release v4 which is a major leap forward again and should hopefully attract new users and developers.

INEX is very well regarded in the IXP community as an exchange that is well run and both operates and teaches best practice. All of what we’ve learnt running a good exchange has fed into IXP Manager and it helps those IXPs that use it to implement those same good practices. IXP Manager has helped raise INEX’s reputation even further.

Lately we’ve begun to realise that as a small team we can’t do it all ourselves – the more exchanges that use it, the more requests for help and features we receive and as a result, new developments take a back seat.

To try and improve this we launched a new website in 2016 – http://www.ixpmanager.org/ – and issued a call for sponsorship so we could hire a full time developer. The ‘we’ here by the way is my and Nick’s own company – Island Bridge Networks. We’re doing this on a purely cost recovery basis. I’m delighted to say we’ve just about reached our funding goal with three top line sponsors all contributing about €20k each – ISOC, Netflix and SwissIX. The hiring process has now begun!

I’m also delighted to say that there are 33 exchanges around the world using IXP Manager /that we know of/.

Nagios Plugin to Check Extreme Networks Devices

Over at INEX we’ve embarked on a forklift upgrade of the primary peering LAN using Extreme Networks Summit x670’s and x460’s. As usual, we need to monitor these 24/7 and we have just written a new Extreme Networks chassis monitoring script which should work with most Extreme devices.

It will check and generate alerts on the following items:

  • a warning if the device was recently rebooted;
  • a warning / critical if any found temperature sensors are in a non-normal state;
  • a warning / critical if any found fans are in a non-normal state;
  • a warning / critical if any found PSUs are in a non-normal state (or missing);
  • a warning / critical if the 5 sec CPU utilisation is above set thresholds;
  • a warning / critical if the memory utilisation is above set thresholds.

You’ll find the script in this Github repository: barryo/nagios-plugins.

A some verbose output follows:

./check_chassis_extreme.php -c  -h 10.11.12.13 -v 
 
CPU: 5sec - 4% 
Last reboot: 4007.7666666667 minutes ago 
Uptime: 2.8 days. 
PSU: 1 - presentOK (Serial: 1429W-xxxxx; Source: ac) 
PSU: 2 - presentOK (Serial: 1429W-xxxxx; Source: ac) 
Fan: 101 - OK (4388 RPM) 
Fan: 102 - OK (9273 RPM) 
Fan: 103 - OK (4428 RPM) 
Fan: 104 - OK (9273 RPM) 
Fan: 105 - OK (4551 RPM) 
Fan: 106 - OK (9452 RPM) 
Temp: 39'C 
Over temp alert: NO 
Memory used in slot 1: 29% 
OK - CPU: 5sec - 4%. Uptime: 2.8 days.  PSUs: 1 - presentOK; 
  2 - presentOK;. Overall system power state: redundant power 
  available. Fans: [101 - OK (4388 RPM)]; [102 - OK (9273 RPM)];
  [103 - OK (4428 RPM)]; [104 - OK (9273 RPM)]; [105 - OK (4551 
  RPM)]; [106 - OK (9452 RPM)];. Temp: 39'C. 
  Memory (slot:usage%): 1:29%.

OS X Built-in tftp Server

Turned out to be very useful during a recent RMA maintenance window:

sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist
sudo launchctl start com.apple.tftpd

The default tftp file path is /private/tftpboot. [Original source]

You can stop it with:

sudo launchctl unload -F /System/Library/LaunchDaemons/tftp.plist

And, speaking of tftp – there are some interesting projects on GitHub: hooktftp, php-tftpserver and ptftpd.

Querying Cisco MST Port Roles via SNMP with OSS_SNMP

OSS_SNMP is a PHP SNMP library written by myself for people who hate SNMP. After a customer migration from PVST to MST (Multiple Spanning Tree), I have added a number of MST functions / MIBs to OSS_SNMP:

During a fairly significant network migration involving breaking / connecting a number of links, I wanted to be able to monitor the MST port role of significant ports at a glance. For this purpose, I wrote the mst-port-roles.php script and have committed it as an example to OSS_SNMP. First, here is what it looks like when run on the command line (with hostnames obfuscated):

MST Port RolesFrom a very simple array of port details at the top of the script, it will poll all switches and for each port print:

  • device and port name;
  • port state and speed;
  • port role for each applicable MST instance.

I run it on bash and use bash colouring. The script is well documented and can easily be repurposed for other networks. You’ll find the source here.

Useful RIPE Database Links

Some very useful RIPE database links that I was recently shown include:

Pastebin Alternatives

Update 2019: we now use PrivateBin a lot. It is a self-hosted, minimalist, open source online pastebin where the server has zero knowledge of pasted data.


Pastebin has been a valuable tool for years – to the extent that pastebin it has entered the common lexicon of sysadmins, network engineers and developers.

There are, however, a few notable alternatives:

  • GitHub Gists – what’s particularly cool about these is that each Gist (which is just pasted text) is also a fully fledged Git repository with versioning and the ability for fork. There’s also syntax highlighting and a nice UI. If you’re a GitHub user, your own Gists are also linked to your account.
  • p.ip.fi – this scores big points for its pure simplicity. You’d argue that a pastebin doesn’t really need a complex UI and p.ip.fi is laudable in its complete lack of a UI. Just paste and hit Ctrl-S and you’re done. Very nice. (Credit to Nick for pointing this out).
  • sprunge.us – this is a command line pastebin which should appeal directly to sysadmins and network engineers. (Credit to dnolan for leading me to this one). This is best demonstrated via an epic traceroute:
$ traceroute -m 60 216.81.59.173 | curl -F 'sprunge=<-' http://sprunge.us
http://sprunge.us/THie

Check out the result at http://sprunge.us/THie.

Bird / Quagga with MD5 Support for IPv4/6 on FreeBSD & Linux

Over in INEX we run a route server cluster which alleviates the burden of setting up bilateral peering sessions for the more than 80% of the members that use them. The current hardware is now about six years old and we have a forklift upgrade in the works.

BGP allows for MD5 authentication between clients (using the TCP MD5 signature option, see RFC 2385) and – while recently obsoleted in RFC 5925 – it is still widely used in shared LAN mediums such as IXPs; primarily to prevent packet spoofing and session hijacking via recycled IP addresses.

Our current route server implementation runs on FreeBSD which does not support TCP MD5 in its stock kernel (you are required to compile a custom kernel – see below for details). Additionally, specifying the session MD5 is not done in the BGP daemon configuration but separately in the IPsec configuration. Lastly, our current FreeBSD version has no support for TCP MD5  over IPv6. These have all led to unnecessarily complex configurations and a degree of confusion.

Because of this, we decided to test up to date Linux and FreeBSD versions for native IPv4 and IPv6 TCP MD5 support with Bird and Quagga (our route server daemons of choice).

In each case, BGP sessions were tested for:

  • no MD5 on each end (expected to work);
  • same MD5 on each end (expected to work);
  • different MD5 on each end (expected not to work); and
  • MD5 on one end with no MD5 on the other end (expected not to work).

For Linux, the platform chosen was Ubuntu 12.04 LTS with the stock 3.2.0-40-generic kernel.

  • Sessions were tested for Quagga to Quagga and Quagga to Bird;
  • Sessions were tested over both IPv4 and IPv6;
  • The presence of valid MD5 signatures were confirmed using tcpdump -M xxx;
  • Stock Quagga and Bird from the 12.04 apt repositories were used.

The results - everything worked and worked as expected:

  • BGP sessions only established when expected (no MD5 configured, same MD5 configured);
  • This held for both IPv4 and IPv6.

Summary: Linux will support TCP MD5 nativily for IPv4 and IPv6 when using Quagga or Bird.

For FreeBSD, we used the latest production release of 9.1. TCP MD5 support is not compiled in by default so a custom kernel must be built with the additional options of:

options   TCP_SIGNATURE
options   IPSEC
device    crypto
device    cryptodev

In addition to this, the MD5 shared secrets need to be added to the IPsec SA/SD database via the setkey utility or, preferably, via the /etc/ipsec.conf file which, for example, would contain entries for IPv4 and IPv6 addresses such as:

add 192.0.2.1 192.0.2.2 tcp 0x1000 -A tcp-md5 "supersecret1";
add 2001:db8::1 2001:db8::2 tcp 0x1000 -A tcp-md5 "supersecret2";

where the addresses ending in .1/:1 are local and .2/:2 are the BGP neighbor addresses. This file can be processed by setting ipsec_enable="YES" in /etc/rc.conf and executing /etc/rc.d/ipsec reload.

  • Sessions were tested for Quagga/Linux to Quagga/FreeBSD and  from Quagga/Linux to Bird/FreeBSD;
  • Sessions were tested over both IPv4 and IPv6;
  • The presence of valid MD5 signatures were confirmed using tcpdump -M xxx;
  • Stock Quagga from the 12.04 apt repositories and stock Quagga and Bird from FreeBSD ports were used.

The results – almost everything worked and worked as expected:

  • BGP sessions only established when expected (no MD5 configured, same MD5 configured);
  • This held for both IPv4 and IPv6;
  • one odd but expected behavior – you only need to set the MD5 via setkey / ipsec.conf – setting it (or not) in the Quagga and Bird config has no effect so long as it is set via setkey (but is useful for documentation purposes). However, trying to set it in Quagga without having rebuilt the kernel will result in an error.

Summary: FreeBSD will support TCP MD5 via a custom kernel and setkey / ipsec.conf for IPv4 and IPv6. Note that there is an additional complexity when changing or removing MD5 passwords as these need to be amended / deleted via setkey which can put an extra burden on automatic route server configuration generators.

NOCtools and OSS_SNMP Get Support for Multiple Spanning Tree (MST) Protocol

NOCtools (a mixed bag collection of tools and utilities for NOC engineers) and OSS_SNMP (a PHP SNMP Library for People Who HATE SNMP, MIBs and OIDs) have just gotten support for Multiple Spanning Tree.

Specifically, OSS_SNMP has two new MIBS (Cisco’s original MST tree which has a lot of deprecated nodes – MIBS\Cisco\MST; and the newer IEEE tree – MIBS\Cisco\SMST). With these, we can, for example, get an array of [instanceID] => instanceName values from a switch by just coding:

$ciscosw = new \OSS_SNMP\SNMP( $ip, $community );
print_r( $ciscosw->useCisco_SMST()->instances() );

NOCtools has the more impressive use cases of these new features. Specifically (and just likes its RSTP/pvrspt functionality), it can:

  • Show MST port roles (root, designated, alternate, etc) for a given (or all) MST instance(s) – this is equivalent to the RSTP version;
  • From a given device, it can crawl all CDP neighbours and create a graph of all devices, their connecting ports and the MST roles of those ports. This is a really useful feature as it means you don’t need to log into multiple switches to get a handle on what links are blocking. See documentation and a sample diagram here.

For OpenVPN Fans – Optimising for GE Networks

I came across this article today which discusses OpenVPN optimisations for Gigebit Ethernet networks.

http://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux

It also shows the phenomenal improvement that can be made thanks to the AES-NI instruction set on newer Intel and AMD chips.