Using IXP Manager’s Grapher API

We call IXP Manager’s statistics and graphing architecture Grapher. It’s a backend agnostic way to collect and present data. Out of the box, we support MRTG for standard interface graphs, sflow for peer to peer and per-protocol graphs, and Smokeping for latency/packet loss graphs. You can see some of this in action on INEX’s public statistics section.

Internet Exchange Points (IXPs) play a significant role in national internet infrastructures and IXP Manager is used in nearly 100 of these IXPs worldwide. In the last couple weeks we have got a number of queries from those IXPs asking for suggestions on how they can extract traffic data to address queries from their national Governments, regulators, media and members. We just published our own analysis of this for traffic over INEX here.

Grapher has a basic API interface (documented here) which we use to help those IXP Manager users address the queries they are getting. What we have provided to date are mostly quick rough-and-ready solutions but we will pull all these together over the weeks (and months) to come to see which of them might be useful permanent features in IXP Manager.

How to Use These Examples

The code snippets below are expected to be placed in a PHP file in the base directory of your IXP Manager installation (e.g. /srv/ixpmanager) and executed on the command line (e.g. php myscript.php).

Each of these scripts need the following header which is not included below for brevity:

<?php

require 'vendor/autoload.php';

use Carbon\Carbon;

$data = json_decode( file_get_contents( 
    'https://www.inex.ie/ixp/grapher/ixp?period=year&type=log&category=bits' 
) );

We’ve placed a working API endpoint for INEX above – change this for your own IXP / scenario.

Data Volume Growth

An IXP was asked by their largest national newspaper to provide daily statistics of traffic growth due to COVID-19. For historical reasons linked to MRTG graph images, the periods in IXP Manager for this data is such that: day is last 33.3 hours; week is last 8.33 days; month is last 33.33 days; and year is last 366 days.

This is fine within IXP Manager when comparing averages and maximums as we are always comparing like with like. But if we’re looking to sum up the data exchanged in a proper 24hr day then we need to process this differently. For that we use the following loop:

$start = new Carbon('2020-01-01 00:00:00');
$bits = 0;
$last = $data[0][0];
$startu = $start->format('U');
$end = $start->copy()->addDay()->format('U');

foreach( $data as $d ) {
  // if the row is before our start time, skip
  if( $d[0] < $startu ) { $last = $d[0]; continue; }

  if( $d[0] > $end ) {
    // if the row is for the next day break out and print the data 
    echo $start->format('Y-m-d') . ',' 
        . $bits/8 / 1024/1024/1024/1024 . "\n";

    // and reset for next day        
    $bits  = $d[1] * ($d[0] - $last);
    $startu = $start->addDay()->format('U');
    $end    = $start->copy()->addDay()->format('U');
  } else {
    $bits += $d[1] * ($d[0] - $last);
  }

  $last = $d[0];
}

The output is comma-separated (CSV) with the date and data volume exchanged in that 24 hour period (in TBs via 8/1024/1024/1024/1024). This can, for example, be pasted into Excel to create a simple graph:

The elements of the $d[] array mirror what you would expect to find in a MRTG log file (but the data unit represents the API request – e.g. bits/sec, pkts/sec, etc.):

  • d[0] – the UNIX timestamp of the data sample.
  • $d[1] and $d[2] – the average incoming and outgoing transfer rate in bits per second. This is valid for the time between the $d[0] value of the current entry and the $d[0] value of the previous entry. For an IXP where traffic is exchanged, we expect to see $d[1] roughly the same as $d[2].
  • $d[3] and $d[4] – the maximum incoming and outgoing transfer rate in bits per second for the current interval. This is calculated from all the updates which have occured in the current interval. If the current interval is 1 hour, and updates have occured every 5 minutes, it will be the biggest 5 minute transfer rate seen during the hour.

Traffic Peaks

The above snippet uses the average traffic values and the time between samples to calculate the overall volume of traffic exchanged. If you just want to know the traffic peaks in bits/sec on a daily basis, you can do something like this:

$daymax = 0;
$day    = null;

foreach( $data as $d ) {

    $c = ( new Carbon($d[0]) )->format('Y-m-d');

    if( $c !== $day ) {
        if( $day !== null ) {
            echo $day . ',' . $daymax / 1000/1000/1000 . "\n";
        }
        $day = $c;
        $daymax = $d[3];
    } else if( $d[3] > $daymax ) {
        $daymax = $d[3];
    }
}

The output is comma-separated (CSV) with the date and data volume exchanged in that 24 hour period (in Gbps via 1000/1000/1000). This can also be pasted into Excel to create a simple graph:

Import to Carbon / Graphite / Grafana

Something that is on our development list for IXP Manager is to integrate Graphite as a Grapher backend. Using this stack, we could create much more visually appealing graphs as well as time-shift comparisons. In fact this is how we created the graphs for this article on INEX’s website which includes graphs such as:

To create this, we need to get the data into Carbon (Graphite’s time-series database). Carbon accepts data via UDP so we used a script of the form:

foreach( $data as $d ) {
    echo "echo \"inex.ixp.run1 " . $d[1] . " " . $d[0] 
        . "\" | nc <carbon-ip-address> 2003\n";
}

This will output lines like the following which can be piped to sh:

echo "inex.ixp.run1 387495973600 1585649700" | nc -u 192.0.2.23 2003

The Carbon / Graphite / Grafana stack is quite complex so unless you are familiar with it, this option for graphing could prove difficult. To get up and running quickly, we used the docker-grafana-graphite Docker image. Beware that the default graphite/storage-schemas.conf in this image limits data retention to only 7 days.

2FA and User Session Management in IXP Manager

We’ve just released IXP Manager v5.3.0. The headline feature in this release is two-factor authentication (2fa) and user session management. This blog post overviews the PHP elements on how we did that.

While IXP Manager is a Laravel framework application, it uses Doctrine ORM as its database layer via the Laravel Doctrine bridge. For those curious, this really is a carry over from when IXP Manager was a Zend Framework application. For the migration, we concentrated on the controller and view elements of the MVC stack leaving the model layer on Doctrine. Over time we’ll probably migrate the model layer over to Laravel’s Eloquent.

Before reading on, it would be useful to first read the official documentation we have written aroud 2fa and user session management:

Hopefully the how we did this will be useful for anyone else in the same boat or even just trying to understand the Laravel authentication stack.

Two factor authentication (2fa) strengthens access security by
requiring two methods (also referred to as factors) to verify your
identity. Two factor authentication protects against phishing, social
engineering and password brute force attacks and secures your logins
from attackers exploiting weak or stolen credentials.

User session management allows a user to be logged in and remembered from multiple browsers / devices and to manage those sessions from within IXP Manager.

For 2fa, we used the antonioribeiro/google2fa-laravel package which is built on antonioribeiro/google2fa. If we were 100% in Laravel’s eco-system the would have been easier but because we use Doctrine, we needed to override a number of classes.

Structurally we need a database table to indicate if a user has 2fa enabled and to hold their 2fa secret – for this we created Entities\User2FA. Similarly, we have a controller to handle the UI interaction of enabling, configuring and disabling 2fa: User2FAController – this also includes generating QR codes for the typical 2fa activation process.

On the user session management side, we created Entities\UserRememberToken to hold multiple tokens per user (rather than Laravel’s default single token in a column in the user’s user database entry. For the frontend UI, UserRememberTokenController allows a user to view their active sessions and invalidate (delete) them if required.

The actual mechanism of enforcing 2fa is via middleware: IXP\Http\Middleware\Google2FA. This is added, as appropriate, to web routes via the RouteServiceProvider. This will check the user’s session and if 2fa is enabled but has not been completed, then the middleware will enforce 2fa before granting access to any routes covered by it.

Note that because we also implemented user session management via long-lived cookies and because the fact that a user has passed 2fa or not is held in the session, we need to persistently store the fact in the user’s specific remember token database entry. This is done via the Google2FALoginSucceeded listener. This is then later checked in the SessionGuard – where, if we log a user in via the long-lived cookie, we also make them as having passed 2fa if so set.

Speaking of the SessionGuard, this was one of the bigger changes we had to make – we overrode the Illuminate\Auth\SessionGuard as we needed to replace a few functions to make 2fa and user session management work. We have kept these to a minimum:

  1. The user() function – Laravel’s long lived session uses a single token but we require a token per device / browser. We also need to side-step 2fa for existing sessions as discussed above and allow for features such as allowing a user to delete other long-lived sessions and to provide functionality to allow these sessions to expire.
  2. The ensureRememberTokenIsSet() to actually create per-browser tokens (and to expire old ones).
  3. The queueRecallerCookie() so we can insert our own token rather than the default Laravel version.
  4. The cycleRememberToken() which is actually used to invalidae a token by changing it in Laravel. We override to delete the token.

Similarly we have to override the DoctrineUserProvider class to:

  1. Change retrieveByToken() to use our new database in which a user may have multiple sessions across different browsers / devices.
  2. Add addRememberToken() and purgeExpiredRememberTokens() to add and remove tokens.

We of course had to ammend the AuthServiceProvider to use our new overridden classes.

The above constitutes a bulk to the changes. Because 2fa can be enforced via middleware, it doesn’t really touch the core Laravel authentication process. The user session management was more invasive and responsible for the bulk of the changes required in the DoctrineUserProvider and SessionGuard.

What’s not mentioned above is the views – these are mainly covered in the views/user-remember-token (with a lot of inheritence from views/frontend) and the views/user/2fa directories.

While there are a lot more changes between v5.2.0 and v5.3.0 than 2fa and user session management, you can see the complete set of changes here.

A Whirlwind Tour of Ireland’s Internet History

I had the pleasure of giving a talk at HEAnet’s National Conference 2019 last Friday on Ireland’s internet history as seen from INEX’s perspective. HEAnet is a founding member of INEX and one of our greatest supporters. They were the first to order a 10Gb port way back when they were new and shiny; and again the first to order a 100Gb port when they became available in 2015. Both of these were collaborative efforts allowing us each to get familiar with this new technology.

Ireland’s internet history – especially the dial-up era – has many fascinating stories. I was of school-going age when this all kicked off but there are some recent excellent projects covering the era and well worth a bedtime read.

  1. The History of the Irish Internetinternethistory.ie – by Niall Richard Murphy. As well as telling his own story, Niall sat down with luminaries of that era including INEX’s own Nick Hilliard and Barry Rhodes.
  2. The TechArchives project which collects stories about Ireland’s long and convoluted relationship with information technology and preserves them. This is done through personal testimonies and includes people such as Barry Flanagan who formed one of Ireland’s first dial-up ISPs from his garage in Galway and gave me my start in the ISP industry; and Barry Rhodes whose history with Ireland’s internet starts long before INEX.
  3. For INEX’s 20th anniversary, we undertook a project to record the history of the exchange which can be found here – it also includes some personal reflections from those involved in its early days.

INEX’s Shiny New Route Servers

Copy of an article I wrote on INEX’s own blog for longevity – original published here on April 10 2019.


In this article, we talk about the new route servers that we deployed across all three peering platforms at INEX during February 2019 and, particularly, RPKI support.

Most route server instances at internet exchanges (IXPs) perform prefix filtering based on route/route6 objects published by internet routing registries (IRRDBs). INEX members would be used to creating these through RIPE’s database. However there are many other registries and the data quality of some of these IRRDB objects is often poor, with problems relating to missing, stale and incorrectly duplicated information.

A typical IRRDB entry would resemble the following:

route:          192.0.2.0/24
descr:          Example IPv4 route object
origin:         AS65500
created:        2004-12-06T11:43:57Z
last-modified:  2016-11-16T22:19:51Z
source:         SOME-IRRDB

RPKI

RPKI is a public key infrastructure framework designed to secure the internet’s routing infrastructure in a way that replaces IRRs with a database where trust is assigned by the resource holder. The equivalent of a route object in RPKI is called a ROA (Route Origin Authorisation). It is a cryptographically secure triplet of information which represents a route, the AS that originates it and the maximum prefix length that may be advertised. An example of an IPv4 and an IPv6 ROA would be:

( Origin AS,  Prefix,         Max Length )
( AS65500,    2001:db8::/32,  /48        )
( AS65501,    192.0.2.0/24,   /24        )

ROAs are typically created through your own RIR (so, RIPE for most INEX members). These RIRs are called trust anchors in RPKI. RIPE have created an extremely easy wizard for creating ROAs through the LIR Portal.

To implement RPKI in a router, the router needs to build and maintain a table of verified ROAs from the five RIRs/trust anchors. The easiest way of doing this is to use a local cache server which pulls and validates the ROAs from the trust anchors and uses a new protocol called RPKI-RTR to feed that information to routers. Currently there are three validators: RIPE’s RPKI Validator v3; Routinator 3000 from NLnetLabs; and Cloudflare’s GoRTR. INEX currently uses the former two.

RPKI validation of a route against the table of ROAs yields one of three possible results:

  • VALID: a ROA exists for the route and both the prefix length is within the allowed range and the origin ASN matches.
  • INVALID: a ROA exists for the route but either (or both) the prefix length is outside the allowed range and/or the origin ASN is different.
  • UNKNOWN: no ROA exists for the route.

UNKNOWN is a common response as the database has only a fraction of the prefix coverage as IRR databases do. We are now in a multi-year transition from IRR to RPKI route validation while ROAs are created.

Bird V2

As well as RPKI support, we have also upgrading all route servers to Bird v2.

This is a significant rewrite to Bird which, for v1, maintained separate code and daemons for IPv4 and IPv6. Bird v2 merges these code bases and also introduces support for new SAFIs such as l3vpns / mpls.

Overall, the configuration changes required were minimal and INEX continues to run separate daemons of Bird v2 for IPv4 and IPv6 daemons. Route servers are CPU intensive and separate daemons allows for maximum stability, keeps the configuration clean and fits into the existing deployment processes we have built up with IXP Manager.

Route Server Filtering Flow

Our work on the new route servers will be released to the community as part of IXP Manager v5 shortly. The new filtering flow is enumerated below. One of the key new features is that if any route fails a step, we use internal large community tagging to indicate this and the specific reason to our members through the IXP Manager looking glass (more on that later).

  1. Filter small prefixes (>/24 for IPv4, >/48 for IPv6).
  2. Filter martian / bogon ranges.
  3. Sanity check to ensure the AS path has at least one ASN and no more than 64.
  4. Sanity check to ensure the peer ASN is the same as first ASN in the prefix’s AS path.
  5. Prevent next-hop hijacking (where a member advertises a route but puts the next hop as another member’s router rather than their own). We do allow same-AS’s to specify their other router(s).
  6. Filter known transit networks.
  7. Ensure that the origin AS is in set of ASNs from member’s AS-SET. See below for some additional detail on this.
  8. RPKI validation. If it is RPKI VALID, accept the route. If it is RPKI INVALID then filter it.
  9. If the route is RPKI UNKNOWN, revert to standard IRRDB filtering.

Regarding step 7 above, an AS-SET is another type of IRRDB database entry where a network which also acts as a transit provider for other networks can enumerate the AS numbers of those downstream networks. This is something RPKI does not yet support but it is being worked on – see AS-Cones.

Lastly we have enhanced the BGP large community support to allow our members request as-path prepends on announcements to specific members over the route servers. For these and more supported communities, see the INEX route server page here.

Bird’s Eye and the Looking Glass

As well as IXP Manager, INEX has also written and open sourced a secure micro service for querying Bird daemons called Bird’s Eye. IXP Manager uses this to provide a web-based looking glass into our route collectors and servers. We have recently released v1.2.1 of Bird’s Eye which adds support for Bird v2.

We have greatly enhanced IXP Manager’s looking glass to support both Bird v2 and the large communities we use to tag filtered reasons. You can explore any of INEX’s route servers to see this yourself – for example this is route server #1 for IPv4 on INEX LAN1. When members log into IXP Manager they will also find a new Filtered Prefixes tool which will summarise any filtered routes across all 12 of INEX’s route server instances.

More Information

We have spoken about this at a number of conferences recently:

Migrating Legacy Web Applications to Laravel

I gave a talk on migrating legacy web applications to Laravel at last year’s Laravel Live UK conference in London. Those that rated it gave it high marks which is always brilliant feedback to receive.

2018 was Laravel Live UK’s inaugural conference and it was a packed house. They’ve just announced the dates for 2019 and I would strongly recommend attending for anyone using or interested in use Laravel.

Following the conference, I wrote up the talk as an article and it has just been published in the March 2019 edition of php[architect]. This is an excellent magazine which I’ve subscribed to for a few years now – the digital edition is very reasonable and comes as a DRM-free PDF onto an app on your phone/pad or downloaded to your computer.

As it happens, they chose this article as the teaser for this issue and so it is freely available online here and downloadable as a PDF here. But seriously, if you are a PHP developer, you need to subscribe to this magazine.

Lastly, if you are interested in the slide deck from the conference, you can download them here – but the article is a much better way to understand this material.