Upgrading Legacy Versions of IXP Manager

Legacy installations of IXP Manager can be very difficult to upgrade as you can find yourself in a dependency nightmare whereby the old version of IXP Manager will not run on modern versions of PHP; and vice versa.

In case you missed it, we have a new modern website for IXP Manager – find it at https://www.ixpmanager.org/. One of the features of this new website is that we now gather IXP Manager usage statistics on a daily basis – including the distribution of versions in use.

Reassuringly, of the installs we can poll for version used, ~65% are using one of the latest three minor versions (5.5.0 – 5.7.0). This is reassuring for a number of reasons including: knowing that IXPs stay current; knowing that IXPs are concerned about security updates; and knowing that the upgrade process is not especially difficult.

Of the 145 installs we know about, we can poll 116 and collect the version is use which yields the following table:

Distribution of major IXP Manager versions in use as at September 8th 2020.

Legacy installations of IXP Manager can be very difficult to upgrade as you can find yourself in a dependency nightmare whereby the old version of IXP Manager will not run on modern versions of PHP; and vice versa.

Community IX Atlanta (CIX-ATL) are in the process of upgrading from 4.9.3 to the latest (5.7.0) and they graciously allowed me to record the process:

The video is a real-life experience where it wasn’t planned in advance allowing the viewer to see the mistakes and thought processes throughout. Also, if you weren’t aware of it, we have an on-going series of IXP Manager tutorials here.

When considering a legacy upgrade, there are two main approaches:

  1. Build a new IXP Manager installation on a new (modern) server and migrate the database (this is what we’ve done here).
  2. Attempt an in place upgrade alternating between IXP Manager upgrades and operating system upgrades. This is probably more awkward with more scope for issues to crop up (especially on non-IXP Manager applications which may be on the same server).

Remember, what’s covered here is “just” the IXP Manager and database upgrade. There’s a bunch of other things that would also need to be done including:

  • Working through the various upgrade actions in the release notes (mentioned throughout the video). Essentially you’ll need to step through each set of release notes for the versions you cycle through (and jump over).
  • If building a new server, pointing elements such as route server cron jobs and other API consumers at the new server.
  • Migrating other applications from the legacy server (e.g. maybe you have mrtg co-installed there).

In a production environment, my goal would be to build the new IXP Manager installation with the copied and upgraded database and run them in parallel. NB: either avoid or duplicate changes made in the UI across both installations of IXP Manager for this period of time.

Once the new installation of IXP Manager is ready for production use, you will then step through all external tools that consume data from it (sflow, mrtg, route servers, route collectors, etc.) and migrate them to the new installation. Sometimes simply updating DNS can achieve most of this but you’ll probably want to take it piece-meal and ensure each external service works as expected.

Take particular care with essential services such as route servers. This is an opportune time to upgrade to Bird v2 and add RPKI. What we did at INEX was do one route server at a time with 1-2 weeks between upgrades. This allowed time to ensure the new system was stable and also to ensure no member issues due to RPKI filtering, etc. (spoiler alert: it was uneventful!).

As you complete the migration, you can also consider if some services should be left on the “old” server. Separating tasks between different servers is good practice and so ask youself if everything should be migrated over to the new server.

More than anything, I hope this video entices you to keep current with your IXP Manager installations!

Using IXP Manager’s Grapher API

We call IXP Manager’s statistics and graphing architecture Grapher. It’s a backend agnostic way to collect and present data. Out of the box, we support MRTG for standard interface graphs, sflow for peer to peer and per-protocol graphs, and Smokeping for latency/packet loss graphs. You can see some of this in action on INEX’s public statistics section.

Internet Exchange Points (IXPs) play a significant role in national internet infrastructures and IXP Manager is used in nearly 100 of these IXPs worldwide. In the last couple weeks we have got a number of queries from those IXPs asking for suggestions on how they can extract traffic data to address queries from their national Governments, regulators, media and members. We just published our own analysis of this for traffic over INEX here.

Grapher has a basic API interface (documented here) which we use to help those IXP Manager users address the queries they are getting. What we have provided to date are mostly quick rough-and-ready solutions but we will pull all these together over the weeks (and months) to come to see which of them might be useful permanent features in IXP Manager.

How to Use These Examples

The code snippets below are expected to be placed in a PHP file in the base directory of your IXP Manager installation (e.g. /srv/ixpmanager) and executed on the command line (e.g. php myscript.php).

Each of these scripts need the following header which is not included below for brevity:


require 'vendor/autoload.php';

use Carbon\Carbon;

$data = json_decode( file_get_contents( 
) );

We’ve placed a working API endpoint for INEX above – change this for your own IXP / scenario.

Data Volume Growth

An IXP was asked by their largest national newspaper to provide daily statistics of traffic growth due to COVID-19. For historical reasons linked to MRTG graph images, the periods in IXP Manager for this data is such that: day is last 33.3 hours; week is last 8.33 days; month is last 33.33 days; and year is last 366 days.

This is fine within IXP Manager when comparing averages and maximums as we are always comparing like with like. But if we’re looking to sum up the data exchanged in a proper 24hr day then we need to process this differently. For that we use the following loop:

$start = new Carbon('2020-01-01 00:00:00');
$bits = 0;
$last = $data[0][0];
$startu = $start->format('U');
$end = $start->copy()->addDay()->format('U');

foreach( $data as $d ) {
  // if the row is before our start time, skip
  if( $d[0] < $startu ) { $last = $d[0]; continue; }

  if( $d[0] > $end ) {
    // if the row is for the next day break out and print the data 
    echo $start->format('Y-m-d') . ',' 
        . $bits/8 / 1024/1024/1024/1024 . "\n";

    // and reset for next day        
    $bits  = $d[1] * ($d[0] - $last);
    $startu = $start->addDay()->format('U');
    $end    = $start->copy()->addDay()->format('U');
  } else {
    $bits += $d[1] * ($d[0] - $last);

  $last = $d[0];

The output is comma-separated (CSV) with the date and data volume exchanged in that 24 hour period (in TBs via 8/1024/1024/1024/1024). This can, for example, be pasted into Excel to create a simple graph:

The elements of the $d[] array mirror what you would expect to find in a MRTG log file (but the data unit represents the API request – e.g. bits/sec, pkts/sec, etc.):

  • d[0] – the UNIX timestamp of the data sample.
  • $d[1] and $d[2] – the average incoming and outgoing transfer rate in bits per second. This is valid for the time between the $d[0] value of the current entry and the $d[0] value of the previous entry. For an IXP where traffic is exchanged, we expect to see $d[1] roughly the same as $d[2].
  • $d[3] and $d[4] – the maximum incoming and outgoing transfer rate in bits per second for the current interval. This is calculated from all the updates which have occured in the current interval. If the current interval is 1 hour, and updates have occured every 5 minutes, it will be the biggest 5 minute transfer rate seen during the hour.

Traffic Peaks

The above snippet uses the average traffic values and the time between samples to calculate the overall volume of traffic exchanged. If you just want to know the traffic peaks in bits/sec on a daily basis, you can do something like this:

$daymax = 0;
$day    = null;

foreach( $data as $d ) {

    $c = ( new Carbon($d[0]) )->format('Y-m-d');

    if( $c !== $day ) {
        if( $day !== null ) {
            echo $day . ',' . $daymax / 1000/1000/1000 . "\n";
        $day = $c;
        $daymax = $d[3];
    } else if( $d[3] > $daymax ) {
        $daymax = $d[3];

The output is comma-separated (CSV) with the date and data volume exchanged in that 24 hour period (in Gbps via 1000/1000/1000). This can also be pasted into Excel to create a simple graph:

Import to Carbon / Graphite / Grafana

Something that is on our development list for IXP Manager is to integrate Graphite as a Grapher backend. Using this stack, we could create much more visually appealing graphs as well as time-shift comparisons. In fact this is how we created the graphs for this article on INEX’s website which includes graphs such as:

To create this, we need to get the data into Carbon (Graphite’s time-series database). Carbon accepts data via UDP so we used a script of the form:

foreach( $data as $d ) {
    echo "echo \"inex.ixp.run1 " . $d[1] . " " . $d[0] 
        . "\" | nc <carbon-ip-address> 2003\n";

This will output lines like the following which can be piped to sh:

echo "inex.ixp.run1 387495973600 1585649700" | nc -u 2003

The Carbon / Graphite / Grafana stack is quite complex so unless you are familiar with it, this option for graphing could prove difficult. To get up and running quickly, we used the docker-grafana-graphite Docker image. Beware that the default graphite/storage-schemas.conf in this image limits data retention to only 7 days.

INEX’s Shiny New Route Servers

Copy of an article I wrote on INEX’s own blog for longevity – original published here on April 10 2019.

In this article, we talk about the new route servers that we deployed across all three peering platforms at INEX during February 2019 and, particularly, RPKI support.

Most route server instances at internet exchanges (IXPs) perform prefix filtering based on route/route6 objects published by internet routing registries (IRRDBs). INEX members would be used to creating these through RIPE’s database. However there are many other registries and the data quality of some of these IRRDB objects is often poor, with problems relating to missing, stale and incorrectly duplicated information.

A typical IRRDB entry would resemble the following:


RPKI is a public key infrastructure framework designed to secure the internet’s routing infrastructure in a way that replaces IRRs with a database where trust is assigned by the resource holder. The equivalent of a route object in RPKI is called a ROA (Route Origin Authorisation). It is a cryptographically secure triplet of information which represents a route, the AS that originates it and the maximum prefix length that may be advertised. An example of an IPv4 and an IPv6 ROA would be:

ROAs are typically created through your own RIR (so, RIPE for most INEX members). These RIRs are called trust anchors in RPKI. RIPE have created an extremely easy wizard for creating ROAs through the LIR Portal.

To implement RPKI in a router, the router needs to build and maintain a table of verified ROAs from the five RIRs/trust anchors. The easiest way of doing this is to use a local cache server which pulls and validates the ROAs from the trust anchors and uses a new protocol called RPKI-RTR to feed that information to routers. Currently there are three validators: RIPE’s RPKI Validator v3; Routinator 3000 from NLnetLabs; and Cloudflare’s GoRTR. INEX currently uses the former two.

RPKI validation of a route against the table of ROAs yields one of three possible results:

  • VALID: a ROA exists for the route and both the prefix length is within the allowed range and the origin ASN matches.
  • INVALID: a ROA exists for the route but either (or both) the prefix length is outside the allowed range and/or the origin ASN is different.
  • UNKNOWN: no ROA exists for the route.

UNKNOWN is a common response as the database has only a fraction of the prefix coverage as IRR databases do. We are now in a multi-year transition from IRR to RPKI route validation while ROAs are created.

Bird V2

As well as RPKI support, we have also upgrading all route servers to Bird v2.

This is a significant rewrite to Bird which, for v1, maintained separate code and daemons for IPv4 and IPv6. Bird v2 merges these code bases and also introduces support for new SAFIs such as l3vpns / mpls.

Overall, the configuration changes required were minimal and INEX continues to run separate daemons of Bird v2 for IPv4 and IPv6 daemons. Route servers are CPU intensive and separate daemons allows for maximum stability, keeps the configuration clean and fits into the existing deployment processes we have built up with IXP Manager.

Route Server Filtering Flow

Our work on the new route servers will be released to the community as part of IXP Manager v5 shortly. The new filtering flow is enumerated below. One of the key new features is that if any route fails a step, we use internal large community tagging to indicate this and the specific reason to our members through the IXP Manager looking glass (more on that later).

  1. Filter small prefixes (>/24 for IPv4, >/48 for IPv6).
  2. Filter martian / bogon ranges.
  3. Sanity check to ensure the AS path has at least one ASN and no more than 64.
  4. Sanity check to ensure the peer ASN is the same as first ASN in the prefix’s AS path.
  5. Prevent next-hop hijacking (where a member advertises a route but puts the next hop as another member’s router rather than their own). We do allow same-AS’s to specify their other router(s).
  6. Filter known transit networks.
  7. Ensure that the origin AS is in set of ASNs from member’s AS-SET. See below for some additional detail on this.
  8. RPKI validation. If it is RPKI VALID, accept the route. If it is RPKI INVALID then filter it.
  9. If the route is RPKI UNKNOWN, revert to standard IRRDB filtering.

Regarding step 7 above, an AS-SET is another type of IRRDB database entry where a network which also acts as a transit provider for other networks can enumerate the AS numbers of those downstream networks. This is something RPKI does not yet support but it is being worked on – see AS-Cones.

Lastly we have enhanced the BGP large community support to allow our members request as-path prepends on announcements to specific members over the route servers. For these and more supported communities, see the INEX route server page here.

Bird’s Eye and the Looking Glass

As well as IXP Manager, INEX has also written and open sourced a secure micro service for querying Bird daemons called Bird’s Eye. IXP Manager uses this to provide a web-based looking glass into our route collectors and servers. We have recently released v1.2.1 of Bird’s Eye which adds support for Bird v2.

We have greatly enhanced IXP Manager’s looking glass to support both Bird v2 and the large communities we use to tag filtered reasons. You can explore any of INEX’s route servers to see this yourself – for example this is route server #1 for IPv4 on INEX LAN1. When members log into IXP Manager they will also find a new Filtered Prefixes tool which will summarise any filtered routes across all 12 of INEX’s route server instances.

More Information

We have spoken about this at a number of conferences recently:

A Brief History of IXP Manager

For another INEX project, I was asked to put together a timeline for IXP Manager – an open source application for managing Internet eXchange Points. Reproduced here:

IXP Manager was originally a web portal written in PHP by Nick Hilliard in 2005. It was a basic database frontend that just did fairly simple CRUD (CReate, Update, Delete operations) and allowed our members to log in and view their traffic usage graphs.

Around this was a ton of Perl scripts that sucked that data out of the the database and created configuration files for route collectors, graphing, monitoring, etc.

The major achievement of Nick’s original system was the database design (the schema). The core of that schema is still the core of IXP Manager over 10 years later.

I started in INEX in 2007 and started to expand IXP Manager using what was becoming a more modern web development paradigm – Model/View/Controller with a framework called Zend Framework.

There wasn’t a grand plan here – it was just “as we needed” organic growth over the coming years.

In 2010 we decided what we had was actually pretty good and could be very useful for other IXPs. We got committee approval to open source the software and we released IXP Manager V2 in 2010 under the GPL2 license (GNU Public License v2).

This license essentially means anyone can use the software free of charge but also that they should contribute back improvements that they may make. The idea being that INEX would eventually benefit from other IXPs contributing to the project.

Open sourcing a project doesn’t mean it’ll be successful though! What we didn’t do in 2010 was put infrastructure around it such as: presentations at IXP conferences, mailing lists for user support, decent documentation, etc.

We corrected all that and re-released an updated version called IXP Manager v3 in 2012. This time it took off! We also started collaborating with ISOC (The Internet Society) around this time to help start-up IXPs (mainly eastern Europe and Africa) use IXP Manager.

Some established IXPs also contributed money towards development of missing features – most notably LONAP in the UK – and these new features fed back into INEX.

We’ve worked hard on v3 and it’s developed well since with many new features and improvements. Sometime in late 2016 – maybe even this month – we’ll release v4 which is a major leap forward again and should hopefully attract new users and developers.

INEX is very well regarded in the IXP community as an exchange that is well run and both operates and teaches best practice. All of what we’ve learnt running a good exchange has fed into IXP Manager and it helps those IXPs that use it to implement those same good practices. IXP Manager has helped raise INEX’s reputation even further.

Lately we’ve begun to realise that as a small team we can’t do it all ourselves – the more exchanges that use it, the more requests for help and features we receive and as a result, new developments take a back seat.

To try and improve this we launched a new website in 2016 – http://www.ixpmanager.org/ – and issued a call for sponsorship so we could hire a full time developer. The ‘we’ here by the way is my and Nick’s own company – Island Bridge Networks. We’re doing this on a purely cost recovery basis. I’m delighted to say we’ve just about reached our funding goal with three top line sponsors all contributing about €20k each – ISOC, Netflix and SwissIX. The hiring process has now begun!

I’m also delighted to say that there are 33 exchanges around the world using IXP Manager /that we know of/.

Personal Profile for INEX

I was asked to write a personal profile for INEX in <= 300 words. Reproduced here.

If you want to confuse Barry, ask him where he’s from: born in Cork, spent his formative years in Galway and married into Dublin. He got a honours degree in Maths, his first love, from NUI, Galway in 2001 and went on to do four years research in information theory in UCD’s Computer Science department.

In 2005 he took a job with imag!ne to help build their ADSL broadband service from scratch to a position where it supported tens of thousands of subscribers. Barry branched out on his own in 2007 when he formed his own consultancy business, Open Solutions, which continued working with imag!ne as well as building up a portfolio of network, VoIP and web application development customers.

It was 2008 when Nick Hilliard, INEX’s CTO, approached Barry to provide a couple days operational support to INEX. Little did Barry realise what a huge part of his life INEX would become – both here in Ireland supporting INEX’s infrastructure and membership but also as part of the larger European and international IXP community.

Barry is the lead developer of IXP Manager (http://www.ixpmanager.org/) – a full stack management system for IXPs which includes an administration and customer portal; provides end to end provisioning; and both teaches and implements best practice. INEX is very proud to say that this project is now in use at over 33 IXPs and has grown legs of its own with the wider community sponsoring a full-time developer.

INEX has always been happy to help other IXPs and, through our relationship with the Internet Society (ISOC), RIPE and Euro-IX, Barry has travelled to countries with a less developed internet infrastructure to advise on best practice, has delivered a number of IXP Manager workshops and contributes to policy development.