Upgrading Legacy Versions of IXP Manager

Legacy installations of IXP Manager can be very difficult to upgrade as you can find yourself in a dependency nightmare whereby the old version of IXP Manager will not run on modern versions of PHP; and vice versa.

In case you missed it, we have a new modern website for IXP Manager – find it at https://www.ixpmanager.org/. One of the features of this new website is that we now gather IXP Manager usage statistics on a daily basis – including the distribution of versions in use.

Reassuringly, of the installs we can poll for version used, ~65% are using one of the latest three minor versions (5.5.0 – 5.7.0). This is reassuring for a number of reasons including: knowing that IXPs stay current; knowing that IXPs are concerned about security updates; and knowing that the upgrade process is not especially difficult.

Of the 145 installs we know about, we can poll 116 and collect the version is use which yields the following table:

5.x.y83
4.x.y26
3.x.y7
Distribution of major IXP Manager versions in use as at September 8th 2020.

Legacy installations of IXP Manager can be very difficult to upgrade as you can find yourself in a dependency nightmare whereby the old version of IXP Manager will not run on modern versions of PHP; and vice versa.

Community IX Atlanta (CIX-ATL) are in the process of upgrading from 4.9.3 to the latest (5.7.0) and they graciously allowed me to record the process:

The video is a real-life experience where it wasn’t planned in advance allowing the viewer to see the mistakes and thought processes throughout. Also, if you weren’t aware of it, we have an on-going series of IXP Manager tutorials here.

When considering a legacy upgrade, there are two main approaches:

  1. Build a new IXP Manager installation on a new (modern) server and migrate the database (this is what we’ve done here).
  2. Attempt an in place upgrade alternating between IXP Manager upgrades and operating system upgrades. This is probably more awkward with more scope for issues to crop up (especially on non-IXP Manager applications which may be on the same server).

Remember, what’s covered here is “just” the IXP Manager and database upgrade. There’s a bunch of other things that would also need to be done including:

  • Working through the various upgrade actions in the release notes (mentioned throughout the video). Essentially you’ll need to step through each set of release notes for the versions you cycle through (and jump over).
  • If building a new server, pointing elements such as route server cron jobs and other API consumers at the new server.
  • Migrating other applications from the legacy server (e.g. maybe you have mrtg co-installed there).

In a production environment, my goal would be to build the new IXP Manager installation with the copied and upgraded database and run them in parallel. NB: either avoid or duplicate changes made in the UI across both installations of IXP Manager for this period of time.

Once the new installation of IXP Manager is ready for production use, you will then step through all external tools that consume data from it (sflow, mrtg, route servers, route collectors, etc.) and migrate them to the new installation. Sometimes simply updating DNS can achieve most of this but you’ll probably want to take it piece-meal and ensure each external service works as expected.

Take particular care with essential services such as route servers. This is an opportune time to upgrade to Bird v2 and add RPKI. What we did at INEX was do one route server at a time with 1-2 weeks between upgrades. This allowed time to ensure the new system was stable and also to ensure no member issues due to RPKI filtering, etc. (spoiler alert: it was uneventful!).

As you complete the migration, you can also consider if some services should be left on the “old” server. Separating tasks between different servers is good practice and so ask youself if everything should be migrated over to the new server.

More than anything, I hope this video entices you to keep current with your IXP Manager installations!

Logging Changes in Laravel Eloquent

So you need to log changes to some / all database records, who made them, when and what was added / changed deleted? An easy an effective way to do this with Laravel Eloquent models is via a custom Observable trait. On any model that you wish to track changes for, you then just add the trait to the model (and an optional static function to set the message format):

use App\Traits\Observable;

Let’s start with the migration we need for a table in the database to record all these changes:

public function up() {
  Schema::create('logs', function (Blueprint $table) {
    $table->id();
    $table->unsignedBigInteger('user_id')->nullable();
    $table->string('model',100);
    $table->string('action',7);
    $table->text('message');
    $table->json('models');
    $table->timestamps();

    $table->foreign('user_id')->references('id')->on('users');
  });
}

The Observable trait looks like:

namespace App\Traits;

use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Facades\Auth;

use App\Models\Log;

trait Observable
{
  // bootObservable() will be called on model instantiation automatically
  public static function bootObservable() {
    static::saved(function (Model $model) {
      // create or update?
      if( $model->wasRecentlyCreated ) {
        static::logChange( $model, 'CREATED' );
      } else {
        if( !$model->getChanges() ) {
          return;
        }
        static::logChange( $model, 'UPDATED' );
      }
    });
      
    static::deleted(function (Model $model) {
      static::logChange( $model, 'DELETED' );
    });
  }
  
  public static function logChange( Model $model, string $action ) {
    Log::create([
      'user_id' => Auth::check() ? Auth::user()->id : null,
      'model'   => static::class,
      'action'  => $action,
      'message' => static::logSubject($model),
      'models'  => [
        'new'     => $action !== 'DELETED' ? $model->getAttributes() : null,
        'old'     => $action !== 'CREATED' ? $model->getOriginal()   : null,
        'changed' => $action === 'UPDATED' ? $model->getChanges()    : null,
      ]
    ]);
  }

  /**
   * String to describe the model being updated / deleted / created
   * Override this in the model class
   * @return string
   */
  public static function logSubject(Model $model): string {
    return static::logImplodeAssoc($model->attributesToArray());
  }

  public static function logImplodeAssoc(array $attrs): string {
    $l = '';
    foreach( $attrs as $k => $v ) {
      $l .= "{ $k => $v } ";
    }
    return $l;
  }
}

So, again, just use this trait in any model and you have full logging of changes to the database.

You’ll find complete files for the above and an example usage with the User class on this GitHub Gist.

Using IXP Manager’s Grapher API

We call IXP Manager’s statistics and graphing architecture Grapher. It’s a backend agnostic way to collect and present data. Out of the box, we support MRTG for standard interface graphs, sflow for peer to peer and per-protocol graphs, and Smokeping for latency/packet loss graphs. You can see some of this in action on INEX’s public statistics section.

Internet Exchange Points (IXPs) play a significant role in national internet infrastructures and IXP Manager is used in nearly 100 of these IXPs worldwide. In the last couple weeks we have got a number of queries from those IXPs asking for suggestions on how they can extract traffic data to address queries from their national Governments, regulators, media and members. We just published our own analysis of this for traffic over INEX here.

Grapher has a basic API interface (documented here) which we use to help those IXP Manager users address the queries they are getting. What we have provided to date are mostly quick rough-and-ready solutions but we will pull all these together over the weeks (and months) to come to see which of them might be useful permanent features in IXP Manager.

How to Use These Examples

The code snippets below are expected to be placed in a PHP file in the base directory of your IXP Manager installation (e.g. /srv/ixpmanager) and executed on the command line (e.g. php myscript.php).

Each of these scripts need the following header which is not included below for brevity:

<?php

require 'vendor/autoload.php';

use Carbon\Carbon;

$data = json_decode( file_get_contents( 
    'https://www.inex.ie/ixp/grapher/ixp?period=year&type=log&category=bits' 
) );

We’ve placed a working API endpoint for INEX above – change this for your own IXP / scenario.

Data Volume Growth

An IXP was asked by their largest national newspaper to provide daily statistics of traffic growth due to COVID-19. For historical reasons linked to MRTG graph images, the periods in IXP Manager for this data is such that: day is last 33.3 hours; week is last 8.33 days; month is last 33.33 days; and year is last 366 days.

This is fine within IXP Manager when comparing averages and maximums as we are always comparing like with like. But if we’re looking to sum up the data exchanged in a proper 24hr day then we need to process this differently. For that we use the following loop:

$start = new Carbon('2020-01-01 00:00:00');
$bits = 0;
$last = $data[0][0];
$startu = $start->format('U');
$end = $start->copy()->addDay()->format('U');

foreach( $data as $d ) {
  // if the row is before our start time, skip
  if( $d[0] < $startu ) { $last = $d[0]; continue; }

  if( $d[0] > $end ) {
    // if the row is for the next day break out and print the data 
    echo $start->format('Y-m-d') . ',' 
        . $bits/8 / 1024/1024/1024/1024 . "\n";

    // and reset for next day        
    $bits  = $d[1] * ($d[0] - $last);
    $startu = $start->addDay()->format('U');
    $end    = $start->copy()->addDay()->format('U');
  } else {
    $bits += $d[1] * ($d[0] - $last);
  }

  $last = $d[0];
}

The output is comma-separated (CSV) with the date and data volume exchanged in that 24 hour period (in TBs via 8/1024/1024/1024/1024). This can, for example, be pasted into Excel to create a simple graph:

The elements of the $d[] array mirror what you would expect to find in a MRTG log file (but the data unit represents the API request – e.g. bits/sec, pkts/sec, etc.):

  • d[0] – the UNIX timestamp of the data sample.
  • $d[1] and $d[2] – the average incoming and outgoing transfer rate in bits per second. This is valid for the time between the $d[0] value of the current entry and the $d[0] value of the previous entry. For an IXP where traffic is exchanged, we expect to see $d[1] roughly the same as $d[2].
  • $d[3] and $d[4] – the maximum incoming and outgoing transfer rate in bits per second for the current interval. This is calculated from all the updates which have occured in the current interval. If the current interval is 1 hour, and updates have occured every 5 minutes, it will be the biggest 5 minute transfer rate seen during the hour.

Traffic Peaks

The above snippet uses the average traffic values and the time between samples to calculate the overall volume of traffic exchanged. If you just want to know the traffic peaks in bits/sec on a daily basis, you can do something like this:

$daymax = 0;
$day    = null;

foreach( $data as $d ) {

    $c = ( new Carbon($d[0]) )->format('Y-m-d');

    if( $c !== $day ) {
        if( $day !== null ) {
            echo $day . ',' . $daymax / 1000/1000/1000 . "\n";
        }
        $day = $c;
        $daymax = $d[3];
    } else if( $d[3] > $daymax ) {
        $daymax = $d[3];
    }
}

The output is comma-separated (CSV) with the date and data volume exchanged in that 24 hour period (in Gbps via 1000/1000/1000). This can also be pasted into Excel to create a simple graph:

Import to Carbon / Graphite / Grafana

Something that is on our development list for IXP Manager is to integrate Graphite as a Grapher backend. Using this stack, we could create much more visually appealing graphs as well as time-shift comparisons. In fact this is how we created the graphs for this article on INEX’s website which includes graphs such as:

To create this, we need to get the data into Carbon (Graphite’s time-series database). Carbon accepts data via UDP so we used a script of the form:

foreach( $data as $d ) {
    echo "echo \"inex.ixp.run1 " . $d[1] . " " . $d[0] 
        . "\" | nc <carbon-ip-address> 2003\n";
}

This will output lines like the following which can be piped to sh:

echo "inex.ixp.run1 387495973600 1585649700" | nc -u 192.0.2.23 2003

The Carbon / Graphite / Grafana stack is quite complex so unless you are familiar with it, this option for graphing could prove difficult. To get up and running quickly, we used the docker-grafana-graphite Docker image. Beware that the default graphite/storage-schemas.conf in this image limits data retention to only 7 days.

Using Laraval Eloquent Models for API Results

There’s a very interesting package called calebporzio/sushi for Laravel that allows one to use arrays as Eloquent drivers / sources of data. @calebporzio posted his own example of using this to front API results here.

It’s a very interesting proof of concept for this use case (probably needs more work and more knobs for production use). So interesting, I had a quick look myself with a bare bones Laravel app:

$ laravel new test-sushi
$ cd test-sushi
$ composer require calebporzio/sushi
$ composer require kitetail/zttp
$ php artisan make:model IxpdbProviders

The only interesting part of the model, IxpdbProviders, is the getRows() function:

public function getRows()
{
  return Cache::remember( 'IxpdbProvider::rows', 3600, function() {

    return array_map( function( $a ) {
      foreach( $a as $k => $v ) {
        if( is_array( $v ) ) {
          unset( $a[$k] );
        }
      }
      return $a;
    },
    Zttp::get('https://api.ixpdb.net/v1/provider/list')->json()
  );

});

There’s a few interesting things happening here:

  1. I’m using the cache to store the array result of:
    • the fairly large API response for one hour;
    • the array_map() which is required to remove sub-arrays (sub-objects) within the response as Sushi requires flat rows.
  2. Using Zttp out of curiosity rather than Guzzle directly.
  3. Sushi then takes the array of IXPs (the result of the API call) and stores these in a dedicated in-memory Sqlite database for the duration of the request.

We can now query this as if it were a typical database table:

$ php artisan tinker

>>> App\IxpdbProvider::count();
=> 581

>>> App\IxpdbProvider::where( 'name', 'like', 'inex%')->pluck('name')
=> Illuminate\Support\Collection {#3002
     all: [
       "INEX LAN1",
       "INEX LAN2",
       "INEX Cork",
     ],
   }

2FA and User Session Management in IXP Manager

We’ve just released IXP Manager v5.3.0. The headline feature in this release is two-factor authentication (2fa) and user session management. This blog post overviews the PHP elements on how we did that.

While IXP Manager is a Laravel framework application, it uses Doctrine ORM as its database layer via the Laravel Doctrine bridge. For those curious, this really is a carry over from when IXP Manager was a Zend Framework application. For the migration, we concentrated on the controller and view elements of the MVC stack leaving the model layer on Doctrine. Over time we’ll probably migrate the model layer over to Laravel’s Eloquent.

Before reading on, it would be useful to first read the official documentation we have written aroud 2fa and user session management:

Hopefully the how we did this will be useful for anyone else in the same boat or even just trying to understand the Laravel authentication stack.

Two factor authentication (2fa) strengthens access security by
requiring two methods (also referred to as factors) to verify your
identity. Two factor authentication protects against phishing, social
engineering and password brute force attacks and secures your logins
from attackers exploiting weak or stolen credentials.

User session management allows a user to be logged in and remembered from multiple browsers / devices and to manage those sessions from within IXP Manager.

For 2fa, we used the antonioribeiro/google2fa-laravel package which is built on antonioribeiro/google2fa. If we were 100% in Laravel’s eco-system the would have been easier but because we use Doctrine, we needed to override a number of classes.

Structurally we need a database table to indicate if a user has 2fa enabled and to hold their 2fa secret – for this we created Entities\User2FA. Similarly, we have a controller to handle the UI interaction of enabling, configuring and disabling 2fa: User2FAController – this also includes generating QR codes for the typical 2fa activation process.

On the user session management side, we created Entities\UserRememberToken to hold multiple tokens per user (rather than Laravel’s default single token in a column in the user’s user database entry. For the frontend UI, UserRememberTokenController allows a user to view their active sessions and invalidate (delete) them if required.

The actual mechanism of enforcing 2fa is via middleware: IXP\Http\Middleware\Google2FA. This is added, as appropriate, to web routes via the RouteServiceProvider. This will check the user’s session and if 2fa is enabled but has not been completed, then the middleware will enforce 2fa before granting access to any routes covered by it.

Note that because we also implemented user session management via long-lived cookies and because the fact that a user has passed 2fa or not is held in the session, we need to persistently store the fact in the user’s specific remember token database entry. This is done via the Google2FALoginSucceeded listener. This is then later checked in the SessionGuard – where, if we log a user in via the long-lived cookie, we also make them as having passed 2fa if so set.

Speaking of the SessionGuard, this was one of the bigger changes we had to make – we overrode the Illuminate\Auth\SessionGuard as we needed to replace a few functions to make 2fa and user session management work. We have kept these to a minimum:

  1. The user() function – Laravel’s long lived session uses a single token but we require a token per device / browser. We also need to side-step 2fa for existing sessions as discussed above and allow for features such as allowing a user to delete other long-lived sessions and to provide functionality to allow these sessions to expire.
  2. The ensureRememberTokenIsSet() to actually create per-browser tokens (and to expire old ones).
  3. The queueRecallerCookie() so we can insert our own token rather than the default Laravel version.
  4. The cycleRememberToken() which is actually used to invalidae a token by changing it in Laravel. We override to delete the token.

Similarly we have to override the DoctrineUserProvider class to:

  1. Change retrieveByToken() to use our new database in which a user may have multiple sessions across different browsers / devices.
  2. Add addRememberToken() and purgeExpiredRememberTokens() to add and remove tokens.

We of course had to ammend the AuthServiceProvider to use our new overridden classes.

The above constitutes a bulk to the changes. Because 2fa can be enforced via middleware, it doesn’t really touch the core Laravel authentication process. The user session management was more invasive and responsible for the bulk of the changes required in the DoctrineUserProvider and SessionGuard.

What’s not mentioned above is the views – these are mainly covered in the views/user-remember-token (with a lot of inheritence from views/frontend) and the views/user/2fa directories.

While there are a lot more changes between v5.2.0 and v5.3.0 than 2fa and user session management, you can see the complete set of changes here.