Our Community and IXP Manager

I was delighted to help celebrate Euro-IX’s 20th anniversary today by giving a talk on Our Community and IXP Manager. My goal was to show that collaboration is a two-way street with IXP Manager by taking a look at how our community has helped the project and reflect on how the project helps our community.

I was delighted to help celebrate Euro-IX’s 20th anniversary today by giving a talk on Our Community and IXP Manager. My goal was to show that cooperation is a two-way street with IXP Manager by taking a look at how our community has helped the project and reflect on how the project helps our community. The event was of course online and you can find full details here: Euro-IX – 20 Years of Cooperation.

Euro-IX is my favourite event in our industry and I attended my first one on behalf of INEX when it was in Rome in 2009, accompanying the late Barry Rhodes. Since then, I’ve attended at least one of the two every year.

You can download the slides for the talk here which would be useful to follow some of the content below. There is also a video available here.

IXP Manager: Project Status

I began with a brief update on the current status of the IXP Manager project. An image I and the team at INEX take great pride in sharing is the following world view with 166 green dots – each dot representing an IXP that uses IXP Manager.

We, INEX and the IXP Manager project, launched a new website in June 2020. This site has a database backend that stores the 166 IXPs noted above and generates the above image dynamically. On a nightly basis, it also polls these IXPs in one of two ways:

  1. If the IX-F Member Export is enabled and publically available, we will pull the current data from this. This is the case for 98 of the 166 and this data is real-time and accurate.
  2. For the rest, we’ll use PeeringDB which requires networks (members of IXPs) to assert that they peer at an exchange and so these numbers will be a significant undercount.

The resulting stats, as of 27th June 2021, were presented next:

You can always find a live and current version of these here.

Our Community – Giving to IXP Manager

Now we’re getting into the meat of the presentation by looking at some of the things that the Euro-IX – and wider – community give IXP Manager. The first tangible item is community contributions which mostly cover the costs of a full-time developer. This program is currently in its fifth year. Rather than repeating what I said, please see this page on the website for details.

The next slide was about knowledge. This is possibly an under-discussed and under-valued aspect of Euro-IX in general. Knowledge is a very valuable commodity in most industries that is aggressively protected with NDAs, patents, and contracts. Yet, at Euro-IX forums, the operators of IXPs come together to learn; to create knowledge; and to share knowledge – openly and freely. The best of this community knowledge gets distilled into IXP Manager which, as we always say, teaches and implements best practices – and these practices have in part come from this creation and sharing.

One of the things I’m most proud of in our new IXP Manager website is the contributor wall of fame. Anyone who makes at least five contributions to the project through the GitHub platform will appear here which, as of today’s date, has just over 40 people. Any of the following will count towards a contribution:

  1. Possibly the hardest: contribute code through a pull request.
  2. Open an issue (bug report or feature request).
  3. Simply comment on an existing issue (or your own!).
  4. Contribute editorial fixes/content to the documentation.

This section was rounded out with a shortlist of other sources of community collaboration. The first was the coffee breaks between sessions and the conversations over drinks at the socials at Euro-IX forums. These relaxed environments allowed for more long-form free-flowing discussions with IXP Manager users to help flesh out potential features and to understand where the problems, grit, and rough edges exist. The information learned from these conversations then feeds into later versions of IXP Manager. I miss these face-to-face engagements and, hopefully, we can get back to them soon.

Finally, I wanted to give a specific mention to the open-source community in general. Before I was ever involved in IXPs, INEX, or IXP Manager, I had (and continue to have) an involvement in other open-source projects. For IXP Manager, the FOSS community has given us wide-ranging tools from the PHP language itself to Laravel – the web application framework we use, and right through to front-end libraries such as TailwindCSS and jQuery.

Of course, when we mention open-source in the IXP community, we have to give a nod to Bird, the internet routing daemon, which is the powerhouse of so many of our route server implementations. Hopefully, we’ll also find the time in 2021 to add OpenBGPd support to IXP Manager.

IXP Manager – Giving to Our Community

The is the other side of the two-way street and I wanted to touch on some of the things we hope IXP Manager gives our community. This isn’t about new releases, or functionality or features. And it’s not about being boastful or giving ourselves a pat on the back. It’s aspirational – it’s about what we think – what we hope – IXP Manager gives the community. It’s also not something we can claim success for – it’s up to the community to decide if we’ve succeeded here.

This first element of this was our mission statement – our vision / hope for the project and why we open-sourced it:

Our vision for IXP Manager and the basis for making it a free-to-use, open-source project was that it might enable the creation of IXPs where they are required.

The existence of these IXPs would, in turn, create a stronger, open, more robust and better-connected internet.

This very much goes back to the kind of people involved in IXPs. For most of us, the notion of for the good of the internet is part of our DNA.

The next thing I hope IXP Manager gives the exchanges that use it is independence. The independence to build a local community and form the best and most appropriate community-led IXP for the region in which it exists. This is also an anti-colonialism / anti-imperialism measure as we’ve seen a number of the larger IXPs establish IXPs well outside their original countries and regions. To my mind, in most cases, this is not the best way to build IXPs and usually leads to transactional IXPs rather than communities.

I also like to think of IXPs as facilitators – if not even guardians to an extent – of net neutrality. When you have for-profit IXPs then the business motive is wealth creation. That is the very definition of for-profit. When this is your motivation then the decisions you make may not necessarily be to the benefit of all your members – especially the weaker / smaller members.

Something I’m fairly satisfied that IXP Manager does give new IXPs is the best possible start. As well as a good leg-up for smaller IXPs that move to the platform. The concepts of secure by design and best current practices are baked into what IXP Manager does. Of course, an IXP will always need staff with network engineering and system admin skills – but we hope that the steep learning curve is very much flattened by the use of IXP Manager. To the extent that the challenges in starting a new IXP should no longer be technical but rather environmental: regional political and regulatory issues, commercial issues such as funding and data centre contracts, and finding members.

Finally, we hope IXP Manager gives new IXPs credibility. The reason (so they’ve told us) that Facebook, Amazon AWS and Netflix (historically) have supported IXP Manager through patronage is that they see the value in coming to a new exchange that has installed and is making good use of IXP Manager. These content networks know that much of the technical elements will be “done right”.

There’s also a reverse angle to this – reputation protection for the rest of us. There are only a few hundred active IXPs worldwide and we’ve all worked really hard to build confidence in our abilities – most as community-led not-for-profit enterprises – to deliver secure, reliable services. It only takes a small few bad actors (such as cowboys looking for a quick profit or inexperienced operators) to sow the seeds of doubt. I hope IXP Manager helps put a large insurance blanket over the rest of us by helping these new IXPs start out in the best possible way.

Epilogue

Our Euro-IX community not only supports IXP Manager. We all support each other – through the forums, mailing lists and virtual meet-ups. That we are all so willing to share information and experiences so openly is the wonderful thing about Euro-IX. Not just an association but a community. It’s a community I’m proud to be a part of and call so many of its members by friends. Here’s to the next 20 years!

Upgrading Legacy Versions of IXP Manager

Legacy installations of IXP Manager can be very difficult to upgrade as you can find yourself in a dependency nightmare whereby the old version of IXP Manager will not run on modern versions of PHP; and vice versa.

In case you missed it, we have a new modern website for IXP Manager – find it at https://www.ixpmanager.org/. One of the features of this new website is that we now gather IXP Manager usage statistics on a daily basis – including the distribution of versions in use.

Reassuringly, of the installs we can poll for version used, ~65% are using one of the latest three minor versions (5.5.0 – 5.7.0). This is reassuring for a number of reasons including: knowing that IXPs stay current; knowing that IXPs are concerned about security updates; and knowing that the upgrade process is not especially difficult.

Of the 145 installs we know about, we can poll 116 and collect the version is use which yields the following table:

5.x.y83
4.x.y26
3.x.y7
Distribution of major IXP Manager versions in use as at September 8th 2020.

Legacy installations of IXP Manager can be very difficult to upgrade as you can find yourself in a dependency nightmare whereby the old version of IXP Manager will not run on modern versions of PHP; and vice versa.

Community IX Atlanta (CIX-ATL) are in the process of upgrading from 4.9.3 to the latest (5.7.0) and they graciously allowed me to record the process:

The video is a real-life experience where it wasn’t planned in advance allowing the viewer to see the mistakes and thought processes throughout. Also, if you weren’t aware of it, we have an on-going series of IXP Manager tutorials here.

When considering a legacy upgrade, there are two main approaches:

  1. Build a new IXP Manager installation on a new (modern) server and migrate the database (this is what we’ve done here).
  2. Attempt an in place upgrade alternating between IXP Manager upgrades and operating system upgrades. This is probably more awkward with more scope for issues to crop up (especially on non-IXP Manager applications which may be on the same server).

Remember, what’s covered here is “just” the IXP Manager and database upgrade. There’s a bunch of other things that would also need to be done including:

  • Working through the various upgrade actions in the release notes (mentioned throughout the video). Essentially you’ll need to step through each set of release notes for the versions you cycle through (and jump over).
  • If building a new server, pointing elements such as route server cron jobs and other API consumers at the new server.
  • Migrating other applications from the legacy server (e.g. maybe you have mrtg co-installed there).

In a production environment, my goal would be to build the new IXP Manager installation with the copied and upgraded database and run them in parallel. NB: either avoid or duplicate changes made in the UI across both installations of IXP Manager for this period of time.

Once the new installation of IXP Manager is ready for production use, you will then step through all external tools that consume data from it (sflow, mrtg, route servers, route collectors, etc.) and migrate them to the new installation. Sometimes simply updating DNS can achieve most of this but you’ll probably want to take it piece-meal and ensure each external service works as expected.

Take particular care with essential services such as route servers. This is an opportune time to upgrade to Bird v2 and add RPKI. What we did at INEX was do one route server at a time with 1-2 weeks between upgrades. This allowed time to ensure the new system was stable and also to ensure no member issues due to RPKI filtering, etc. (spoiler alert: it was uneventful!).

As you complete the migration, you can also consider if some services should be left on the “old” server. Separating tasks between different servers is good practice and so ask youself if everything should be migrated over to the new server.

More than anything, I hope this video entices you to keep current with your IXP Manager installations!

Using IXP Manager’s Grapher API

We call IXP Manager’s statistics and graphing architecture Grapher. It’s a backend agnostic way to collect and present data. Out of the box, we support MRTG for standard interface graphs, sflow for peer to peer and per-protocol graphs, and Smokeping for latency/packet loss graphs. You can see some of this in action on INEX’s public statistics section.

Internet Exchange Points (IXPs) play a significant role in national internet infrastructures and IXP Manager is used in nearly 100 of these IXPs worldwide. In the last couple weeks we have got a number of queries from those IXPs asking for suggestions on how they can extract traffic data to address queries from their national Governments, regulators, media and members. We just published our own analysis of this for traffic over INEX here.

Grapher has a basic API interface (documented here) which we use to help those IXP Manager users address the queries they are getting. What we have provided to date are mostly quick rough-and-ready solutions but we will pull all these together over the weeks (and months) to come to see which of them might be useful permanent features in IXP Manager.

How to Use These Examples

The code snippets below are expected to be placed in a PHP file in the base directory of your IXP Manager installation (e.g. /srv/ixpmanager) and executed on the command line (e.g. php myscript.php).

Each of these scripts need the following header which is not included below for brevity:

<?php

require 'vendor/autoload.php';

use Carbon\Carbon;

$data = json_decode( file_get_contents( 
    'https://www.inex.ie/ixp/grapher/ixp?period=year&type=log&category=bits' 
) );

We’ve placed a working API endpoint for INEX above – change this for your own IXP / scenario.

Data Volume Growth

An IXP was asked by their largest national newspaper to provide daily statistics of traffic growth due to COVID-19. For historical reasons linked to MRTG graph images, the periods in IXP Manager for this data is such that: day is last 33.3 hours; week is last 8.33 days; month is last 33.33 days; and year is last 366 days.

This is fine within IXP Manager when comparing averages and maximums as we are always comparing like with like. But if we’re looking to sum up the data exchanged in a proper 24hr day then we need to process this differently. For that we use the following loop:

$start = new Carbon('2020-01-01 00:00:00');
$bits = 0;
$last = $data[0][0];
$startu = $start->format('U');
$end = $start->copy()->addDay()->format('U');

foreach( $data as $d ) {
  // if the row is before our start time, skip
  if( $d[0] < $startu ) { $last = $d[0]; continue; }

  if( $d[0] > $end ) {
    // if the row is for the next day break out and print the data 
    echo $start->format('Y-m-d') . ',' 
        . $bits/8 / 1024/1024/1024/1024 . "\n";

    // and reset for next day        
    $bits  = $d[1] * ($d[0] - $last);
    $startu = $start->addDay()->format('U');
    $end    = $start->copy()->addDay()->format('U');
  } else {
    $bits += $d[1] * ($d[0] - $last);
  }

  $last = $d[0];
}

The output is comma-separated (CSV) with the date and data volume exchanged in that 24 hour period (in TBs via 8/1024/1024/1024/1024). This can, for example, be pasted into Excel to create a simple graph:

The elements of the $d[] array mirror what you would expect to find in a MRTG log file (but the data unit represents the API request – e.g. bits/sec, pkts/sec, etc.):

  • d[0] – the UNIX timestamp of the data sample.
  • $d[1] and $d[2] – the average incoming and outgoing transfer rate in bits per second. This is valid for the time between the $d[0] value of the current entry and the $d[0] value of the previous entry. For an IXP where traffic is exchanged, we expect to see $d[1] roughly the same as $d[2].
  • $d[3] and $d[4] – the maximum incoming and outgoing transfer rate in bits per second for the current interval. This is calculated from all the updates which have occured in the current interval. If the current interval is 1 hour, and updates have occured every 5 minutes, it will be the biggest 5 minute transfer rate seen during the hour.

Traffic Peaks

The above snippet uses the average traffic values and the time between samples to calculate the overall volume of traffic exchanged. If you just want to know the traffic peaks in bits/sec on a daily basis, you can do something like this:

$daymax = 0;
$day    = null;

foreach( $data as $d ) {

    $c = ( new Carbon($d[0]) )->format('Y-m-d');

    if( $c !== $day ) {
        if( $day !== null ) {
            echo $day . ',' . $daymax / 1000/1000/1000 . "\n";
        }
        $day = $c;
        $daymax = $d[3];
    } else if( $d[3] > $daymax ) {
        $daymax = $d[3];
    }
}

The output is comma-separated (CSV) with the date and data volume exchanged in that 24 hour period (in Gbps via 1000/1000/1000). This can also be pasted into Excel to create a simple graph:

Import to Carbon / Graphite / Grafana

Something that is on our development list for IXP Manager is to integrate Graphite as a Grapher backend. Using this stack, we could create much more visually appealing graphs as well as time-shift comparisons. In fact this is how we created the graphs for this article on INEX’s website which includes graphs such as:

To create this, we need to get the data into Carbon (Graphite’s time-series database). Carbon accepts data via UDP so we used a script of the form:

foreach( $data as $d ) {
    echo "echo \"inex.ixp.run1 " . $d[1] . " " . $d[0] 
        . "\" | nc <carbon-ip-address> 2003\n";
}

This will output lines like the following which can be piped to sh:

echo "inex.ixp.run1 387495973600 1585649700" | nc -u 192.0.2.23 2003

The Carbon / Graphite / Grafana stack is quite complex so unless you are familiar with it, this option for graphing could prove difficult. To get up and running quickly, we used the docker-grafana-graphite Docker image. Beware that the default graphite/storage-schemas.conf in this image limits data retention to only 7 days.

2FA and User Session Management in IXP Manager

We’ve just released IXP Manager v5.3.0. The headline feature in this release is two-factor authentication (2fa) and user session management. This blog post overviews the PHP elements on how we did that.

While IXP Manager is a Laravel framework application, it uses Doctrine ORM as its database layer via the Laravel Doctrine bridge. For those curious, this really is a carry over from when IXP Manager was a Zend Framework application. For the migration, we concentrated on the controller and view elements of the MVC stack leaving the model layer on Doctrine. Over time we’ll probably migrate the model layer over to Laravel’s Eloquent.

Before reading on, it would be useful to first read the official documentation we have written aroud 2fa and user session management:

Hopefully the how we did this will be useful for anyone else in the same boat or even just trying to understand the Laravel authentication stack.

Two factor authentication (2fa) strengthens access security by
requiring two methods (also referred to as factors) to verify your
identity. Two factor authentication protects against phishing, social
engineering and password brute force attacks and secures your logins
from attackers exploiting weak or stolen credentials.

User session management allows a user to be logged in and remembered from multiple browsers / devices and to manage those sessions from within IXP Manager.

For 2fa, we used the antonioribeiro/google2fa-laravel package which is built on antonioribeiro/google2fa. If we were 100% in Laravel’s eco-system the would have been easier but because we use Doctrine, we needed to override a number of classes.

Structurally we need a database table to indicate if a user has 2fa enabled and to hold their 2fa secret – for this we created Entities\User2FA. Similarly, we have a controller to handle the UI interaction of enabling, configuring and disabling 2fa: User2FAController – this also includes generating QR codes for the typical 2fa activation process.

On the user session management side, we created Entities\UserRememberToken to hold multiple tokens per user (rather than Laravel’s default single token in a column in the user’s user database entry. For the frontend UI, UserRememberTokenController allows a user to view their active sessions and invalidate (delete) them if required.

The actual mechanism of enforcing 2fa is via middleware: IXP\Http\Middleware\Google2FA. This is added, as appropriate, to web routes via the RouteServiceProvider. This will check the user’s session and if 2fa is enabled but has not been completed, then the middleware will enforce 2fa before granting access to any routes covered by it.

Note that because we also implemented user session management via long-lived cookies and because the fact that a user has passed 2fa or not is held in the session, we need to persistently store the fact in the user’s specific remember token database entry. This is done via the Google2FALoginSucceeded listener. This is then later checked in the SessionGuard – where, if we log a user in via the long-lived cookie, we also make them as having passed 2fa if so set.

Speaking of the SessionGuard, this was one of the bigger changes we had to make – we overrode the Illuminate\Auth\SessionGuard as we needed to replace a few functions to make 2fa and user session management work. We have kept these to a minimum:

  1. The user() function – Laravel’s long lived session uses a single token but we require a token per device / browser. We also need to side-step 2fa for existing sessions as discussed above and allow for features such as allowing a user to delete other long-lived sessions and to provide functionality to allow these sessions to expire.
  2. The ensureRememberTokenIsSet() to actually create per-browser tokens (and to expire old ones).
  3. The queueRecallerCookie() so we can insert our own token rather than the default Laravel version.
  4. The cycleRememberToken() which is actually used to invalidae a token by changing it in Laravel. We override to delete the token.

Similarly we have to override the DoctrineUserProvider class to:

  1. Change retrieveByToken() to use our new database in which a user may have multiple sessions across different browsers / devices.
  2. Add addRememberToken() and purgeExpiredRememberTokens() to add and remove tokens.

We of course had to ammend the AuthServiceProvider to use our new overridden classes.

The above constitutes a bulk to the changes. Because 2fa can be enforced via middleware, it doesn’t really touch the core Laravel authentication process. The user session management was more invasive and responsible for the bulk of the changes required in the DoctrineUserProvider and SessionGuard.

What’s not mentioned above is the views – these are mainly covered in the views/user-remember-token (with a lot of inheritence from views/frontend) and the views/user/2fa directories.

While there are a lot more changes between v5.2.0 and v5.3.0 than 2fa and user session management, you can see the complete set of changes here.

Migrating Legacy Web Applications to Laravel

Originally published in php[architect] Magazine, March 2019 issue. [PDF] and discussed in Building Bridges (podcast), php[podcast] – The Official Podcast of php[architect], March 25th 2019. [Official Link] [Local Copy]


Thanks to Taylor Otwell’s Laravel framework, PHP is reclaiming its rightful place as the go-to language for web application development. For those of us maintaining and developing applications using legacy frameworks, the grass certainly looks greener on Laravel’s side. In this article, I will show you how to do an in-place migration from a legacy framework to Laravel.

Introduction

IXP Manager is an open source tool we developed at INEX for managing IXPs (internet exchange points – network switching centers which facilitate the regional exchange of internet traffic between different networks). It has run on Zend Framework V1 (ZF1) since 2008.

As most readers will know, ZF1 went end-of-life in 2016, but its obituary was written a couple years before that. In 2015, we released V4 of IXP Manager which was a framework transition release. Over the course of nine minor releases of V4, we migrated from ZF1 to Laravel finally completing the project with V4.9 released in January of 2019.

Admittedly, a two and half year transition sounds like a long time but this was an in-place migration where Laravel handled new and migrated controllers while anything still to be migrated fell back on ZF1. You should also note the IXP Manager project has a single full-time developer plus me when time allows.

The Approach

There are two possible approaches to migrating your application to Laravel: a flag-day or an in-place/side-by-side migration.

Your gut feeling may lean towards a flag day – let’s just get this done” – but it is the more drastic path. It means pausing all feature development and rewriting the application completely. In any project, commercial or open source, this is a very difficult argument to make. For a commercial project, it puts a real cost on the migration: ( number of developers * monthly salary * n months ) + the opportunity cost of the development freeze where n will realistically be six months at an absolute minimum. This will be very difficult to get approved by the higher-ups! Plus, have you ever met a development project that finished on time? That six months will creep to a year and even beyond very quickly.

With the in-place migration, we add Laravel to our application so that it has the first opportunity to service a request (route). Otherwise, it hands off to the legacy framework. This has two immediate advantages: you can develop all new features immediately on Laravel as well as use Laravel features and facades within the legacy framework. It also means you can migrate legacy controllers on a case-by-case basis as time and resources allow. Migrating the smaller/easier legacy controllers are also excellent projects for interns, student work experience or new hires getting up to speed. The true cost is buried in day-to-day development, there’s no promised flag-day deadline to miss, and there’s no frustrating feature freeze.

Making the Case

Part of making the case to fellow developers and decision makers in your organization is being able to reference that Laravel is now the number one web application framework on GitHub – across all languages. Other important arguments include:

  1. Prevent developer apathy: or, better phrased for management, retain key employees and attract more developers. Let’s face it, as developers, we prefer to engage in projects that use current frameworks and which support modern versions of PHP (i.e., greater than or equal to 7.1).
  2. You will have to eventually: this is a corollary of the above point. If you do not migrate to a modern framework, then you will inevitably face each of the following consequences. You will hemorrhage employees/developers, and your code will grow more outdated and consequently prove more difficult and costly to upgrade eventually. You’ll be running on frameworks that have passed end-of-life and end-of-support which means security holes will be discovered but remain unpatched and you’ll be forced to run older operating systems to run older versions of PHP for framework compatibility yielding yet more known but unpatched security holes.
  3. Develop with modern techniques and services: Laravel makes it incredibly easy to use modern features such as job queues, an integrated command line interface, broadcasting, caching, events with listeners, scheduling, modern templating engine, database abstraction and ORM, and more.
  4. Reference applications: refer to projects that successfully demonstrate an in-place migration including IXP Manager which supports critical internet infrastructure in 70 locations around the world and has successfully completed the migration, and LibreNMS, a hugely popular network monitoring system with thousands of installations that is also well along the path of replacing a custom framework with Laravel.

Prerequisites

Before you start the process of integrating Laravel for an in place migration, you need to ensure your existing application is ready for it.

Your legacy application needs to use Composer, a dependency manager for PHP. If you are not using it already, it will need to be integrated into your application by using autoloaders (classmap, psr-0/4) for existing namespaces (whether modern PHP namespaces or the Zend_ type prefix).

Your application should have a single point of entry (e.g., index.php). If it doesn’t, you can create an index.php to handle this by (carefully and securely) examining the $_REQUEST object and running the requested script from a new index.php.

Your application entry point should exist in a dedicated subdirectory such as public/ – i.e., the framework and other PHP files should not be exposed by your web server. This should be fairly easy to retrofit if not already in place.

The Migration

Step One: Install Laravel

The first step is to install the Laravel application base files alongside your existing application files. Begin by installing Laravel using its own documentation into a separate directory and then move the files over to your application root directory in a piecemeal fashion.

You will need to resolve any filename or directory conflicts, and you should do this by moving your own files out of the way and renaming or refactoring them rather than altering Laravel’s files. The level of effort here will be framework dependent, but the good news is it was very easy for ZF1. I also looked at the file and directory structures for CodeIgniter and Symfony, and both also seem like they shouldn’t pose any significant problems. Lastly, if you are running a custom or non-application framework (LibreNMS was in this category), you will still be able to use the technique I am demonstrating here. Continue reading and pay particular attention to moving but keeping your index.php in step two below.

When you complete the file moves as shown by example in Listing 1, examine any files remaining in the Laravel directory and move them if necessary/desired. Also, note the example was based on Laravel v5.7 so your mileage may vary for other versions.

# Get the Laravel files from GitHub:
git clone https://github.com/laravel/laravel.git

# Switch to the version of Laravel you want to migrate to:
cd laravel
git checkout vx.y.z

# Assuming you are in the new Laravel app directory above
# and your legacy application is located at ../legacyapp

# You can start to move the files as follows (and feel free
# to break this into smaller steps if there are conflicts):
mv app/ artisan bootstrap/ config/ database/ package.json \
   phpunit.xml resources/ routes/ server.php storage/     \
   tests/ webpack.mix.js    ../legacyapp

mv .env.example ../legacyapp/.env
mv public/js/app.js ../legacyapp/public/js
mv public/css/app.css ../legacyapp/public/css

# For now, we ignore public/index.php and we do not need
# any of composer.json, readme.md, vendor/ or CHANGELOG.md

As well as the base Laravel files, you also need the actual Laravel framework and supporting packages. Integrate the lines shown in Listing 2 to your composer.json file (ensuring you match this to your version of Laravel).

{
    "require": {
        "fideloper/proxy": "^4.0",
        "laravel/tinker": "^1.0",
        "laravel/framework": "5.7.*"
    },
    "require-dev": {
        "beyondcode/laravel-dump-server": "^1.0",
        "filp/whoops": "^2.0",
        "fzaninotto/faker": "^1.4",
        "mockery/mockery": "^1.0",
        "nunomaduro/collision": "^2.0",
        "phpunit/phpunit": "^7.0"
    },
    "autoload": {
        "psr-4": {
            "App\\": "app/"
        },
        "classmap": [
            "database/seeds",
            "database/factories"
        ]
    },
    "autoload-dev": {
        "psr-4": {
            "Tests\\": "tests/"
        }
    }
}

You should now run composer update to install Laravel and its dependencies. You should also examine the other sections of Laravel’s composer.json file including the config, extra, and scripts sections and copy them across.

Before you proceed any further, you should check that your legacy application continues to work as expected. While we have installed Laravel’s files and supporting libraries, we have not changed index.php so your application should run as it always has. If you have integration tests, they can really shine here. If you don’t, consider writing them as you port functionality over to Laravel. Diagnose and fix any issues now.

Step Two: Activate Laravel as the Default Framework

You need to verify you successfully completed Step One. To do this, move your index.php out of the way (e.g., mv index.php legacy_index.php) and copy over Laravel’s own index.php to replace it. Ensure Laravel starts up instead of your own legacy application. If it works, you will see the standard Laravel application welcome page. If this does not work, diagnose and fix those issues now.

When finished, leave Laravel’s index.php in place. The handoff to the legacy framework will happen within the Laravel application and not index.php.

Step Three: Hand Off to Legacy Framework

There are two ways to hand off to the legacy framework I have seen in use: the way we did it with IXP Manager via a 404 error handler and the way LibreNMS did it using a catch all route. I will show you both methods here, and you can choose which suits you.

Using a 404 Handler

In Laravel, if a route does not exist to handle a request, it throws a 404 exception. In Laravel v5.7, this gets handled in app/Exceptions/Handler.php:

class Handler extends ExceptionHandler {
    // ...

    public function render($request, Exception $exception)
    {
        return parent::render($request, $exception);
    }
}

We augment this render() function to handle 404 exceptions differently by handing them off to the legacy framework – here’s a skeleton example.

use Symfony\Component\HttpKernel\Exception\{
    NotFoundHttpException };

public function render($request, Exception $exception) {
   if( $exception instanceof NotFoundHttpException ) {
      // pass to legacy framework - contents of index.php
      die();
   }
}

Before we fill in the detail of pass to legacy framework contents of index.php above, we need to decide how to actually handoff. We could just jam in the contents of legacy_index.php and it would work fine. But as we migrate more and more elements to Laravel, we’ll find various complications that make this unwieldy. A better way to handle the legacy framework within Laravel is to treat it as a service provider. For example, we could create a file app/Providers/ZendFrameworkServiceProvider.php as shown in Listing 3.

class ZendFrameworkServiceProvider extends ServiceProvider{
   protected $defer = true;

   public function register() {
      $this->app->singleton("ZendFramework",function($app){

         // here are the contents of the legacy index.php:
         require_once “Zend/Application.php”;
         $zf = new Zend_Application(
            $app->environment(), $this->createOptions()
         );

         return $zf->bootstrap();
      });
   }
}

IXP Manager’s actual production version of this can be seen here in our v4.8 GitHub tree. You should note we have completely removed Zend’s own configuration INI files at this point and instead take the configuration directly from Laravel’s config/ files. This is then passed into the legacy framework as an array. Our application only has one configuration mechanism (more on this later).

Also, to make require_once "Zend/Application.php" work, we installed the ZF1 library via Composer. As mentioned above, you can use classmaps, psr-0, or psr-4 within Composer to ensure Laravel can resolve your legacy application’s namespace.

Do not forget to register the new service provider in config/app.php:

   'providers' => [
      // ...
      App\Providers\ZendFrameworkServiceProvider::class,
      // ...
   ],

Now that we have our legacy framework service provider, we can return to the 404 exception handler’s (app/Exceptions/Handler.php) render() function and fill in the missing piece:

// Render an exception into a HTTP response
public function render( $request, Exception $exception ) {
  if( $exception instanceof NotFoundHttpException ) {
    // pass to legacy framework
    App::make("ZendFramework")->run();
    die(); // prevent Laravel sending a 404 response
  }
}

There are some great advantages to using a service provider and putting Laravel first:

  • You can use all of Laravel’s facades immediately in your legacy code (e.g., Cache::, Queue::, Mail::, etc.).
  • You can migrate code on an action by action basis rather than controller by controller or even have Laravel handle new action based requests for existing legacy controllers.
  • you can eventually cleanly and simply remove the legacy framework by removing the 404 handler lines, the entry in config/app.php, legacy related packages from composer.json, and the legacy service provider.

Using a Default Route

This is how the LibreNMS project handled the side-by-side migration. At the end of Laravel’s routes/web.php file, they added:

// Legacy Framework Routes
Route::any( "/{path?}", "LegacyController@index" )
    ->where( "path", ".*" );

This catches all routes not having a specific previous match in Laravel in the same way the 404 handler does. They then hand off to to the legacy framework in a controller (app/Http/Controllers/LegacyController.php) as follows:

namespace App\Http\Controllers;
class LegacyController extends Controller {
    public function index($path = "") {
        ob_start();
        include base_path("html/legacy_index.php");
        return response( ob_get_clean() );
    }
}

This will also work, but be aware you’ve entered Laravel’s HTTP kernel handling, and loaded and run all middleware associated with the web routes. This can be useful in some circumstances, but the 404 handler method will generally be more efficient.

Continuing the Migration

You can now proceed with the migration on a controller by controller basis (or action by action) along with the views and models as necessary.

Other Considerations

Parallel Configurations

For our own project, our users download, install, and maintain it themselves. As the in-place migration went on for two years, it would have been completely unreasonable – and downright confusing – to ask those users to configure and maintain settings in two different places and using two different methods (ZF1’s application.ini file and Laravel’s .env).

Instead, we chose to configure everything in Laravel from the beginning. In our ZendFrameworkServiceProvider we then build an array using Laravel’s config() function in the same format ZF1 would have when reading the application.ini file. This array is then passed as a parameter when instantiating the legacy service provider. We already provided a link to the production version of this file in GitHub above.

If your application is an in-house enterprise system or a cloud-based hosted service, this may not be an issue for you. But if you expect your end users to install and configure the application, switching to use Laravel configuration only and passing that to the legacy framework is definitely the developer-friendly choice.

Session Management

I was quite worried about this one from the outset and had nightmares of the legacy framework and Laravel tripping over each other in PHP’s default session management system. Then, I discovered these comments within Laravel’s session middleware framework files:

// If a session driver has been configured, we will need to
// start the session here so that the data is ready for an
// application. Note that the Laravel sessions do not make
// use of PHP "native" sessions in any way since they are
// crappy.

I won’t start an argument on whether the statement is true or not, but from a migration point of view, it’s a really useful position for Laravel to take. Essentially, as Laravel implements its own cookie-based session management system, there are no conflicts with any other legacy frameworks. It essentially just works.

If you need to access the Laravel session in your legacy code, you can use the Session:: facade.

User Authentication

Frameworks typically handle user authentication using sessions. As Laravel has its own session management system, our goal is to ensure when a user has logged into one framework, they are logged into the other framework (and same for logging out).

We choose to leave the migration of the authentication controller until last – there was no particular reason for this, but it was going to be the first thing we did or the last. In the end, we just felt it was one of the more complex systems, and it would be easier to start with some of the simpler controllers. This meant we needed to ensure we logged into Laravel if we were logged into ZF1 (and logged out as appropriate).

There are a few ways (and places) to handle this. We chose to add a block of temporary code to the top of routes/web.php as it is executed on every request and it is a file that is edited regularly so we could be confident we would also remember to remove it when the migration was complete. It looked like this:

if( php_sapi_name() !== "cli" ) {
    $auth = Zend_Auth::getInstance();
    if( $auth->hasIdentity() && Auth::guest() ) {
        Auth::login( App\User::findOrFail( $auth->getId() ) );
    } else if( !$auth->hasIdentity() ) {
        Auth::logout();
    }
}

First, we do not run the code if we are running on the CLI (e.g., an Artisan command).

The if() statement says if we are logged into ZF1 and not Laravel, then log into Laravel. Conversely, the else if() asks if we are not logged into ZF1 then ensure we are also not logged into Laravel.

When the time comes to plan the migration of the authentication system, it is an opportune moment to consider other enhancements including:

  • integrate Laravel Socialite which allows users to log in with OAuth providers such as Facebook, Twitter, LinkedIn, Google, GitHub, GitLab, and many more;
  • add 2-factor authentication;
  • add Log in As functionality which is useful for diagnosing issues as end users see them (see the viacreative/sudo-su package for a good example of this);
  • and, of course, upgrade password hashing to bcrypt/Argon2.

Duplication of Views/Templates

One of the more significant headaches of an in-place migration is having to duplicate your layout views (menus, headers, footers, etc.) and maintain both versions during the process. When you do this, you will want to keep the new Laravel view template layouts as close to the legacy ones as possible. This ensures your end users will not realize two frameworks are running the backend.

This doesn’t mean you can’t modernize the frontend libraries. For example, you could still upgrade from Bootstrap v2 to Bootstrap v4 and smooth out the differences with custom CSS.

Also, as you migrate actions and controllers, don’t forget to update links in both sets of views.

ORM/Database Model Migration

Laravel has a very nice ORM called Eloquent. It also has its own DBAL (database abstraction layer) on which Eloquent is built. As you migrate the legacy application, you will also need to consider how best to migrate the legacy database code.

If you have been using PHP’s mysql_* functions directly or have built up a custom library to wrap the usage of these functions, you should just bite the bullet and move to Eloquent as you migrate.

Our situation with IXP Manager was a little different as we migrated to Doctrine2 in 2012 so we were already using a high-performance modern ORM library. Rather than try and migrate this, we were fortunate the Laravel Doctrine project provides a drop-in Doctrine2 implementation for Laravel 5+. This allowed us to use our Doctrine entities and repositories in both the legacy and Laravel framework in parallel.

Tracking Progress

This is accomplished by watching the number of legacy controllers (or files) you still have to reduce with each iteration. As each action/controller is migrated, the legacy code should be removed. This is accompanied by a nice endorphin release when you commit and push deletions of legacy code!

The typical decision to migrate a controller was either:

  • no pressing feature requests so pick off the next controller and migrate it; or,
  • there was a required new feature in a legacy controller, so the feature was implemented in Laravel as part of the process of migrating the controller.

About 18 months into the IXP Manager migration project, we estimated we were about 75 percent of the way to removing ZF1. The remaining legacy controllers were static code which were rarely touched. To bring it over the line, we put two months of concentrated effort into this while still not neglecting other smaller improvements, bug reports, and feature requests.

Summary

This article is a write-up of a talk I gave at the Laravel Live UK conference in 2018. Let me close with some encouragement: while migrating a large application to a new framework is a daunting and time-consuming task, it is possible. IXP Manager has roughly 85k lines of PHP code, and we got through it with a single full-time developer in a little over two years while still adding and improving features.

Please feel free to reach out to me on @barryo79 with comments and questions.

Upgrading to PHP 7.3 on Ubuntu Bionic 18.04 LTS

Ubuntu 18.04 ships with PHP 7.2 by default but there are various reasons why you may wish to upgrade to newer versions. For example, active support for it ends later this year – far sooner than the 2023 support window for the OS.

In addition, applications will be released that will require newer versions in that 2018 – 2023 window. For IXP Manager, we are releasing v5 this month and mandating PHP 7.3 support. We do this to stay current and to prevent developer apathy – insisting on legacy frameworks and packages that have been EOL’d provides a major stumbling block for bringing on new developers and contributors. There’s also a real opportunity cost – I have a couple free hours, will I work on project A or project B? If project A uses an old stale toolchain where everything is that much more awkward that project B then which would you choose?

So, from a typical LAMP stack install of Ubuntu 18.04, you’ll find something like the following packages for PHP:

root@ubuntu:/var/www/html# dpkg -l | grep php | cut - -b 1-65
 ii  libapache2-mod-php                    1:7.2+60ubuntu1
 ii  libapache2-mod-php7.2                 7.2.17-0ubuntu0.18.04.1
 ii  php-common                            1:60ubuntu1
 ii  php-mysql                             1:7.2+60ubuntu1
 ii  php7.2-cli                            7.2.17-0ubuntu0.18.04.1
 ii  php7.2-common                         7.2.17-0ubuntu0.18.04.1
 ii  php7.2-json                           7.2.17-0ubuntu0.18.04.1
 ii  php7.2-mysql                          7.2.17-0ubuntu0.18.04.1
 ii  php7.2-opcache                        7.2.17-0ubuntu0.18.04.1
 ii  php7.2-readline                       7.2.17-0ubuntu0.18.04.1

Obviously your exact list will vary depending on what you installed. I find the easiest way to upgrade is to start by removing all installed PHP packages. Based on the above:

dpkg -r libapache2-mod-php libapache2-mod-php7.2 php-common   \
  php-mysql php7.2-cli php7.2-common php7.2-json php7.2-mysql \
  php7.2-opcache php7.2-readline

The goto place for current versions of PHP on Ubuntu is OndÅ™ej Surý’s PPA (Personal Package Archive). OndÅ™ej maintains this in his own time so don’t be afraid to tip him here.

It’s easy to add this to 18.04 as follows:

add-apt-repository ppa:ondrej/php
apt-get update

Then install the PHP 7.3 packages you want / need. For example we can just take the package removal line above and install the 7.3 equivalents with:

apt install libapache2-mod-php libapache2-mod-php7.3 php-common \
    php-mysql php7.3-cli php7.3-common php7.3-json php7.3-mysql \
    php7.3-opcache php7.3-readline

And voilà:

php -v
 PHP 7.3.5-1+ubuntu18.04.1+deb.sury.org+1 (cli) (built: May  3 2019 10:00:24) ( NTS )

One post-installation check is to replicate and custom php.ini changes you may have made (max upload size, max post size, max memory usage, etc.).

UI Tests with Laravel Dusk for IXP Manager

We use standard PHPUnit tests for IXP Manager for some mission critical aspects. These take data from a test database filled with known sample data (representing a range of different member configurations). The tests then use this information to generate configurations and compare these against known-good configurations.

This way, we know that for a given set of data, we will get a predictable output so long as we haven’t accidentally broken anything in development.

But, as an end user, how do you know that what you stick in a web-based form gets put into the database correctly? And conversely, how do you know that form represents the database data correctly when editing?

This is an issue we ran into recently around some checkbox logic and a dropdown showing the wrong selected element. These issues are every bit as dangerous to mission critical elements as the output tests we do with PHPUnit.

To test the frontend, we turn to Laravel Duskan expressive, easy-to-use browser automation and testing API. What this actually means is that we can right code like this:

$this->browse( function( $browser ) use ( $user ) {
    $browser->visit('/login')
        ->type( 'username', $user->email )
        ->type( 'password', 'secret' )
        ->press('Login')
        ->assertPathIs('/home')
        ->assertSee( 'Welcome to IXP Manager ' . $user->name );

We have now added Dusk tests for UI elements that involve adding, editing and deleting all aspects of a member interface and all aspects of adding, editing and delete a router object. Here’s an example of the latter:

Laravel Dusk - Animated Gif Example

A Brief History of IXP Manager

For another INEX project, I was asked to put together a timeline for IXP Manager – an open source application for managing Internet eXchange Points. Reproduced here:

IXP Manager was originally a web portal written in PHP by Nick Hilliard in 2005. It was a basic database frontend that just did fairly simple CRUD (CReate, Update, Delete operations) and allowed our members to log in and view their traffic usage graphs.

Around this was a ton of Perl scripts that sucked that data out of the the database and created configuration files for route collectors, graphing, monitoring, etc.

The major achievement of Nick’s original system was the database design (the schema). The core of that schema is still the core of IXP Manager over 10 years later.

I started in INEX in 2007 and started to expand IXP Manager using what was becoming a more modern web development paradigm – Model/View/Controller with a framework called Zend Framework.

There wasn’t a grand plan here – it was just “as we needed” organic growth over the coming years.

In 2010 we decided what we had was actually pretty good and could be very useful for other IXPs. We got committee approval to open source the software and we released IXP Manager V2 in 2010 under the GPL2 license (GNU Public License v2).

This license essentially means anyone can use the software free of charge but also that they should contribute back improvements that they may make. The idea being that INEX would eventually benefit from other IXPs contributing to the project.

Open sourcing a project doesn’t mean it’ll be successful though! What we didn’t do in 2010 was put infrastructure around it such as: presentations at IXP conferences, mailing lists for user support, decent documentation, etc.

We corrected all that and re-released an updated version called IXP Manager v3 in 2012. This time it took off! We also started collaborating with ISOC (The Internet Society) around this time to help start-up IXPs (mainly eastern Europe and Africa) use IXP Manager.

Some established IXPs also contributed money towards development of missing features – most notably LONAP in the UK – and these new features fed back into INEX.

We’ve worked hard on v3 and it’s developed well since with many new features and improvements. Sometime in late 2016 – maybe even this month – we’ll release v4 which is a major leap forward again and should hopefully attract new users and developers.

INEX is very well regarded in the IXP community as an exchange that is well run and both operates and teaches best practice. All of what we’ve learnt running a good exchange has fed into IXP Manager and it helps those IXPs that use it to implement those same good practices. IXP Manager has helped raise INEX’s reputation even further.

Lately we’ve begun to realise that as a small team we can’t do it all ourselves – the more exchanges that use it, the more requests for help and features we receive and as a result, new developments take a back seat.

To try and improve this we launched a new website in 2016 – http://www.ixpmanager.org/ – and issued a call for sponsorship so we could hire a full time developer. The ‘we’ here by the way is my and Nick’s own company – Island Bridge Networks. We’re doing this on a purely cost recovery basis. I’m delighted to say we’ve just about reached our funding goal with three top line sponsors all contributing about €20k each – ISOC, Netflix and SwissIX. The hiring process has now begun!

I’m also delighted to say that there are 33 exchanges around the world using IXP Manager /that we know of/.

Personal Profile for INEX

I was asked to write a personal profile for INEX in <= 300 words. Reproduced here.

If you want to confuse Barry, ask him where he’s from: born in Cork, spent his formative years in Galway and married into Dublin. He got a honours degree in Maths, his first love, from NUI, Galway in 2001 and went on to do four years research in information theory in UCD’s Computer Science department.

In 2005 he took a job with imag!ne to help build their ADSL broadband service from scratch to a position where it supported tens of thousands of subscribers. Barry branched out on his own in 2007 when he formed his own consultancy business, Open Solutions, which continued working with imag!ne as well as building up a portfolio of network, VoIP and web application development customers.

It was 2008 when Nick Hilliard, INEX’s CTO, approached Barry to provide a couple days operational support to INEX. Little did Barry realise what a huge part of his life INEX would become – both here in Ireland supporting INEX’s infrastructure and membership but also as part of the larger European and international IXP community.

Barry is the lead developer of IXP Manager (http://www.ixpmanager.org/) – a full stack management system for IXPs which includes an administration and customer portal; provides end to end provisioning; and both teaches and implements best practice. INEX is very proud to say that this project is now in use at over 33 IXPs and has grown legs of its own with the wider community sponsoring a full-time developer.

INEX has always been happy to help other IXPs and, through our relationship with the Internet Society (ISOC), RIPE and Euro-IX, Barry has travelled to countries with a less developed internet infrastructure to advise on best practice, has delivered a number of IXP Manager workshops and contributes to policy development.

IXP Manager – Planning for v4

A lot has changed in the 3 to 5 years that the decision was made to use certain libraries / technologies / methods on IXP Manager.

In previous major version changes we made some serious architecture changes in one sweep. For example v2 -> v3 saw the complete migration from Doctrine ORM v1 to v2 (which was a change from the Active Record pattern to the Data Mapper pattern).

Today, IXP Manager is a very large project and to do such a sweeping migration in one go would stifle development, break something that isn’t actually broken and take a lot of time.

But, sticking with older technologies and libraries has negative effects also. It creates developer apathy (for which I can personally vouch for). It also provides a major stumbling block for bringing on new developers and contributors (who wants to learn Zend Framework 1 now which has been EOL’d for sometime?).

So, our plan for v4 is to bring in new technologies without throwing away or rewriting everything we have.

IXP Manager is a MVC application that currently uses Doctrine2 as the Model, Smarty as the View and ZF1 as the Controller. Doctrine2 is still current and won’t be changing.

Smarty will remain as the view engine for current / unmigrated functionality. But Smarty is… oh my God… soooooo bad. v4 will default to Twig which is more modern and far better structured from a programming point of view. Coupled with the new framework, it will also allow for a nicer means of skinning. For the interested, Twig has some very nice features including layouts, macros and also some nice security features.

ZF1 has served us well but it’s been EOL’d and is now quite outdated. The new hotness in PHP is Laravel, which I’ve been using to great effect for a while now. Laravel show cases some of the new and best functionality of PHP and using very modern techniques (such as IoC).

But more importantly, Laravel will let us do things in a much different and much more flexible manner for the IXPs using IXP Manager. Some of these include:

  •  Job queues: built-in and simple (to use) support for job queues via Beanstalkd and others. Queuing jobs will provide functionality that we at INEX have been looking for (and it’s also an FAQ from other IXPs) -> reconfiguring services on demand (or, at least quicker than a twice daily cronjob).

Put this together with:

  • Events: Laravel allows us to trigger events and subscribe to them.

A key example of queue and event functionality would be that a change to a VLAN interface (such as checking the route server client box) would trigger a vlan interface changed event. One subscriber to this event would be the route server configuration manager. Based on the VLAN change, this event handler can then queue events. The route servers themselves would monitor these queues and rebuild / reconfigure the route servers appropriately on demand.

Similar handlers for route collectors, DNS ARPA changes, etc. can offer much more real time control of all the services at an IXP.

IoC decouples logic from the controller. What this means is that IXPs who want to do things differently than INEX (let’s say use Cacti instead of MRTG as an example), can swap out MRTG with Cacti with one line of code (that’s assuming we write contracts – interfaces – for such handlers and a Cacti version is coded of course!). But that’s the kind of power and flexibility we’re looking to bring in.

Other features Laravel provides includes:

  • Much improved unit testing on controller actions. Right now, we spin up Apache and MySQL to test controller actions. This is no longer required with Laravel making tests easier to write, more robust and more focused with built in support for mock objects.
  • A much nicer and more structured way of creating command line interfaces rather than the quite clunky way we have of doing it currently.
  • A much more natural way to develop REST API endpoints with json:api compatible responses.

And that leads us to the front end. Right now, the front end and the back end are tightly coupled. During the development lifetime of v4, we want to move more towards an API is Everything back end with a decoupled front end.

This separation will again aid unit testing providing a more reliable and robust IXP Manager. It will allow other IXPs to create their own front end on member facing portals or, even, move to IXP Manager as their back end system but retaining investment of current member portals by adding new features from IXP Manager through API endpoints. It will also allow existing systems in IXPs to integrate with IXP Manager to provision services and ports for example.

One of the bigger tests of this plan will be the (long awaited and badly needed) revamp of the member facing area. We’re currently planning the UI / UX of this to deliver key information to members in the best way possible. This will include Bootstrap v3 which is fluid from the ground up so mobile browsers to wide screen browsers should be supported naturally.

During the early stages of v4, we’ll create the API endpoints necessary to support the member portal functions and then create a front end on that using Ember.js.

Other changes in v4 will include:

  • A switch from package management via Git sub-modules to composer and Packagist as is current standard practice.
  • Introduction of Bower for front end asset management.
  • And we’ll need a task runner for pulling everything together – for that we’ll use Grunt (although that’ll mostly be a development / release prep tool rather than an end user requirement).

So, that’s what we’re looking at! It won’t happen overnight but we’ll continue our policy of release early, release often and we’ll update the documentation and provide complete upgrade instructions at the appropriate times. Some of the above is also subject to change depending on practical experience / issues as we move towards it.

Comments, ideas, etc. are all welcome.