Tag: ixp manager

INEX’s Shiny New Route Servers

Copy of an article I wrote on INEX’s own blog for longevity – original published here on April 10 2019.

In this article, we talk about the new route servers that we deployed across all three peering platforms at INEX during February 2019 and, particularly, RPKI support.

Most route server instances at internet exchanges (IXPs) perform prefix filtering based on route/route6 objects published by internet routing registries (IRRDBs). INEX members would be used to creating these through RIPE’s database. However there are many other registries and the data quality of some of these IRRDB objects is often poor, with problems relating to missing, stale and incorrectly duplicated information.

A typical IRRDB entry would resemble the following:

route:          192.0.2.0/24
descr:          Example IPv4 route object
origin:         AS65500
created:        2004-12-06T11:43:57Z
last-modified:  2016-11-16T22:19:51Z
source:         SOME-IRRDB

RPKI

RPKI is a public key infrastructure framework designed to secure the internet’s routing infrastructure in a way that replaces IRRs with a database where trust is assigned by the resource holder. The equivalent of a route object in RPKI is called a ROA (Route Origin Authorisation). It is a cryptographically secure triplet of information which represents a route, the AS that originates it and the maximum prefix length that may be advertised. An example of an IPv4 and an IPv6 ROA would be:

( Origin AS,  Prefix,         Max Length )
( AS65500,    2001:db8::/32,  /48        )
( AS65501,    192.0.2.0/24,   /24        )

ROAs are typically created through your own RIR (so, RIPE for most INEX members). These RIRs are called trust anchors in RPKI. RIPE have created an extremely easy wizard for creating ROAs through the LIR Portal.

To implement RPKI in a router, the router needs to build and maintain a table of verified ROAs from the five RIRs/trust anchors. The easiest way of doing this is to use a local cache server which pulls and validates the ROAs from the trust anchors and uses a new protocol called RPKI-RTR to feed that information to routers. Currently there are three validators: RIPE’s RPKI Validator v3; Routinator 3000 from NLnetLabs; and Cloudflare’s GoRTR. INEX currently uses the former two.

RPKI validation of a route against the table of ROAs yields one of three possible results:

  • VALID: a ROA exists for the route and both the prefix length is within the allowed range and the origin ASN matches.
  • INVALID: a ROA exists for the route but either (or both) the prefix length is outside the allowed range and/or the origin ASN is different.
  • UNKNOWN: no ROA exists for the route.

UNKNOWN is a common response as the database has only a fraction of the prefix coverage as IRR databases do. We are now in a multi-year transition from IRR to RPKI route validation while ROAs are created.

Bird V2

As well as RPKI support, we have also upgrading all route servers to Bird v2.

This is a significant rewrite to Bird which, for v1, maintained separate code and daemons for IPv4 and IPv6. Bird v2 merges these code bases and also introduces support for new SAFIs such as l3vpns / mpls.

Overall, the configuration changes required were minimal and INEX continues to run separate daemons of Bird v2 for IPv4 and IPv6 daemons. Route servers are CPU intensive and separate daemons allows for maximum stability, keeps the configuration clean and fits into the existing deployment processes we have built up with IXP Manager.

Route Server Filtering Flow

Our work on the new route servers will be released to the community as part of IXP Manager v5 shortly. The new filtering flow is enumerated below. One of the key new features is that if any route fails a step, we use internal large community tagging to indicate this and the specific reason to our members through the IXP Manager looking glass (more on that later).

  1. Filter small prefixes (>/24 for IPv4, >/48 for IPv6).
  2. Filter martian / bogon ranges.
  3. Sanity check to ensure the AS path has at least one ASN and no more than 64.
  4. Sanity check to ensure the peer ASN is the same as first ASN in the prefix’s AS path.
  5. Prevent next-hop hijacking (where a member advertises a route but puts the next hop as another member’s router rather than their own). We do allow same-AS’s to specify their other router(s).
  6. Filter known transit networks.
  7. Ensure that the origin AS is in set of ASNs from member’s AS-SET. See below for some additional detail on this.
  8. RPKI validation. If it is RPKI VALID, accept the route. If it is RPKI INVALID then filter it.
  9. If the route is RPKI UNKNOWN, revert to standard IRRDB filtering.

Regarding step 7 above, an AS-SET is another type of IRRDB database entry where a network which also acts as a transit provider for other networks can enumerate the AS numbers of those downstream networks. This is something RPKI does not yet support but it is being worked on – see AS-Cones.

Lastly we have enhanced the BGP large community support to allow our members request as-path prepends on announcements to specific members over the route servers. For these and more supported communities, see the INEX route server page here.

Bird’s Eye and the Looking Glass

As well as IXP Manager, INEX has also written and open sourced a secure micro service for querying Bird daemons called Bird’s Eye. IXP Manager uses this to provide a web-based looking glass into our route collectors and servers. We have recently released v1.2.1 of Bird’s Eye which adds support for Bird v2.

We have greatly enhanced IXP Manager’s looking glass to support both Bird v2 and the large communities we use to tag filtered reasons. You can explore any of INEX’s route servers to see this yourself – for example this is route server #1 for IPv4 on INEX LAN1. When members log into IXP Manager they will also find a new Filtered Prefixes tool which will summarise any filtered routes across all 12 of INEX’s route server instances.

More Information

We have spoken about this at a number of conferences recently:

Migrating Legacy Web Applications to Laravel

I gave a talk on migrating legacy web applications to Laravel at last year’s Laravel Live UK conference in London. Those that rated it gave it high marks which is always brilliant feedback to receive.

2018 was Laravel Live UK’s inaugural conference and it was a packed house. They’ve just announced the dates for 2019 and I would strongly recommend attending for anyone using or interested in use Laravel.

Following the conference, I wrote up the talk as an article and it has just been published in the March 2019 edition of php[architect]. This is an excellent magazine which I’ve subscribed to for a few years now – the digital edition is very reasonable and comes as a DRM-free PDF onto an app on your phone/pad or downloaded to your computer.

As it happens, they chose this article as the teaser for this issue and so it is freely available online here and downloadable as a PDF here. But seriously, if you are a PHP developer, you need to subscribe to this magazine.

Lastly, if you are interested in the slide deck from the conference, you can download them here – but the article is a much better way to understand this material.

UI Tests with Laravel Dusk for IXP Manager

We use standard PHPUnit tests for IXP Manager for some mission critical aspects. These take data from a test database filled with known sample data (representing a range of different member configurations). The tests then use this information to generate configurations and compare these against known-good configurations.

This way, we know that for a given set of data, we will get a predictable output so long as we haven’t accidentally broken anything in development.

But, as an end user, how do you know that what you stick in a web-based form gets put into the database correctly? And conversely, how do you know that form represents the database data correctly when editing?

This is an issue we ran into recently around some checkbox logic and a dropdown showing the wrong selected element. These issues are every bit as dangerous to mission critical elements as the output tests we do with PHPUnit.

To test the frontend, we turn to Laravel Dusk – an expressive, easy-to-use browser automation and testing API. What this actually means is that we can right code like this:

$this->browse( function( $browser ) use ( $user ) {
    $browser->visit('/login')
        ->type( 'username', $user->email )
        ->type( 'password', 'secret' )
        ->press('Login')
        ->assertPathIs('/home')
        ->assertSee( 'Welcome to IXP Manager ' . $user->name );

We have now added Dusk tests for UI elements that involve adding, editing and deleting all aspects of a member interface and all aspects of adding, editing and delete a router object. Here’s an example of the latter:

Laravel Dusk - Animated Gif Example

Doctrine2 with GROUP_CONCAT and non-related JOIN

Doctrine2 ORM is a fantastic and powerful object relational mapper (ORM) for PHP. We use it for IXP Manager to great effect and we only support MySQL so our hands are not tied to pure Doctrine2 DQL supported functions. We also use the excellent Laravel Doctrine project with the Berberlei extensions.

Sometimes time is against you as a developer and the documentation (and StackOverflow!) lacks the obvious solutions you need and you end up solving what could be a single elegant query very inefficiently in code with iterative database queries. Yuck. 

I spent a bit of time last night trying to unravel one very bad example of this where the solution would require DQL that could:

  1. group / concatenate multiple column results from a one-to-many relationship;
  2. join a table without a relationship;
  3. ensure the joining of the table without the relationship would not exclude results where the joint table had no matches;
  4. provide a default value for (3).

Each of these was solved as follows:

  1. via MySQL’s GROUP_CONCAT() aggregator. The specific example here is that when a MAC address associated with a virtual interface can be visible in multiple switch ports. We want to present the switch ports to the user and GROUP_CONCAT() allows us to aggregate these as a comma separated concatenated string (e.g. "Ethernet1,Ethernet8,Ethernet9").
  2. Normally with Doctrine2, all relationships would be well-defined with foreign keys. This is not always practical and sometimes we need to join tables on the result of some equality test. We can do this using a DQL construct such as: JOIN Entities\OUI o WITH SUBSTRING( m.mac, 1, 6 ) = o.oui.
  3. This is as simple as ensuring you LEFT JOIN.
  4. The COALESCE() function is used for this: COALESCE( o.organisation, 'Unknown' ) AS organisation.

We have not yet pushed the updated code into IXP Manager mainline but the above referenced function / code is not replaced with the DQL query:

SELECT m.mac AS mac, vi.id AS viid, m.id AS id, 
    m.firstseen AS firstseen, m.lastseen AS lastseen,  
    c.id AS customerid, c.abbreviatedName AS customer,
    s.name AS switchname, 
    GROUP_CONCAT( sp.name ) AS switchport, 
    GROUP_CONCAT( DISTINCT ipv4.address ) AS ip4, 
    GROUP_CONCAT( DISTINCT ipv6.address ) AS ip6,
    COALESCE( o.organisation, 'Unknown' ) AS organisation

FROM Entities\\MACAddress m
    JOIN m.VirtualInterface vi
    JOIN vi.VlanInterfaces vli
    LEFT JOIN vli.IPv4Address ipv4
    LEFT JOIN vli.IPv6Address ipv6
    JOIN vi.Customer c
    LEFT JOIN vi.PhysicalInterfaces pi
    LEFT JOIN pi.SwitchPort sp
    LEFT JOIN sp.Switcher s
    LEFT JOIN Entities\\OUI o WITH SUBSTRING( m.mac, 1, 6 ) = o.oui

GROUP BY m.mac, vi.id, m.id, m.firstseen, m.lastseen, 
    c.id, c.abbreviatedName, s.name, o.organisation

ORDER BY c.abbreviatedName ASC

 

A Brief History of IXP Manager

For another INEX project, I was asked to put together a timeline for IXP Manager – an open source application for managing Internet eXchange Points. Reproduced here:

IXP Manager was originally a web portal written in PHP by Nick Hilliard in 2005. It was a basic database frontend that just did fairly simple CRUD (CReate, Update, Delete operations) and allowed our members to log in and view their traffic usage graphs.

Around this was a ton of Perl scripts that sucked that data out of the the database and created configuration files for route collectors, graphing, monitoring, etc.

The major achievement of Nick’s original system was the database design (the schema). The core of that schema is still the core of IXP Manager over 10 years later.

I started in INEX in 2007 and started to expand IXP Manager using what was becoming a more modern web development paradigm – Model/View/Controller with a framework called Zend Framework.

There wasn’t a grand plan here – it was just “as we needed” organic growth over the coming years.

In 2010 we decided what we had was actually pretty good and could be very useful for other IXPs. We got committee approval to open source the software and we released IXP Manager V2 in 2010 under the GPL2 license (GNU Public License v2).

This license essentially means anyone can use the software free of charge but also that they should contribute back improvements that they may make. The idea being that INEX would eventually benefit from other IXPs contributing to the project.

Open sourcing a project doesn’t mean it’ll be successful though! What we didn’t do in 2010 was put infrastructure around it such as: presentations at IXP conferences, mailing lists for user support, decent documentation, etc.

We corrected all that and re-released an updated version called IXP Manager v3 in 2012. This time it took off! We also started collaborating with ISOC (The Internet Society) around this time to help start-up IXPs (mainly eastern Europe and Africa) use IXP Manager.

Some established IXPs also contributed money towards development of missing features – most notably LONAP in the UK – and these new features fed back into INEX.

We’ve worked hard on v3 and it’s developed well since with many new features and improvements. Sometime in late 2016 – maybe even this month – we’ll release v4 which is a major leap forward again and should hopefully attract new users and developers.

INEX is very well regarded in the IXP community as an exchange that is well run and both operates and teaches best practice. All of what we’ve learnt running a good exchange has fed into IXP Manager and it helps those IXPs that use it to implement those same good practices. IXP Manager has helped raise INEX’s reputation even further.

Lately we’ve begun to realise that as a small team we can’t do it all ourselves – the more exchanges that use it, the more requests for help and features we receive and as a result, new developments take a back seat.

To try and improve this we launched a new website in 2016 – http://www.ixpmanager.org/ – and issued a call for sponsorship so we could hire a full time developer. The ‘we’ here by the way is my and Nick’s own company – Island Bridge Networks. We’re doing this on a purely cost recovery basis. I’m delighted to say we’ve just about reached our funding goal with three top line sponsors all contributing about €20k each – ISOC, Netflix and SwissIX. The hiring process has now begun!

I’m also delighted to say that there are 33 exchanges around the world using IXP Manager /that we know of/.