Month: June 2012

Migrating SVN with Branches and Tags to Git

Following my love affair with Git, I’ve also started using a local install of Gitorious for private and commercial projects at Open Solutions. Before Gitorious, this meant setting up authentication and Apache aliases for each new Git repository which meant we were pretty disinclined to create repositories when we should have.

With Gitorious, it’s just a couple of clicks and we have internal public repositories, team repositories or individual developer private repositories. It’s grrrrrrreat!

Last night and this morning, I’ve stated a process of finding the many SVN repositories I / we have scattered around to import them into Git (with all branches and tags). Here’s the process:

Importing Subversion Repositories with Branches and Tags to Git

1. Create a users file so you can correctly map SVN commit usernames to Git users. For example, users.txt:

user1 = First Last Name 
user2 = First Last Name  
...

2. Now clone the Subversion repository:

git svn clone -s -A users.txt svn://hostname/path dest_dir-tmp
cd dest_dir-tmp
git svn fetch --fetch-all

3. Now we have all the SVN repository. We need to create Git tags to match the SVN tags:

git branch -r | sed -rne 's, *tags/([^@]+)$,\1,p' | \
while read tag; do \
 echo "git tag $tag 'tags/${tag}^'; git branch -r -d tags/$tag"; \
done | sh

4. Now we need to create matching branches:

git branch -r | grep -v tags | sed -rne 's, *([^@]+)$,\1,p' | \
 while read branch; \
 do echo "git branch $branch $branch"; \
done | sh

5. To help speed up a remote push, we’ll compact the repository:

git repack -d -f -a --depth=50 --window=100

6. Then we remove the meta-data that was used by git-svn:

git config --remove-section svn-remote.svn
git config --remove-section svn
rm -r .git/svn

7. And finally, we push it to our own Gitorious server:

git remote add origin git@example.com:zobie/my_great_app.git
git push --all && git push --tags

References:

  1. http://stackoverflow.com/questions/3239759/checkout-remote-branch-using-git-svn
  2. http://stackoverflow.com/questions/79165/how-to-migrate-svn-with-history-to-a-new-git-repository
  3. http://blog.zobie.com/2008/12/migrating-from-svn-to-git/

Centralised Logging

I’m currently looking at some centralised logging tools and the following stand out:

  • Octopussy – one I cam across a long time ago but looking at some of the others below it may be past its sell by date?
  • Graylog2 – GSOH (in dating parlance) – “Manage your logs in the dark and have lasers going and make it look like you’re from space.“
  • logstash – “a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). Speaking of searching, logstash comes with a web interface for searching and drilling into all of your logs.“
  • Kibana - You have logs. Billions of lines of data. You shipped, dated it, parsed it and stored it. Now what do you do with it? Now you make sense of it… Kibana is an alternative browser based interface for Logstash… that allows you to efficiently search, graph, analyze and otherwise make sense of a mountain of logs.

Kibana has a Bootstrap UI and is written in PHP which immediately bumps it up my list 😉

Monitoring Asterisk via SNMP (with OSS_SNMP)

Over on the company blog, we just announced that we committed a complete Asterisk MIB implementation to OSS_SNMP in addition to a proof of concept Asterisk wallboard to NOCtools which includes active channels, uptime, version information and more.

Full details are on the company post but here’s a snippet of how easy it is to query Asterisk over SNMP:

 $host = new \OSS_SNMP\SNMP( $asteriskIP, $community );
 echo "Asterisk version: " . $host->useAsterisk()->version();
 echo "Active calls: " . $host->useAsterisk()->callsActive();
 echo "Calls processed: " . $host->useAsterisk()->callsProcessed();

And here’s a screenshot of the proof of concept Asterisk wallboard:

[NOCtools Asterisk Wallboard]
NOCtools – Asterisk Wallboard

NOCtools – A Mixed Bag of Tools and Utilities for NOC Engineers

NOCtools is a mixed bag of tools and utilities that are useful for NOC engineers. This project originally started out as a way to highlight and utilise our OSS_SNMP library (a PHP SNMP library for people who hate SNMP).

It since grew into a way to graphically present information on network topology that is normally difficult and cumbersome to do by logging into individual devices. Such information includes a discovered L2 topology by CDP, using this to present rapid-PVST port roles and so forth.

From the company blog:

Today, we are introducing NOCtools which uses this library to provide a number of useful tools including:

  • CDP Neighbours: for a given CDP enabled device, display its CDP neighbours with information and also a graph showing connected ports.
  • CDP L2 Topology: graph the layer 2 network topology based on a recursive crawl of CDP neighbours.
  • RSTP Topology & Port Roles: similar to CDP L2 Topology above but this takes a specific VLAN and identifies and graphs Per-VLAN Spanning Tree port roles.
  • Per-VLAN RSTP Port Roles: a tool that will display the per-VLAN Rapid STP port roles for a given VLAN (or for all VLANs) on a device.
  • Inter-Device VLAN Comparison: a tool that will compare VLANs available (and their respective names) across selected devices allowing you to ensure consistency as well as perform simple security audits.

Follow the links about for screen shots and more details. We are releasing this under a GNU GPL license in the hope that the wider networking community will benefit from them.

A PHP SNMP Library for People Who HATE SNMP, MIBs and OIDs!

I hate SNMP! But I have to use it on a daily basis with my company, Open Solutions.

Don’t get me wrong, it’s an essential tool in the trade of network engineering but it’s also a serious PITA. Finding MIBs, OIBs, making them work, translating them, cross-vendor translations, etc, blah, blah. And then, when you do find what you need, you’ll have forgotten it months later when you need it again.

Anyway, while trying to create some automatic L2 topology graphing tools (via Cisco/Foundry Discovery Protocol for example) and also some per VLAN RSTP tools to show port states, I started writing this library. As I wrote, I realised it was actually very useful and present it here now in the hopes that the wider network engineering community will find it useful and also contribute back MIBs.

An example may best illustrate the library:

First, we need to instantiate an SNMP object with a hostname / IP address and a community string:

$ciscosw = new \OSS\SNMP( $ip, $community );

Assuming the above is a standard Cisco switch, let’s say we want to get an associate array of VLAN names indexed by the VLAN ids:

print_r( $ciscosw->useCisco_VTP()->vlanNames() );

This yields something like the following:

Array (
    [1] => default
    [2] => mgmt
    [100] => cust-widgets
    [1002] => fddi-default
    ...
)

It really is that easy.

See the GitHub project page: https://github.com/opensolutions/OSS_SNMP

What the Hell is INEX? An IXP?

In a few recent posts, I’ve mentioned INEX.

INEX is a neutral, industry-owned association, founded in 1996, that provides IP peering facilities for its members. INEX membership is open to all organisations that can benefit from peering their IP traffic and there are currently 57 members.

INEX can also be considered Ireland’s IP Peering Hub. INEX membership provides high-speed, reliable and resilient IP traffic exchange facilities for both Irish and International organisations, allowing them to route IP traffic efficiently thereby providing faster, more reliable and lower-latency internet access for their customers.

So what the hell is an IXP? Well, Euro-IX commissioned the following, the Internet Revealed: A file about IXPs, a couple of years ago which brilliantly explains IXPs.

Follow Up – IPv6 Statistics at INEX

A couple of days ago, I was talking about World IPv6 day with some notes on the Irish context.

INEX is a neutral, industry-owned association, founded in 1996, that provides IP peering facilities for its members. INEX membership is open to all organisations that can benefit from peering their IP traffic and there are currently 57 members.

INEX can also be considered Ireland’s IP Peering Hub. INEX membership provides high-speed, reliable and resilient IP traffic exchange facilities for both Irish and International organisations, allowing them to route IP traffic efficiently thereby providing faster, more reliable and lower-latency internet access for their customers.

As a follow up to the previous post, here’s a like for like comparison of IPv4 and IPv6 traffic over peering LAN 1 of the exchange:

Notes:

  • As a layer 2 exchange, traffic over INEX is symmetrical – traffic originating from one member is destined for another.
  • INEX runs two peering LANs for resiliency. The IPv6 traffic on LAN 2 was negligible over the same period. See the public statistics and the weathermaps of each LAN showing the network topology.

 

World IPv6 Day with Irish Statistics

In case it passed you by, today was World IPv6 Day. In a nutshell: “Major Internet service providers (ISPs), home networking equipment manufacturers, and web companies around the world are coming together to permanently enable IPv6 for their products and services by 6 June 2012.” This includes top content providers such as Facebook (see under their hood), Google (read what they had to say), Yahoo! and Microsoft. In fact, you may not even have noticed but Google were advertising it front and centre on their search page:

Google Announcing World IPv6 Day on Their Search Page

Over at INEX, we were unable to pull out IPv6 traffic statistics on the exchange until recently and my colleague just got the first pass of that project complete this week in time for World IPv6 Day. Here’s how it looked over the hours leading up to and into World IPv6 Day:

Now, the peek of almost 40Mbps is, most assuredly, small compared to the overall peek of 24Gbps, but there is a very pronounced jump in IPv6 traffic which is certainly a good sign and a move in the right direction. The overall peering statistics at INEX are public and we’ll be breaking out IPv4 and IPv6 into separate graphs shortly also.

Why does IPv6 amount to < 0.2% of the traffic at the exchange? Well there are two main factors:

  • Until today, there has been very little mass or popular content available over IPv6. So, even if you were IPv6 enabled, there was very little for you access.
  • None of the large ISPs in Ireland are providing IPv6 connectivity to end users outside of certain closed test programs.

This is the classic chicken and egg problem: with no content available the ISPs were not motivated to provide IPv6 connectivity; and, conversely, with no IPv6 enabled eyeballs the content providers were not motivated to make their services available over IPv6.

While today was not necessarily a content provider only day, I’m unaware of any Irish ISPs that got involved. But, now that we have significant content available over IPv6, hopefully the ISPs will begin to ramp up their own programs. And – to be fair – it’s not all bad news with the ISPs in Ireland. Most have their core and edge networks IPv6 enabled, it’s the access layer that’s the issue (and it’s a really really big issue and a very difficult issue).

AMS-IX (the Amsterdam Internet Exchange) is in the top three IXPs in the world by traffic volume and they also make their IPv6 statistics public. As a second demonstration of traffic levels on World IPv6 Day, here is the week to date showing a huge differential for today:

If you’re not sure what all this is about, well then here are a few words from the creator of the Internet himself:

And if you’re keen to start experimenting with IPv6, first email and ask your ISP. They’ll say no, but do it anyway! Then head over to SixXS (and be sure to choose either HEAnet or Digiweb as your PoP as both are INEX members and as such you’ll have the lowest possible latency).

Top 15 UI Libraries on GitHub (with 1500+ watchers)

I was just reviewing an accessibility presentation where the author had an interesting slide on the top 15 UI libraries on GitHub with 1500+ watchers. Here they are with links and descriptions (my own comments in italics):

  1. Bootstrap, from Twitter - Simple and flexible HTML, CSS, and Javascript for popular user interface components and interactions. A favourite of mine and I’ve used it on Open Solutions, ViMbAdmin, TallyStick, ePayroll and more.
  2. impress.js – It’s a presentation framework based on the power of CSS3 transforms and transitions in modern browsers and inspired by the idea behind prezi.com. (GitHub page).
  3. chosen - Chosen is a library for making long, unwieldy select boxes more friendly. Another plugin we love and use in a number of projects.
  4. jQuery-File-Upload - File Upload widget with multiple file selection, drag&drop support, progress bars and preview images for jQuery.
  5. jquery-ui – jQuery UI provides abstractions for low-level interaction and animation, advanced effects and high-level, themeable widgets, built on top of the jQuery JavaScript Library, that you can use to build highly interactive web applications. (GitHub page). Another favourite of ours and we use it along side Bootstrap with the jquery-ui Bootstrap theme.
  6. spin.js - An animated CSS3 loading spinner with VML fallback for IE.
  7. deck.js – Modern HTML presentations (GitHub page).
  8. Skeleton - A Beautiful Boilerplate for Responsive, Mobile-Friendly Development.
  9. Foundation - An easy to use, powerful, and flexible framework for building prototypes and production code on any kind of device.
  10. showoff - the best damn presentation software a developer could ever love (example).
  11. ajax-upload - A file upload script with progress-bar, drag-and-drop (GitHub page).
  12. isotope - An exquisite jQuery plugin for magical layouts.
  13. Timeline JS - Beautifully crafted timelines that are easy, and intuitive to use. This actually looks really cool and very pretty.
  14. etherpad-lite - An Etherpad based on node.js – Our goal is to make collaborative editing the standard on the web.
  15. ColorBox - A lightweight customizable lightbox plugin for jQuery.

 

As an honourable mention, if you use Bootstrap and it’s modal dialog, take a look at Bootbox which provides wrappers for JavaScript alert(), confirm() and other flexible dialogs.

Some thinks that jump out at me from the above is that frameworks are very popular and, similarly, prestation frameworks are also very popular. There must be a deep hatred of PowerPoint and Keynote among web developers! The other take away for me is how a very small project – such as chosen – can become hugely hugely popular.