Category: OSS

Git Web Applications (aka GitHub Alternatives)

I’ve been through the mill with a few of these and it’s a growing list of options. I should start by saying that I love GitHub and use it regularly for a large number of projects (including some of my own, some for my company and some for my customers). The only problem I have with GitHub is that I just haven’t made the jump to trust it with our proprietary code – the primary asset of the company. Particularly after a high profile security breech.

Now, nothing I’ve come across yet comes close to GitHub. But two we use daily have good matching features:

  • Gitorious –  whether they intended it or not, it’s a good clone of GitHub but always a little behind on features. It’s also a Ruby on Rails application and the documentation is getting far better. When we started, it was pretty… shite, to be honest. Installing and upgrading was a pain. They have addressed this with http://getgitorious.com/ which includes a virtual appliance for VirtualBox so you can be up and running in minutes.
  • Atlassian Stash – if you’re a small team (<= 10 users) then this could be for you. Installation is pretty easy, it looks great, is very fast (over HTTP/S, but it’s inbuilt SSH client is extremely slow) and is very easy to use. Pull requests are also done really really well. It’s a full Java webapp so it does require some CPU and memory. For <= 10 users, my definite favorite. Thereafter though it gets expensive – e.g. for up to 25 users you’ll fork out $1,800 / annum. That’s not entirely unaffordable but if you’re using Stash, then you may also be using their other products such as Jira, Confluence, Bamboo, Fisheye and Crucible. Now that adds up to a hefty bill!

One’s I have not tested and so cannot speak authoritively on include:

  • GitLab – Looks like a very interesting alternative.
  • RhodeCode – they say it will “change the way you manage your code”.
  • Gitolite – a quick look at the available information for this definitely puts it in the also ran category. There’s no polish or anything nice to entice one to even try it.
  • Gitosis – in their own words: “Manage git repositories, provide access to them over SSH, with tight access control and not needing shell accounts”. The project appears dead.

 

Monitoring SSL Certificate Expiry Dates with Nagios

It is good practice to separate Nagios checks of your web server being available from checking SSL certificate expiry. The latter need only be run once per day and should not add unnecessary noise to a more immediately important web service failure.

To use check_http to monitor SSL certificate expiry dates, first ensure you have a daily service definition – let’s call this service-daily. Now create two service commands as follows:

define command{
    command_name check_cert
    command_line /usr/lib/nagios/plugins/check_http -S \
        -I $HOSTADDRESS$ -w 5 -c 10 -p $ARG1$ -C $ARG2$
}

define command{
    command_name check_named_cert
    command_line /usr/lib/nagios/plugins/check_http -S \
        -I $ARG3$ -w 5 -c 10 -p $ARG1$ -C $ARG2$
}

The second is useful for checking named certificates on additional IP addresses on web servers serving multiple SSL domains.

We can use these to check SSL certificates for POP3, IMAP, SMTP and HTTP:

define service{
    use service-daily
    host_name mailserver
    service_description POP3 SSL Certificate
    check_command check_cert!993!21
}

define service{
    use service-daily
    host_name mailserver
    service_description IMAP SSL Certificate
    check_command check_cert!995!21
}

define service{
    use service-daily
    host_name mailserver
    service_description SMPT SSL Certificate
    check_command check_cert!465!21
}

define service{
    use service-daily
    host_name webserver
    service_description SSL Cert: www.example.com
    check_command check_named_cert!443!21!www.example.com
}

define service{
    use service-daily
    host_name webserver
    service_description SSL Cert: www.example.net
    check_command check_named_cert!443!21!www.example.net
}

Nagios Plugin for Checking Backups via rsnapshot

We’ve just added a check_rsnapshot.php script to our nagios-plugins bundle on Github. This script will verify rsnapshot backups via Nagios using a number of checks / tests:

  • minfiles – checks the number of files in a snapshot against a minimum expected number;
  • minsize – checks the size of a snapshot against a minimum expected size;
  • log – parses the rsnapshot log to ensure the most recent runs for each retention period completed successfully;
  • timestamp – checks for files created server side containing a timestamp and thus ensuring snapshots are succeeding;
  • rotation – checks that retention directories are being rotated; and
  • dir-creation – checks that retention directories are being created.

Please see this Github wiki page for more information including instructions.

Analysing MySQL Slow Query Logs

MySQL has a really useful feature that allows it to log slow queries where slow is a minimum time defined by you in micro seconds. It helps a lot is diagnosing website outages or slow responsiveness issues after the fact.

Unfortunately I couldn’t find any nice graphical tools for analysing these but there are a few command line tools:

mysqldumpslow

MySQL’s own tool, mysqldumpslow, which aggregates queries and allows you to sort them by: query time or average query time; lock time or average lock time; rows sent or average rows sent; or the number of queries.

Percona’s MySQL Slow Query Log Analyser

Dating from 2006, Percona’s Peter Zaitsev wrote about their own version of a slow query log analyser (local copy) which has given me good results. Note that their micro time patch has since been incorporated into MySQL mainstream.

One of the main differences over MySQL’s own version is that as well as printing the aggregated query (with number and string literals wildcarded), it also prints a real example of the query allowing a copy and paste to MySQL for execution with EXPLAIN.

Example output with query details redacted:

### 230 Queries 
### Total time: 4708.948293, Average time: 20.4736882304348
### Taking 0.093420 to 203.693466 seconds to complete
### Rows analyzed 0 - 141008
SET timestamp=XXX;
SELECT ... FROM ... AS A 
        INNER JOIN ... AS C ON C.item_id = A.item_id 
    WHERE XXX AND C.item_lang = 'XXX' AND ... 
    ORDER BY CATALOG.item_sort LIMIT XXX;

SET timestamp=1348032761;
SELECT ... FROM ... AS A 
        INNER JOIN ... AS C ON C.item_id = A.item_id 
    WHERE 1 AND C.item_lang = '1' AND ... 
    ORDER BY C.item_sort LIMIT 1;

 

Thunderbird 15 Released with Chat – But Shite Error Handling

I’ve just moved a couple of my chat and IRC accounts over to Thunderbird 15 to see how they look. After all, one communications application is better than three. Mozilla have a good instruction set here and they work well for standard chat accounts.

However, most corporations using in house XMPP servers with self signed certificates and these fail with no error messages whatsoever. It was by fluke I found this post: How to Make Thunderbird Chat Work with Most XMPP/Jabber Accounts. Thanks for doing the legwork Rod, this helped a lot.

 

Two Git Branching Models

In current projects, we tend to float between two branching models depending on the requirements of the customer / project and the planned deployment process.

git-flow

This was previously known as A successful Git branching model with an extremely detailed overview and instructions here. It has since spawned a project to help make using this model easier – git-flow.

This is a great branching model for a team of people who work towards creating planned versioned deployments (whether it be time based such as every second Tuesday, or milestone based).

In essence, this model uses two main branches:

  • origin/master – the main branch where the source code of HEAD always reflects aproduction-ready state.
  • origin/develop - where the source code of HEAD always reflects a state with the latest delivered development changes for the next release. Some would call this the “integration branch”. This is where any automatic nightly builds are built from.

Then developers create supporting branches as part of their development process:

  • feature branches – branched off origin/develop for active development of a new feature / enhancement;
  • hotfix branches – a branch off origin/master to apply a critical fix to production code;
  • release branches – short lived branches off origin/develop which are used for final testing and patched before being merged to origin/master.

The model works very well in practice and the above linked document is a excellent read on Git branching practices in general as well as git-flow in particular.

Github Flow

This is the model used at Github and discussed by Github developer Scott Chacon here.

This is suited for projects that deploy continuously rather than around the concept of releases and so it’s less constrained than git-flow. These are two very different models and Scott puts forth his arguments in favour of the Github model in his post linked above.

In essence, Github flow works as follows:

  • origin/master is always deployable. Always.
  • Similarly to git-flow, new features are done in their own branch – but off of origin/master in this case.
  • Now, when you believe your new feature is complete, a merge request is opened so someone else can review and check your work. This is the QA process.
  • Once someone else signs off, you merge back into origin/master.
  • This is now deployable and can and will be deployed at anytime.

 

Monitoring Asterisk via SNMP (with OSS_SNMP)

Over on the company blog, we just announced that we committed a complete Asterisk MIB implementation to OSS_SNMP in addition to a proof of concept Asterisk wallboard to NOCtools which includes active channels, uptime, version information and more.

Full details are on the company post but here’s a snippet of how easy it is to query Asterisk over SNMP:

 $host = new \OSS_SNMP\SNMP( $asteriskIP, $community );
 echo "Asterisk version: " . $host->useAsterisk()->version();
 echo "Active calls: " . $host->useAsterisk()->callsActive();
 echo "Calls processed: " . $host->useAsterisk()->callsProcessed();

And here’s a screenshot of the proof of concept Asterisk wallboard:

[NOCtools Asterisk Wallboard]
NOCtools – Asterisk Wallboard

NOCtools – A Mixed Bag of Tools and Utilities for NOC Engineers

NOCtools is a mixed bag of tools and utilities that are useful for NOC engineers. This project originally started out as a way to highlight and utilise our OSS_SNMP library (a PHP SNMP library for people who hate SNMP).

It since grew into a way to graphically present information on network topology that is normally difficult and cumbersome to do by logging into individual devices. Such information includes a discovered L2 topology by CDP, using this to present rapid-PVST port roles and so forth.

From the company blog:

Today, we are introducing NOCtools which uses this library to provide a number of useful tools including:

  • CDP Neighbours: for a given CDP enabled device, display its CDP neighbours with information and also a graph showing connected ports.
  • CDP L2 Topology: graph the layer 2 network topology based on a recursive crawl of CDP neighbours.
  • RSTP Topology & Port Roles: similar to CDP L2 Topology above but this takes a specific VLAN and identifies and graphs Per-VLAN Spanning Tree port roles.
  • Per-VLAN RSTP Port Roles: a tool that will display the per-VLAN Rapid STP port roles for a given VLAN (or for all VLANs) on a device.
  • Inter-Device VLAN Comparison: a tool that will compare VLANs available (and their respective names) across selected devices allowing you to ensure consistency as well as perform simple security audits.

Follow the links about for screen shots and more details. We are releasing this under a GNU GPL license in the hope that the wider networking community will benefit from them.

A PHP SNMP Library for People Who HATE SNMP, MIBs and OIDs!

I hate SNMP! But I have to use it on a daily basis with my company, Open Solutions.

Don’t get me wrong, it’s an essential tool in the trade of network engineering but it’s also a serious PITA. Finding MIBs, OIBs, making them work, translating them, cross-vendor translations, etc, blah, blah. And then, when you do find what you need, you’ll have forgotten it months later when you need it again.

Anyway, while trying to create some automatic L2 topology graphing tools (via Cisco/Foundry Discovery Protocol for example) and also some per VLAN RSTP tools to show port states, I started writing this library. As I wrote, I realised it was actually very useful and present it here now in the hopes that the wider network engineering community will find it useful and also contribute back MIBs.

An example may best illustrate the library:

First, we need to instantiate an SNMP object with a hostname / IP address and a community string:

$ciscosw = new \OSS\SNMP( $ip, $community );

Assuming the above is a standard Cisco switch, let’s say we want to get an associate array of VLAN names indexed by the VLAN ids:

print_r( $ciscosw->useCisco_VTP()->vlanNames() );

This yields something like the following:

Array (
    [1] => default
    [2] => mgmt
    [100] => cust-widgets
    [1002] => fddi-default
    ...
)

It really is that easy.

See the GitHub project page: https://github.com/opensolutions/OSS_SNMP

Top 15 UI Libraries on GitHub (with 1500+ watchers)

I was just reviewing an accessibility presentation where the author had an interesting slide on the top 15 UI libraries on GitHub with 1500+ watchers. Here they are with links and descriptions (my own comments in italics):

  1. Bootstrap, from Twitter - Simple and flexible HTML, CSS, and Javascript for popular user interface components and interactions. A favourite of mine and I’ve used it on Open Solutions, ViMbAdmin, TallyStick, ePayroll and more.
  2. impress.js – It’s a presentation framework based on the power of CSS3 transforms and transitions in modern browsers and inspired by the idea behind prezi.com. (GitHub page).
  3. chosen - Chosen is a library for making long, unwieldy select boxes more friendly. Another plugin we love and use in a number of projects.
  4. jQuery-File-Upload - File Upload widget with multiple file selection, drag&drop support, progress bars and preview images for jQuery.
  5. jquery-ui – jQuery UI provides abstractions for low-level interaction and animation, advanced effects and high-level, themeable widgets, built on top of the jQuery JavaScript Library, that you can use to build highly interactive web applications. (GitHub page). Another favourite of ours and we use it along side Bootstrap with the jquery-ui Bootstrap theme.
  6. spin.js - An animated CSS3 loading spinner with VML fallback for IE.
  7. deck.js – Modern HTML presentations (GitHub page).
  8. Skeleton - A Beautiful Boilerplate for Responsive, Mobile-Friendly Development.
  9. Foundation - An easy to use, powerful, and flexible framework for building prototypes and production code on any kind of device.
  10. showoff - the best damn presentation software a developer could ever love (example).
  11. ajax-upload - A file upload script with progress-bar, drag-and-drop (GitHub page).
  12. isotope - An exquisite jQuery plugin for magical layouts.
  13. Timeline JS - Beautifully crafted timelines that are easy, and intuitive to use. This actually looks really cool and very pretty.
  14. etherpad-lite - An Etherpad based on node.js – Our goal is to make collaborative editing the standard on the web.
  15. ColorBox - A lightweight customizable lightbox plugin for jQuery.

 

As an honourable mention, if you use Bootstrap and it’s modal dialog, take a look at Bootbox which provides wrappers for JavaScript alert(), confirm() and other flexible dialogs.

Some thinks that jump out at me from the above is that frameworks are very popular and, similarly, prestation frameworks are also very popular. There must be a deep hatred of PowerPoint and Keynote among web developers! The other take away for me is how a very small project – such as chosen – can become hugely hugely popular.