Introducing ViMbAdmin – Virtual Mailbox Administration

Open Solutions are pleased to announce the immediate availability of our latest free and open source web application, ViMbAdmin, a web based interface which will allow you to manage mailboxes, virtual domains and aliases.

Open Solutions are pleased to announce the immediate availability of our latest free and open source web application, ViMbAdmin (vim-be-admin). ViMbAdmin is a web based interface which will allow you to manage mailboxes, virtual domains and aliases.

ViMbAdmin is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License version 3, or (at your option) any later version.

ViMbAdmin was entirely funded by Open Solutions and developed by our staff. If you find this application of value, please consider making a donation to our chosen charity.

Do you want to see it in action? We have a live demo which you can access here. You can also browse screenshots by clicking the image on this page.

ViMbAdmin was written in PHP using our own web application framework which includes the Zend Framework, the Doctrine ORM and the Smarty templating system with JQuery on the frontend.

ViMbAdmin is hosted on its own Google Code project page where you can find documentation, browse the source code and access our Subversion repository. We have set up a Google Groups discussion group and you can read our ViMbAdmin blog posts.

ViMbAdmin can work as a slot in replacement for Postfix Admin with a few MySQL ALTER statements.

Features

  • Super admin(s) user level with full access;
  • Admin(s) user level with access only to assigned domains and their mailboxes and aliases;
  • Super admins can create and modify super admins and admins;
  • JQuery Datatable throughout for quick in browser searching and pagination;
  • Create, modify and purge domains including limited the number of mailboxes and aliases a non-super admin can create per-domain;
  • Activate / deactivate admins, domains, mailboxes and aliases at the click of a button;
  • Full logging;
  • Facility for users (mailbox owners) to change their password;
  • Forgotten Password / Password Reset function for admins;
  • Very configurable including:
    • set default values for quotas, number of mailboxes and aliases for domain creation;
    • templated welcome and settings email for users;
    • either plain or MD5 mailbox password support;

 

Useful RANCID Debugging Tips

I always find it difficult to find a good reference for RANCID debugging strategies and, after spending the afternoon on doing same on one installation, put together my own list.

I always find it difficult to find a good reference for RANCID debugging strategies and, after spending the afternoon on doing same on one installation, put together my own list.

Note that in the following, I use clogin and rancid which assumes a Cisco device. Change to the appropriate variations if you’re not trying to work with a Cisco.

  1. Test logging into a device:
    > clogin rtr1.example.com
  2. Test logging into a device and a single command:
    > clogin -t 90 -c"show version" rtr1.example.com
  3. Test logging into a device and run a sequence of commands:
    > clogin -t 90 -c"show version;show calendar" rtr1.example.com
  4. Show what RANCID does with debugging output:
    > rancid -d rtr1.example.com

    If the above throws some errors (especially a list of missed commands, and if you’re using TACACS, ensure you have authorisation to run all the commands RANCID tries but logging into the router as the RANCID user and executing them one at a time.

  5. Same as (4) but record all router / switch output for analysis:
    > setenv NOPIPE YES
    > rancid -d rtr1.example.com

    and then complete output can be found in the file: rtr1.example.com.raw (in this example).

  6. Run RANCID on a single switch / router tree rather than all:
    > /usr/local/bin/rancid-run [tree]
  7. Run RANCID normally:
> /usr/local/bin/rancid-run
  1. Don’t forget that logs are available in RANCID’s logs/ directory.

IANA IPv4 Free Pool Exhausted

The IANA IPv4 free pool was exhausted today, 3 February 2011. Each of the Regional Internet Registries (RIRs) has now received one of the final five /8s.

The RIPE NCC updated yesterday that the IANA IPv4 free pool has been exhausted:

The IANA IPv4 free pool was exhausted today, 3 February 2011. Each of the Regional Internet Registries (RIRs) has now received one of the final five /8s. The RIPE NCC has been allocated 185/8.

The RIPE NCC is holding reserves totaling approximately four /8s (around 75 million individual IPv4 addresses), not including 185/8.

RIPE will most likely exhaust their reserves sometime in 2011:

As we unable to anticipate consumption rates, we cannot fully predict how long our reserves will last. However, we would like to reassure you that our supplies will not be exhausted within the coming months.

INEX Breaks 10Gbps Barrier – Again

INEX, Ireland’s Neutral Internet Exchange Point, broke the 10Gbps barrier last week coinciding with the Government’s budget announcement. It didn’t quite break the previous record from the announcement of the four year plan.

This traffic spike would have been primarily driven by HEAnet and RTE streaming the Dáil proceedings live to Irish Internet viewers – INEX’s members would account for about 97% of all eyeballs in Ireland.

INEX makes its overall traffic statistics publically accessible.

 

INEX Breaks 10Gbps Barrier

INEX, Ireland’s neutral Internet Exchange Point, broke the 10Gbps barrier last week coinciding with the Government’s launch of their four year plan:

Reaching a new record high, traffic at INEX, Ireland’s Internet Exchange Point, has exceeded the 10Gigabit per second barrier for the first time.  This record, of 12.137* Gigabits per second, comes in a year when INEX is experiencing its greatest growth in traffic in the exchange’s 14 year history.  The traffic peak, which coincided with the announcement of the National Recovery Plan, is an indication of the increase in consumption of rich media content (TV, video, music, conferencing) over the Internet by users in Ireland, including the new wave of mobile internet users. INEX has also welcomed a number of new Members in recent months who have also contributed to the rapid traffic growth.

Read the full release on INEX‘s own website. We’re looking forward to seeing this record being smashed tomorrow with the announcement of the 2011 budget.

Open Solutions (my company) has been a part of INEX’s operations team since April 2008, working with the expanding number of INEX Members and ensuring the smooth running of the exchange. We assist with the administration of the switching frabic, provide member support, and develop INEX’s provisioning and management systems.

Combatting SPAM in ISP Mail Servers

Fighting SPAM is not easy. Here’s an example of how we configured an ISP mail server cluster to reduce the SPAM load…

Introduction

ISPs must provide a mail relay service to their end users. However, mail relay isn’t a revenue generating service and ISPs can’t allocate unlimited staff resources to these services. The problem administrators face is trying to find a balance between unobstructively allowing their clients to send legitimate mail but to stop them from spamming.

The main source of SPAM from clients (SMTP clients in the ISPs IP space) is typically computers infected with bots / malware behind the users router. It may be their own or even a neighbor taking advantage of an unsecured wireless network. There is a smaller secondary source als0 – spammers who have someone gotten hold of a customers SMTP authentication details and are logging in from outside of the ISPs IP space to relay mail through.

Now, the problem this causes is that when the ISP mail servers pass on this SPAM to its destination such as a Yahoo! or GMail recipient, these companies examine the ratio of SPAM to HAM coming from the ISPs servers. When this ratio goes the wrong way, they legitimately block the mail servers meaning that customers of the ISP cannot get their mail delivered to these destinations. This is a big problem.

In this article, I’ll describe a new mail cluster set-up for an ISP to try and ensure their SPAM:HAM ratio stays on the right side.

Solution  Overview

The mail cluster comprises a number of SMTP servers all configured identically (in fact all booting from the same image using PXE and an NFS root file system with dedicated local /var partitions). They are running Debian Stable (Lenny at the time of writing) and the mail server is the excellent Postfix.

Postfix is configured to accept authentication using SASL with Dovecot (configured only for authentication, no POP3 or IMAP services) over TLS on port 25 or SSL on port 465. Users on the ISP’s address space do not need to authenticate to relay mail. This is actually a legacy configuration and quite common among ISPs. If we were to go back many years starting all over again, I’d certainly try and evaluate the merits of only allowing relay after authentication irrespective of your ISP.

Mail server server selection is made via DNS round robin but this could easily be changed to a load balancer. As part of this retrofit, we took the opportunity to IPv6 enable all the servers.

We now need to look at securing these boxes to ensure the level of SPAM is controller. There are three points at which we examine incoming mail:

  1. before it hits the SMTP daemon at all;
  2. after the SMTP headers (MAIL FROM, RCPT TO) have been passed to the daemon but before the mail has been accepted (i.e. before the SMTP DATA command);
  3. after the daemon has accepted the mail.

We’ll examine those in each of the sections below.

Rate Limiting Connections

In order to limit repeated connections to the mail server – such as from a SPAM bot that creates a new SMTP session for every mail (rather than sending many mails in one session), we use iptables recent match module.

This module keeps count of the number of connections in a given time period allow us to take alternative action if the number is above a threshold. In our case, we just drop additional connections until the number of connections drops below that threshold. For example:

-A INPUT -p tcp -m state --state NEW -m multiport
        --dports 25,465 -j SMTP_IN

-A SMTP_IN -p tcp --syn -m recent --name SMTP_RATE_LIMIT --set
-A SMTP_IN -p tcp --syn -m recent --name SMTP_RATE_LIMIT --update
        --seconds 60 ! --hitcount 10 -j SMTP_IN_LIMIT
-A SMTP_IN -j ACCEPT

-A SMTP_IN_LIMIT -j LOG --log-prefix "SMTP_IN: LIMIT EXCEEDED:: "
-A SMTP_IN_LIMIT -j DROP

We have some additional rules where, for example, we have whitelisted core infrastructure IP addresses and also mail from non ISP IP addresses (the purpose here is to stop SPAM bots from within the network).

Checks Before Accepting the Mail

Once the client connects to Postfix we check a number of other items.

We of course use some of Postfix’s own policy checks (such as reject_non_fqdn_sender, reject_non_fqdn_recipient, etc) but that’s pretty par for the course and not discussed below.

The ISP uses a commercial subscription to the Spamhaus data feed service. This is a very reliable database of IP addresses that are known to be sending SPAM (and other classifications). This is the first thing we check – customer or not. If you’re on this list, we do not accept mail from you.

If you’re not coming from the ISP address space, we then initiate gray listing using a Debian package called postfix-gld (see this site) which uses a MySQL database as its storage engine. See greylisting.org for more details on this useful anti-SPAM measure.

Now, if you’ve gotten this far, we use a package called postfix-policyd (see this site) to initiate rate limiting. Clients are limited in an hourly window on how many mails they can send and on how many recipients in total these mails can reach. The exact numbers are a moving goal post for now while we tweak it.

We could of course have used policyd for gray listing also but this posed a significant problem. Our Postfix configuration looks like this:

smtpd_recipient_restrictions =
    ...
    check_policy_service inet:127.0.0.1:10031,
    permit_mynetworks,
    permit_sasl_authenticated,
    check_policy_service inet:127.0.0.1:2525
    ...

As you can see, the issue is that we do not want to gray list our own IP range or people with a valid username and password but we do want to rate limit these. Using policyd for both is not possible unfortunately.

At this point, your mail will be accepted for relay. We not start the next batch of checks.

Content Filtering

We now need to examine these mails to discard SPAM and mails infected with a virus. For this we use the excellent amavisd-new package with SpamAssassin and ClamAV installed on the servers.

We’re still tweaking the rules for best effect but as it stands, if a virus is detected in the mail it’s silently discarded. A mail with a reasonable SPAM score is marked as SPAM in the subject but mails with a high score are also just discarded. DSNs are not sent.

If a mail passes all these checks, it’ll be relayed onwards.

Fighting SPAM is far from easy but the above will make a big dent. Unfortunately we can’t stop just there. We also have scripts to analyse the logs and rate limiting database to pull out the bad guys so the ISP’s tech support team can chase these up retroactively and instruct them on protecting their machines.

PostgreSQL 9.0 Released with Mission Critical Features

At work, we’ve always used and advocated MySQL for high availability mission critical applications thanks to it’s built in binary replication features.

This was a feature PostgreSQL sorely lacked but it is some thing that has also been rectified in the 9.0 release which, among many other features, includes:

  • Hot standby;
  • Streaming replication; and
  • In-place upgrades.

We’ll be studying and testing this new release carefully as, despite the previous lack of the above features, PostgreSQL had a lot of other features over MySQL and was also ahead of MySQL with such taken for granted features as views, triggers and stored procedures.

SIP Brute Force Attacks on the Increase

On our own Asterisk PBX server for our office and on some customer boxes with open SIP ports, we have seen a dramatic rise in brute force SIP attacks.

They all follow a very common pattern – just over 41,000 login attempts on common extensions such as 200, 201, 202, etc. We were even asked to provide some consultancy about two weeks ago for a company using an Asterisk PBX who saw strange (irregular) calls to African countries.

They were one example of such a brute force attack succeeding because of a common mistake:

  1. an open SIP port with:
  2. common extensions with:
  3. a bad password.

(1) is often unavoidable. (2) can be mitigated by not using the predictable three of four digit extension as the username. (3) is inexcusable. We’ve even seen entries such as extension 201, username 201, password 201. The password should always be a random string mixing alphanumeric characters. A good recipe for generating these passwords is to use openssl as follows:

openssl rand -base64 12

Users should never be allowed to choose their own and dictionary words should not be chosen. The brute force attack tried >41k common passwords.

Preventing or Mitigating These Attacks

You can mitigate against these attacks by putting external SIP users into dedicated contexts which limit the kinds of calls they can make (internal only, local and national, etc); ask for a PIN for international calls; limit time and cost; etc.

However, the above might be a lot of work when simply blocking users after a number of failed attempts can be much easier and more effective. Fail2ban is a tool which can scan log files like /var/log/asterisk/full and firewall IP addresses that makes too many failed authentication attempts.

See VoIP-Info.org for generic instructions or below for a quick recipe to get it running on Debian Lenny.

Quick Install for Fail2ban with Asterisk SIP on Debian Lenny

  1. apt-get install fail2ban
  2. Create a file called /etc/fail2ban/filter.d/asterisk.conf with the following (thanks to this page):
  3. Put the following in /etc/fail2ban/jail.local:
  4. Edit /etc/asterisk/logger.conf such that the date format under [general]reads:
    dateformat=%F %T
  5. Also in /etc/asterisk/logger.conf, ensure full logging is enable with a line such as the following under [logfiles]:
    full => notice,warning,error,debug,verbose
  6. Resart / reload Asterisk
  7. Start Fail2ban: /etc/init.d/fail2ban startand check:
    • Fail2ban’s log at /etc/log/fail2ban.logfor:
      INFO   Jail 'asterisk-iptables' started
    • The output of iptables --list -vfor something like:
      Chain fail2ban-ASTERISK (1 references)
       pkts bytes target     prot opt in     out     source               destination
        227 28683 RETURN     all  --  any    any     anywhere             anywhere

You should now receive emails (assuming you replaced the example addresses above with your own and your MTA is correctly configured) when Fail2ban starts or stops and when a host is blocked.

Link: MySQL Best Practices

I came across this site today which has some good advice for MySQL. I’m particularly happy to see that Doctrine, a relatively new ORM for PHP which we’re big fans of, is gaining some traction.

I came across this site today which has some good advice for MySQL. I’m particularly happy to see that Doctrine, a relatively new ORM for PHP which we’re big fans of, is gaining some traction.

I also noticed that Piwik, an open source analytics package, are using some interesting quality assurance tools which may be of interest to PHP developers (along with a continuous integration tool I came across recently: phpUnderControl).