This is just a copy of my notes rather than a how-to but any competent Linux user should have no problem. Apologies in advance for the brevity; with luck you’ll be using a later version of Linux which will already have solved the network issue…

The two main issues and fixes were:

• The SATA CD-ROM was not recognised initially and the fix was set the following parameter in the BIOS:

  BIOS -> Integrated Peripherals -> SATA Mode -> RAID

• The network card is not recognised during a Linux install. Allow install to complete without network and then download Intel’s e1000 driver from http://downloadcenter.intel.com/ or specifically by clicking here. The one I used then was e1000-7.6.9.tar.gz but the current version appears to be e1000-7.6.15.tar.gz (where the above link heads to – check for later versions).

  My only notes of the install just say “essentially follow instructions in README” so I assume they were good enough! Obviously you’ll need Linux kernel headers at least as well as gcc and related tools.

  Once built and installed, a:

  modprobe e1000

  should have you working. Use dmesg to confirm.

Now on this particular server, sda and sdb were/are a RAID1 (containing the OS) and a RAID5 partition respectively and sdc was/is a 4TB RAID5 partition from an externally attached disk chassis. This was a server re-installation and all data from sda and sdb had multiple snapshots off site. sdc had no backups of its 4TBs of data.

The installer discovered the drives in a different order and sda became the externally attached drive. I, believing it to be the internal RAID1 array, allowed the installer to initialise it. Oh shit…

Now this wouldn’t be the end of the world. It wasn’t backed up because a copy of the data exists on removable drives in the UK. It would mean someone flying in with the drives, handing them off to me at the airport, bringing them to the data center and copying all the data back. Then returning the drives to the UK again. A major inconvenience. And it’s also an embarrassment as I should have ensured that sda is what I thought it was via the installers other screens.

Anyway – from what I could make out, the installer initialised the drive with a single partition spanning the entire drive.

Once I got the operating system reinstalled, I needed to try and recover the LVM partitions. There’s not a whole lot of obvious information on the Internet for this and hence why I’m writing this post.

The first thing I needed to do was recreate the physical volume. Now, as I said above, I had backups of the original operating system. LVM creates a file containing the metadata of each volume group in /etc/lvm/backup in a file named the same as the volume group name. In this file, there is a section listing the physical volumes and their ids that make up the volume group. For example (the id is fabricated):

physical_volumes {
        pv0 {
                id = "fvrw-GHKde-hgbf43-JKBdew-rvKLJc-cewbn"
                device = "/dev/sdc"     # Hint only

                status = ["ALLOCATABLE"]
                pe_start = 384
                pe_count = 1072319      # 4.09057 Terabytes
        }
}

Note that after I realised my mistake, I installed the OS on the correct partition and after booting, the external drive became /dev/sdc^* again. Now, to recreate the physical volume with the same id, I tried:

# pvcreate -u fvrw-GHKde-hgbf43-JKBdew-rvKLJc-cewbn /dev/sdc
  Device /dev/sdc not found (or ignored by filtering).

Eh? By turning on verbosity, you find the reason among a few hundred lines of debugging:

# pvcreate -vvvv -u fvrw-GHKde-hgbf43-JKBdew-rvKLJc-cewbn /dev/sdc
...
#filters/filter.c:121         /dev/sdc: Skipping: Partition table signature found
#device/dev-io.c:486         Closed /dev/sdc
#pvcreate.c:84   Device /dev/sdc not found (or ignored by filtering).

So pvcreate will not create a physical volume using the entire disk unless I remove partition(s) first. I do this with fdisk and try again:

# pvcreate -u fvrw-GHKde-hgbf43-JKBdew-rvKLJc-cewbn /dev/sdc
  Physical volume "/dev/sdc" successfully created

Great. Now to recreate the volume group on this physical volume:

# vgcreate -v md1000 /dev/sdc
    Wiping cache of LVM-capable devices
    Adding physical volume '/dev/sdc' to volume group 'md1000'
    Archiving volume group "md1000" metadata (seqno 0).
    Creating volume group backup "/etc/lvm/backup/md1000" (seqno 1).
  Volume group "md1000" successfully created

Now I have an “empty” volume group but with no logical volumes. I know all the data is there as the initialization didn’t format or wipe the drive. I’ve retrieved the LVM backup file called md1000 and placed it in /tmp/lvm-md1000. When I try to restore it to the new volume group I get:

# vgcfgrestore -f /tmp/lvm-md1000 md1000
  /tmp/lvm-md1000: stat failed: Permission denied
  Couldn't read volume group metadata.
  Restore failed.

After a lot of messing, I copied it to /etc/lvm/backup/md1000 and tried again:

# vgcfgrestore -f /etc/lvm/backup/md1000 md1000
  Restored volume group md1000

I don’t know if it was the location, the renaming or both but it worked.

Now the last hurdle is that on a lvdisplay, the logical volumes show up but are marked as:

  LV Status              NOT available

This is easily fixed by marking the logical volumes as available:

#  vgchange -ay
  2 logical volume(s) in volume group "md1000" now active

Agus sin é. My logical volumes are recovered with all data intact.

^* how these are assigned is not particularly relevant to this story.

To this end, I have written a Nagios plugin which queries Asterisk’s manager interface and executes the pri show spans CLI command (this is Asterisk 1.4 by the way). The script then parses the output to ascertain whether a PRI is up or not.

The actual code to connect to the manager interface and execute the query is simply:

if( ( $astsock = fsockopen( $host, $port, $errno, $errstr, $timeout ) ) === false )
{
    echo "Could not connect to Asterisk manager: $errstr";
    exit( STATUS_CRITICAL );
}

fputs( $astsock, "Action: Login\r\n");
fputs( $astsock, "UserName: $username\r\n");
fputs( $astsock, "Secret: $password\r\n\r\n"); 

fputs( $astsock, "Action: Command\r\n");
fputs( $astsock, "Command: pri show spans\r\n\r\n");

fputs( $astsock, "Action: Logoff\r\n\r\n");

while( !feof( $astsock ) )
{
    $asttext .= fread( $astsock, 8192 );
}

fclose( $astsock );

if( strpos( $asttext, "Authentication failed" ) !== false )
{
    echo "Asterisk manager authentication failed.";
    exit( STATUS_CRITICAL );
}

This plugin is hard coded to English and expects to find Provisioned, Up, Active for a good PRI. For example, the Asterisk implementations that support the pri show spans command that I have access to return one of:

• PRI span 1/0: Provisioned, In Alarm, Down, Active
• PRI span 3/0: Provisioned, Up, Active

I’m actually running a slightly older version of Nagios at the moment, version 1.3. To integrate the plugin, first add the following command definition to an appropriate existing or new file under /etc/nagios-plugings/config/:

define command{
        command_name    check_asterisk_pri
        command_line    /usr/lib/nagios/plugins/check_asterisk_pri.php \\
             -H $HOSTADDRESS$ -U $ARG1$ -P $ARG2$ -w $ARG3$ \\
             -c $ARG4$ -n $ARG5$
}

where $ARG1$ is the Asterisk manager username and $ARG2$ is the password. $ARG3$ and $ARG4$ are the warning and critical thresholds respectively whereby if the number of available PRIs reaches one of these values, the appropriate error condition will be set. Lastly, $ARG5$ is the number of PRIs the plugin should expect to find.

NB: the command_line line above is split for readability but it should all be on the one line.

Now create a test for a host in an appropriate file in /etc/nagios/config/:

define service{
        use                             core-service
        host_name                       hostname.domain.ie
        service_description             Asterisk PRIs
        check_command                   check_asterisk_pri!user!pass!2!1!4
}

Ensure that your Nagios server has permissions to access the Asterisk server via TCP on the Asterisk manager port (5038 by default). If on a public network, this should be done via stunnel or a VPN for security reasons.

Lastly, you’ll need a user with the appropriate permissions and host allow statements in your Asterisk configuration (/etc/asterisk/manager.conf):

[username]
secret = password
deny=0.0.0.0/0.0.0.0
permit=1.2.3.4/255.255.255.255
read = command
write = command

The next version may include support for BRI and Zap FXO ports also. I also plan on a Cacti plug in to show the channels on each PRI (up – on a call, down, etc). In any case, updates will be posted here.

The plug in can be download from: http://www.opensolutions.ie/misc/check_asterisk_pri.php.txt

For those that don’t know, this feature has existing in KDE in multiple forms for some time. For example, pressing ALT-F2 opens the Run Command dialog and typing, for example:

php:fopen

will bring up PHP.net’s own search page. The same goes for the location bar in Konqueror.

By the way, other nice short cuts in the Run Command dialog include:

• gg: <keywords> for a quick Google search;
• wp: <keywords> for a quick Wikipedia search;
• dict: <keyword> for a quick dictionary look-up;
• man: <keyword> for a man page look-up;
• info: <keyword> for an info page look-up;
• rfc: <number> to be brought to the relevant RFC page;

Of course, entering a command will execute it and just entering a URL will open it in Konqueror.

Unfortunately there didn’t seem to be a lot of information out there on Promise’s SNMP MIBs so with a bit of playing about, I was able to dig out the ones I needed. The Nagios plug-in I wrote and am making available here will monitor the shelf via SNMP and alert on the following chassis issues:

• critical if any of the shelf’s disk states changes from “OK”;
• warning if the battery state changes from “FullyCharged”;
• critical if either of the PSU states change from “Powered On and Functional”;
• critical is any of the cooling devices (fans) change from “Functional”;
• critical if any of the temperature sensors’ states change from “normal”;
• critical if any of the drives go offline or are missing; and
• warning if any of the drives go into the rebuilding state or have their PFA flag set.

While this is specifically designed for a single M200i, it should be easily customisable for other models.

It can be downloaded from here (http://www.opensolutions.ie/). It will also appear on the development section of this site and Nagios Plugins.

OIDs Used

1.3.6.1.4.1.7933.1.10.2.1.1.1.8

The table of physical disk statuses.

.1.3.6.1.4.1.7933.2.1.7.1.1.14.1.1

The battery status.

.1.3.6.1.4.1.7933.2.1.4.1.1.2.1

The table of Power Supply Unit statuses.

.1.3.6.1.4.1.7933.2.1.3.1.1.3.1

The table of cooling device/fan statuses.

.1.3.6.1.4.1.7933.2.1.5.1.1.3

The table of temperature sensor statuses.

.1.3.6.1.4.1.7933.1.10.1.2.1.1.22.1

The number of drives that are offline.

.1.3.6.1.4.1.7933.1.10.1.2.1.1.23.1

The number of drives in the PFA status set.

.1.3.6.1.4.1.7933.1.10.1.2.1.1.24.1

The number of drives in rebuild status.

.1.3.6.1.4.1.7933.1.10.1.2.1.1.25.1

The number of drives that are missing.

There were two aspects required:

1. The sending of alerts via the SMS gateway;
2. The monitoring of available credits on the SMS gateway;

1. Send Alerts via SMS Gateway

The sendsms script is:

#! /bin/bash

USERNAME=username
PASSWORD=password
SENDSMSADDRESS="https://www.kapow.co.uk/scripts/sendsms.php"
MAXMSGLENGTH=320

read -n $MAXMSGLENGTH -r MSG

MSG=`php -r "echo urlencode( \"$MSG\" );"`

wget -q -O - "$SENDSMSADDRESS?username=$USERNAME&password=$PASSWORD&mobile=$1&sms=$MSG"

I use a quick hack with PHP to URL encode the string. I didn’t know a shell command off hand but I’m open to suggestions. This can be tested with:

echo This is a test message | sendsms 353861234567

Edit /etc/nagios/misccommands.cfg to include the following:

# 'host-notify-by-sms' command definition
define command{
        command_name    host-notify-by-sms
        command_line    /usr/bin/printf "%b" "Host '$HOSTALIAS$' is $HOSTSTATE$: $OUTPUT$" | /usr/local/bin/sendsms $CONTACTPAGER$
        }

# 'notify-by-sms' command definition
define command{
        command_name    notify-by-sms
        command_line    /usr/bin/printf "%b" "$NOTIFICATIONTYPE$: $SERVICEDESC$@$HOSTNAME$: $SERVICESTATE$ ($OUTPUT$)" | /usr/local/bin/sendsms $CONTACTPAGER$
        }

Ensure your /etc/nagios/contacts.cfg is updated to include notification by SMS with your mobile number:

define contact{
        contact_name                    barryo
        alias                           Barry O'Donovan
        service_notification_period     barryoworkhours
        host_notification_period        barryoworkhours
        service_notification_options    w,u,c,r
        host_notification_options       d,u,r
        service_notification_commands   notify-by-email,notify-by-sms
        host_notification_commands      host-notify-by-email,host-notify-by-sms
        email                           joe@bloggs.com
        pager                           353868017669
}

Sin é.

2. Monitor SMS Gateway Credits

The plugin code is:

#! /bin/bash

USERNAME=username
PASSWORD=password
CHECKCREDITSADDRES="https://www.kapow.co.uk/scripts/chk_credit.php"

CRIT=$1
WARN=$2

CREDITS=`wget -q -O - "$CHECKCREDITSADDRES?username=$USERNAME&password=$PASSWORD"`

if [[ -z $CREDITS || ! $CREDITS -ge 0 ]]; then
        echo -e "$CREDITS\\n";
        exit 3;
elif [[ $CREDITS -le $CRIT ]]; then
        echo -e "$CREDITS SMS credits remaining\\n";
        exit 2;
elif [[ $CREDITS -le $WARN ]]; then
        echo -e "$CREDITS SMS credits remaining\\n";
        exit 1;
else
        echo -e "$CREDITS SMS credits remaining\\n";
        exit 0;
fi

Create a plugin configuration file for Nagios, say /etc/nagios-plugins/config/sms_credits.cfg:

# 'check_sms_credits' command definition
define command{
        command_name    check_sms_credits
        command_line    /usr/local/bin/check_sms_credit $ARG2$ $ARG1$
        }

Where $ARG1$ is the warning threshold and $ARG2$ is the critical threshold.

I add the service to the Nagios monitoring box via /etc/nagios/config/sms_credit.cfg:

#
# check sms credits on Kapow - barryo 20070519
#

define service{
        use                             core-service
        host_name                       noc
        service_description             SMS Credits
        check_command                   check_sms_credits!50!100
}

And I believe that’s it.

*) The monitoring box is in a different country to the servers it monitors so a network failure will not prevent the alert getting out.