BarryODonovan.com

I use Cacti to monitor a lot of Dell servers, primarily 1850s and 2850s but also the newer models of same (1950s and 2950s). One itch that I’ve meant to scratch for a while is graphing some of the information available through the servers’ IPMI interface; specifically the servers’ various temperatures and and fan speeds.

IPMI Details

There are patches available for the Linux kernel to allow the IPMI information to be read via the lm_sensors project but I chose to avoid this (at least for now) as I’d have to schedule downtime to reboot the servers for a new kernel. It’d also ruin their uptime – most of the servers (serving many thousands of users daily) have almost two years of uptime. (The kernels are monolithic.)

Instead, I went with the already compiled in Linux IPMI Driver (see kernel source: Documentation/IPMI.txt) which is available in the ‘Character Devices’ menu. I specifically needed the following options for the Dells:

• drivers/char/ipmi/ipmi_msghandler
• drivers/char/ipmi/ipmi_devintf
• drivers/char/ipmi/ipmi_si

In order to read information from the IPMI, you need the ipmitool utility which is available on most recent Linux distributions or from here.

Lastly, I needed to create a character special file to interface with the IPMI:

mknod /dev/ipmi0 c 254 0

The sensor information was then available via:

# ipmitool sensor
Temp             | 30.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 90.000    | na
Temp             | 34.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 90.000    | na
Ambient Temp     | 16.000     | degrees C  | ok    | na        | 3.000     | 8.000     | 42.000    | 47.000    | na
...

Making IPMI Sensor Information Available via SNMP

I make the IPMI sensor information available over SNMP by adding the following to the snmpd.conf file:

# Monitor IPMI Temperature and Fan stats
exec    .1.3.6.1.4.1.X.1000 ipmitemp        /usr/local/sbin/ipmi-temp-stats
exec    .1.3.6.1.4.1.X.1001 ipmifan         /usr/local/sbin/ipmi-fan-stats

(Replace X above as appropriate.)

The scripts referenced are: /usr/local/sbin/ipmi-temp-stats:

#! /bin/sh

PATH=/usr/bin:/bin
STATS=/tmp/ipmisensor-snmp

printf "%f\n" `cat $STATS | grep Temp | cut -s -d "|" -f 2`

And /usr/local/sbin/ipmi-fan-stats:

#! /bin/sh

PATH=/usr/bin:/bin
STATS=/tmp/ipmisensor-snmp

printf "%f\n" `cat $STATS | grep FAN | cut -s -d "|" -f 2`

The file they reference is generated every 5mins (Cacti polling interval) via a cron entry in the file /etc/cron.d/ipmitool:

*/5 * * * * root /usr/bin/ipmitool sensor >/tmp/ipmisensor-snmp

After restarting SNMP and allowing the cron job to execute at least once, you can test the results via:

# snmpwalk -c <community> -v <version> <ip/hostname> .1.3.6.1.4.1.X.1000
SNMPv2-SMI::enterprises.X.1000.1.1 = INTEGER: 1
SNMPv2-SMI::enterprises.X.1000.2.1 = STRING: "ipmitemp"
SNMPv2-SMI::enterprises.X.1000.3.1 = STRING: "/usr/local/sbin/ipmi-temp-stats"
SNMPv2-SMI::enterprises.X.1000.100.1 = INTEGER: 0
SNMPv2-SMI::enterprises.X.1000.101.1 = STRING: "37.000000"
SNMPv2-SMI::enterprises.X.1000.101.2 = STRING: "39.000000"
SNMPv2-SMI::enterprises.X.1000.101.3 = STRING: "23.000000"
SNMPv2-SMI::enterprises.X.1000.101.4 = STRING: "36.000000"
...
SNMPv2-SMI::enterprises.X.1000.102.1 = INTEGER: 0
SNMPv2-SMI::enterprises.X.1000.103.1 = ""

Graphing This Information in Cacti

Finally, I graph this information on Cacti (see end of post for examples).

I am making six templates available here which can be imported into Cacti (these were generated using version 0.8.6j) for graphing the above:

1. Cacti graph template for Dell 1850 temperatures (see first image below);
2. Cacti graph template for Dell 2850 temperatures (see second image below);
3. Cacti graph template for Dell 1850 fan speeds (see third image below);
4. Cacti graph template for Dell 2850 fan speeds (see fourth image below);
5. Cacti host template for Dell 1850; and
6. Cacti host template for Dell 2850.

The last two templates available are host templates for Dell 1850s and 2850s (I’m sure they’ll work fine with 1950s and 2950s also). These templates include:

• Host MIB – Logged in Users;
• Host MIB – Processes;
• IPMI Fan Speeds (Dell x850) (from above);
• IPMI Temperatures (Cel) (Dell x850) (from above);
• ucd/net – CPU Usage;
• ucd/net – Load Average;
• ucd/net – Memory Usage;
• SNMP – Get Mounted Partitions (data query); and
• SNMP – Interface Statistics (data query).

Example graphs are shown below; they’re not the cleanest given the amount of information they contain but they serve my purposes.

© 2007 Barry O’Donovan. All text is licensed under a Creative Commons Attribution 3.0 License. All scripts and Cacti templates are licensed under the MIT License.

Pádraig Brady wrote a very useful script for getting the optimum gcc options for your CPU which I keep coming back to (and forgetting where to find it).

The last version can be found at http://www.pixelbeat.org/scripts/gcccpuopt along with a lot of other useful scripts and scripting examples here.

A question came up today on ILUG regarding Skype on Linux which then spill-ed over onto a conversation about VoIP clients. KPhone was mentioned which is what I have been using to date with my Blueface VoIP account. Unfortunately I can’t give KPhone a good review as I have always found it buggy, unintuitive and it crashes regularly.

The conversation reminded me of a news bite I read on KDE.news about new Linux VoIP clients that are gaining momentum. One in particular looked very promising: Twinkle. The first version of Twinkle, 0.1, was only released last month but it’s already a formidable application when compared with KPhone.

Although Twinkleuses the cross platform application development framework known as Qt (which is also the foundation of KDE), it is only compatible with Linux’s audio system. Some of the features already completed include two lines, three-way conference calls, call redirection, DTMF sopport and the G.771 and GSM audio codecs.

Some obvious features that are currently missing but that the author plans to add include an address book, a history function to log incoming and outgoing calls, instant messaging and video support. So far I’m very impressed and I have already replaced KPhone with Twinkle.

The only negative comment I have to make, and it’s not really a reflection on Twinkle, is that although the author decided to use Qt it is really a shame he didn’t go the extra step and use the KDE application framework so that it would better integrate with that desktop environment and the other KDE PIM and networking applications. No doubt Michel de Boer, the author, has his reasons – perhaps he plans to extend Twinkle‘s compatibility to other operating systems.

Mikolaj Machowski posted an nice example of using KDialog with DCOP to the KDE Developers mailing list a while back:


#!/bin/bash

if [ $PROGRESS ]; then
  for (( i=0; i<30; i++ )); do
    dcop $PROGRESS setProgress $i
    sleep 1
  done
  dcop $PROGRESS close
fi


The purpose of Mikolaj's post was to suggest a method of regimenting work periods - e.g. 20 minutes on, 5 minutes off - using the above script and KAlarm, a personal alarm message, command and email scheduler. But, more than that, it shows off one of the many hiddens treasures of KDE: KDialog, which allows shell scripts to take advantage of some of the KDE widget set, and DCOP, KDE's Desktop COmmunications Protocol.

More information and a tutorial can be found at:

http://developer.kde.org/documentation/tutorials/kdialog/t1.html

I got CC’d a reply to an e-mail from Ms Kathy Sinnott MEP today which asserts her strong opposition to EU Directive COD/2002/0047 which includes the issue of software patentability. Even in one small paragraph she shows a good understanding of the issue and I am delighted that such a hard working and articulate MEP is on board.

In her own words:

I am most certainly not [undecided on this issue]. In response to your concerns about software patentability, I would like to reassure you that I am wholeheartedly opposed to this legislation. It will stifle innovation and prevent SME’s and individuals from having the chance to compete equally with the Microsoft’s of this world. You may rest assured that we will be fighting this directive (again).

As it currently stands, we now have eight of our sixteen (inc. Northern Ireland) MEPs committed to opposing^* software patentability with five yet to inform us of their position and three taking the matter under consideration. Optimistically, the final count could be as good as twelve or thirteen opposing this.

The next big task will be to ensure that all those opposing software patentability will actually turn up in parliament as an absolute majority will be required to defeat or amend the directive; this essentially means that any MEP who does not show up in parliament to vote will be effectively casting their vote in favour of the directive. What a wonderful democracy we have.

More information on the Irish campaign can be found at KDE.ie and at the Irish Free Software Organisation. The current status of the sixteen Irish MEPs is also available at KDE.ie and with more detail here.

^* by oppose I refer to an MEP being against the patentability of software ideas and business methods; this does not reflect whether an MEP is against the directive itself or would prefer to see it amended.