There are times when you need to set a PHP array value from evaluated PHP code. Sometimes, you need to this using variables that won’t conflict with the current scope and with throwaway code that you won’t need again – so a closure is ideal.
Typically, you’ll need to assign the closure to a variable but this will negate the above requirement to not interfere with the current scope.
During patching for the recent GHOST bug, I updated all packages (including kernel) on a Ubuntu 14.04 file server (filer). This filer provided static content (mainly tens of thousands of images) to a number of web servers. You can see the effect in the following load graph from the filer:
Load average on the filer
You may notice from the above, that there were actually two issues. The first was solved by upgrading the filer from 14.04 to 14.10 based on a number of online references to symptoms and fixes. About an hour after this upgrade, a new form of NFS slowness manifested and, needless to say, sites that rendered in <1sec were now taking >15secs.
Diagnosing the second issue took a while longer but some tips and utilities include:
check /var/log and see if any log files are increasing rapidly;
check top and check any processes with high / unusual utilisation;
use iostat (apt-get install sysstat) and pay particular attention to any devices with high volumes of transactions per second. In my case it was the root filesystem rather than any of the mounted partitions exported by NFS.
use iotop (apt-get install iotop) and note any processes with high utilisation (in my case jbd2/xvda1-8 was at 100% and xvda1-8 is my root partition)
The jbd2 process is the ext4 journaling process. At this point you can evaluate fsck’ing your partition but I wanted to see if I could discover what was happening here. I enabled some debugging via:
# enable tracing:
echo 1 > /sys/kernel/debug/tracing/events/ext4/ext4_sync_file_enter/enable
# wait a couple of seconds and:
cat /sys/kernel/debug/tracing/trace
# and disable tracing:
echo 0 > /sys/kernel/debug/tracing/events/ext4/ext4_sync_file_enter/enable
where every entry related to the same inode number (276278). We found this via:
find / -inum 276278
/var/lib/nfs/v4recovery
The solution was to stop nfs_kernal_server, remove that directory entirely, add it back and restart the nfs_kernel_server. We got the permissions wrong on the first attempt but this’ll be obvious from dmesg / kernel log messages such as:
kernel: [53731827.778104] NFSD: Failed to remove expired client state directory 8d97cccceb37641d3804a84683a9282a
kernel: [53731827.779204] NFSD: failed to write recovery record (err -13); please check that /var/lib/nfs/v4recovery exists and is writeableNFSD: Failed to remove expired client state directory 8d97cccceb37641d3804a84683a9282a
Over at INEX we’ve embarked on a forklift upgrade of the primary peering LAN using Extreme Networks Summit x670’s and x460’s. As usual, we need to monitor these 24/7 and we have just written a new Extreme Networks chassis monitoring script which should work with most Extreme devices.
It will check and generate alerts on the following items:
a warning if the device was recently rebooted;
a warning / critical if any found temperature sensors are in a non-normal state;
a warning / critical if any found fans are in a non-normal state;
a warning / critical if any found PSUs are in a non-normal state (or missing);
a warning / critical if the 5 sec CPU utilisation is above set thresholds;
a warning / critical if the memory utilisation is above set thresholds.
This post relates to creating an Android application with Facebook support (via phonegap-facebook-plugin using Cordova 3.5 and, specifically, without Eclipse. Most existing documentation expects you to spin up Eclipse to link the Facebook libraries – but, we’re using an automated build system with Grunt so that’s not a possibility.
I arrived to work this morning to find that the Heartbleed Bug happened.
This is one of the biggest security issues to crop up in a long time – allowing the data normally protected by TLS/SSL to be compromised. This is the kind of data that normally passes securely between clients and protected websites, email services, instant messaging, etc.
Upgrade all your systems now. This is where my well planned day went.
Be sure to restart all services that use OpenSSL (or reboot your servers). A useful command [source] for this post-upgrade is:
A useful Python script for testing your web servers can be found in this Gist. NB: it’s not just web servers affected – any services with SSL/TLS may be affected.
OSS_SNMP is a PHP SNMP library written by myself for people who hate SNMP. After a customer migration from PVST to MST (Multiple Spanning Tree), I have added a number of MST functions / MIBs to OSS_SNMP:
During a fairly significant network migration involving breaking / connecting a number of links, I wanted to be able to monitor the MST port role of significant ports at a glance. For this purpose, I wrote the mst-port-roles.php script and have committed it as an example to OSS_SNMP. First, here is what it looks like when run on the command line (with hostnames obfuscated):
From a very simple array of port details at the top of the script, it will poll all switches and for each port print:
device and port name;
port state and speed;
port role for each applicable MST instance.
I run it on bash and use bash colouring. The script is well documented and can easily be repurposed for other networks. You’ll find the source here.
With the documentation above, I created a null modem with loop back handshaking cable using two wire-able female DB9-to-RJ45 converters and a standard straight-through network cable following the pin positions in the above link.
To locally connect and loop back  pins  1+4+6 and 7+8, I snipped those wires, striped the ends and just twisted together and covered with electrical tape. Pin 5 is connected to its opposite number and you only then need to ensure that pins 2 and 3 are crossed over.
This worked a charm with my USB serial port. After I tried about a dozen various unlabeled console cables I have lying around that is…
Update 2019: we now use PrivateBin a lot. It is a self-hosted, minimalist, open source online pastebin where the server has zero knowledge of pasted data.
Pastebin has been a valuable tool for years – to the extent that pastebin it has entered the common lexicon of sysadmins, network engineers and developers.
There are, however, a few notable alternatives:
GitHub Gists – what’s particularly cool about these is that each Gist (which is just pasted text) is also a fully fledged Git repository with versioning and the ability for fork. There’s also syntax highlighting and a nice UI. If you’re a GitHub user, your own Gists are also linked to your account.
p.ip.fi – this scores big points for its pure simplicity. You’d argue that a pastebin doesn’t really need a complex UI and p.ip.fi is laudable in its complete lack of a UI. Just paste and hit Ctrl-S and you’re done. Very nice. (Credit to Nick for pointing this out).
sprunge.us – this is a command line pastebin which should appeal directly to sysadmins and network engineers. (Credit to dnolan for leading me to this one). This is best demonstrated via an epic traceroute:
