If you’re like me and most humans, the Nagios logfile timestamp (a unix timestamp) isn’t terribly useful when trying to grep through the logs and correlate events:
# head -2 nagios.log
[1350360000] LOG ROTATION: DAILY
[1350360000] LOG VERSION: 2.0
Here’s a nifty Perl one-liner that you can pipe your logs through:
perl -pe ‘s/(\d+)/localtime($1)/e’
to get nicer output like:
# head -2 nagios.log
[Tue Oct 16 00:00:00 2012] LOG ROTATION: DAILY
[Tue Oct 16 00:00:00 2012] LOG VERSION: 2.0
Tag Archives: perl
Tools for watching apache httpd and memcached
Recently I was working on a code release on a site running PHP on Apache httpd, and using >memcached. Without getting into specifics, we had a number of issues that were both Apache and memcached problems, and little visibility into them as it was running on an older server without much monitoring in place. I started looking around for simple tools that could provide a bit more insight, without many dependencies (as the machine is a relatively minimalist install). Here are some of the options I found:
- memcache-top – A top-like script that pulls stats from memcached instances and can show both per-instance, total and average usage %, hit rate, number of connections, time to run the stats query, evictions, gets, sets, and read and write amounts. Best of all, it’s a very small perl script that requires only IO::Socket and Time::HiRes. Here’s a small example of the output:
memcache-top v0.6 (default port: 11211, color: on, refresh: 3 seconds) INSTANCE USAGE HIT % CONN TIME EVICT GETS SETS READ WRITE 127.0.0.1:11211 86.6% 99.4% 115 0.6ms 0.0 4114 1669 1.3M 24.2M 127.0.0.1:11212 85.5% 59.9% 2 0.4ms 0.0 0 0 90 8055 AVERAGE: 86.0% 79.6% 58 0.5ms 0.0 2057 834 682.4K 12.1M TOTAL: 0.9GB/ 1.0GB 117 1.0ms 0.0 4114 1669 1.3M 24.2M
- damemtop is also a nice top-like memcached tool. On the positive side, you can specify any column from “stats”, “stats items” or “stats slabs” in the configuration file, and can choose between average or one-second snapshots for each column. On the down side, it requires the YAML and AnyEvent Perl modules, so it has some uncommon dependencies.
damemtop: Tue Jun 26 14:02:24 2012 [sort: hostname asc] [delay: 3s] hostname all_version all_fill_rate hit_rate evictions curr_items curr_connections cmd_get cmd_set bytes_written bytes_read get_hits get_misses TOTAL: NA NA NA NA NA NA NA 87 32 491,735 30,894 86 1 AVERAGE: NA NA 86.00% 99.00% NA NA NA 43 16 122,933 7,723 43 1 10.200.1.78:11211 1.2.6 86.63% 98.04% 0 0 -1.00204024880524 51 19 386,492 21,613 50 1 10.200.1.78:11212 1.2.6 85.46% NA 0 0 0 0 0 11,373 31 0 0 10.200.1.79:11211 1.2.6 87.31% 100.00% 0 0 -1.00204024880524 36 13 82,479 9,219 36 0 10.200.1.79:11212 1.2.6 85.08% NA 0 0 0 0 0 11,389 31 0 0 loop took: 0.305617094039917
I’m still looking around for something for apache that uses mod_status and isn’t too verbose; ideally I’d like to be able to watch memcached, apache response codes/times, and apache mod_status all in the same terminal window.
Script to Chart Intervals Between Problem and Recovery from Nagios/Icinga Log Files
At work, we use Icinga (a fork of Nagios) for monitoring. We have a few services which are restarted or otherwise poked by event handlers, but the recovery takes a while – so we often get paged for problems which recover in a few minutes. I wrote a small perl script that greps through the archived log files for a given regex (service and/or host name) and then calculates the time from problem to recovery and graphs those times.
The script is called nagios_log_problem_interval.pl and can be downloaded from my github. Below is some sample output, the number of minutes from problem to recovery are along the Y axis and the count is along the X axis:
> nagios_log_problem_interval.pl --archivedir=/var/icinga/archive --match=myhost --backtrack=10
myhost;HTTP
Count
1:########(8)
2:##(2)
3:#(1)
4:##(2)
5:#######(7)
6:(0)
7:(0)
8:#(1)
9:(0)
10:(0)
11:#(1)
12:(0)
13:#(1)
14:(0)
15:(0)
16-29:(0)
30-59:(0)
60+:(0)
Creating RPMs from Perl CPAN Modules
I try my absolute best to always install software on my Linux boxes as RPMs, installed through Yum (yes, I use CentOS on servers and Fedora on my desktops/laptops). Not only is this more-or-less required to sanely manage configuration through Puppet, but it also lets me recreate a machine, or install dependencies for something, in one simple command line. Unfortunately, I run quite a bit of Perl code, and there are a lot of CPAN Perl modules that aren’t in any of the usual CentOS/Fedora repositories.
Enter cpan2rpm: a Perl script that, in its simplest invocation, downloads a specified CPAN module and automatically builds RPMs and SRPMs for it. The original version by Erick Calder hasn’t been touched since 2005, but there’s a newer version from Mediaburst, cpan2rpmmb, that seems to incorporate some nice improvements and worked quite well for me.
Perl script to convert F5 BigIp VIP address to list of internal pool member addresses
I often find myself logging in to the web UI of F5 BigIp load balancers and tracing down a VIP address to the servers that actually back it. This is an arduous, repetitive task of tracing from the VIP list to the VIP details page to find the default pool, then matching up that in the pool list and checking the pool members page. Luckily, the F5 boxes have a web service API that can be used for tasks like this. They have GPL sample code in Perl that uses only SOAP::Lite (as well as Getopt::Long and Pod::Usage) to interact with an F5 BigIp. I wrote a simple script to trace a VIP to the appropriate internal pool member addresses, assuming you have a simple configuration of VIP -> Single default pool -> pool members.
Usage is quite simple:
> ./VipToInternalHosts.pl --host=prod-lb1.example.com --user=myname --pass=mypassword --vip=128.6.30.130:80 VIP 128.6.30.130:80 (f5_vip_name) -> Pool 'pool_name' Members of Pool 'pool_name': 10.145.15.10:80 10.145.15.11:80 |
The code can be found at http://svn.jasonantman.com/misc-scripts/VipToInternalHosts.pl via either HTTP or SVN. I hope it’s of use to someone else as well.
VMWare vSphere CLI and Perl SDK as an RPM
Lately I’ve been playing around with the VMWare vSphere SDK for Perl, since the new job uses a bunch of VMWare stuff (and I’ve been starting my foray into Perl as a new language, and am amazed by the massive number of modules out there). As much as I find yum limiting having used zypper on OpenSuSE, I’m not much of a fan of non-natively-packaged software. Not only is it more difficult to maintain and upgrade a system and nearly impossible to nicely automate when building from source (or a proprietary installer script), it’s also much more difficult to transition from a development environment to production.
In a quick search, I found a perfectly working spec file and some RHEL/Cent-specific patches (and even beginner-level rpmbuild instructions) for the current 5.0.0-422456 VMWare CLI and Perl SDK for x86_64 at http://www.firetooth.net/confluence/display/public/vSphere+Perl+SDK+and+CLI+RPM+Packages. Many thanks to Vaughan Whitteron of NSW in Australia for posting this! It built and installed without any problems on my Fedora 16 desktop, and a CentOS 6.2 development box.
Sending AOL Instant Messenger (AIM) Messages from a Perl Script
I’ve been doing some work on icinga (a Nagios fork) and wanted to implement notification via AOL Instant Messenger (AIM), since I’m almost always signed on when I’m at a computer. Unfortunately, most of the scripts that I could find use Net::AIM::TOC which implements a now-defunct protocol. So, I found Perl’s Net::OSCAR and James Nonnemaker’s script, and decided to rework them into something a bit more full-featured.
The below script sends a single IM to a single contact via the command line (using a specified AIM username and password). It’s intended to be a Nagios notification script (using the configurations shown below), but could be used for any purpose. The most up-to-date version of the script will be available at: github.com/jantman/public-nagios/master/send_aim.pl
#!/usr/bin/perl # # Script to send AIM messages from the command line # # Copyright 2012 Jason Antman <http://blog.jasonantman.com> <jason@jasonantman.com> # based on the simple version (C) 2008 James Nonnemaker / james[at]ustelcom[dot]net # found at: <http://moo.net/code/aim.html> # # The canonical, up-to-date version of this script can be found at: # <http://svn.jasonantman.com/public-nagios/send_aim.pl> # # For updates, news, etc., see: # <http://blog.jasonantman.com/2012/02/sending-aim-messages-from-a-perl-script/> # # $HeadURL$ # $LastChangedRevision$ # use strict; use warnings; use Net::OSCAR qw(:standard); use Getopt::Long; my ($screenname, $passwd, $ToSn, $Msg); my $VERSION = "r17"; my $result = GetOptions ("screenname=s" => \$screenname, "password=s" => \$passwd, "to=s" => \$ToSn); if(! $screenname || ! $passwd || ! $ToSn) { print "send_aim.pl $VERSION by Jason Antman <jason\@jasonantman.com>\n\n"; print "USAGE: send_aim.pl --screenname=<sn> --password=<pass> --to=<to_screenname>\n\n"; } # slurp message from STDIN my $holdTerminator = $/; undef $/; $Msg = <STDIN>; $/ = $holdTerminator; my @lines = split /$holdTerminator/, $Msg; $Msg = "init"; $Msg = join $holdTerminator, @lines; my $oscar = Net::OSCAR->new(); $oscar->loglevel(0); $oscar->signon($screenname, $passwd); $oscar->set_callback_snac_unknown(\&snac_unknown); $oscar->set_callback_im_ok (\&log_out); $oscar->set_callback_signon_done (\&do_it); while (1) { $oscar->do_one_loop(); } sub do_it { $oscar->send_im($ToSn, $Msg); } sub log_out { $oscar->signoff; exit; } sub snac_unknown { my($oscar, $connection, $snac, $data) = @_; # just use this to override the default snac_unknown handler, which prints a data dump of the packet } |
The command line usage is pretty simple – it takes the message to send on stdin and parameters for the sender’s screen name and password, and the recipient’s screen name, like:
echo -e "Hello\nworld\n" | send_aim.pl --screenname=mySN --password=myPass --to=recipientSN |
The Icinga configs that I used for this are as follows. I just used the default Icinga 1.6 notify by email commands, since AIM should handle the full length fine.
# host notification command define command{ command_name notify-host-by-aim command_line /usr/bin/printf "%b" "***** Icinga *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/lib/nagios/plugins/notification/send_aim.pl --screenname=mySN --password=myPass --to=$CONTACTADDRESS1$ } # service notification command define command{ command_name notify-service-by-aim command_line /usr/bin/printf "%b" "***** Icinga *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /usr/lib/nagios/plugins/notification/send_aim.pl --screenname=mySN --password=myPass --to=$CONTACTADDRESS1$ } # example contact define contact{ contact_name joeadmin alias Joe Admin use generic-with-AIM-contact email joeadmin@example.com pager 5555555555@vtext.com address1 joeAdminSN ; AIM screen name } |
F5 BigIp – Manually Changing Session Persistence Cookies on the Client Side
Yesterday I was asked to help out a bit debugging issues with a site that sits behind a F5 BIG-IP load balancer (LB). It’s a pretty simple site, load balanced between two web servers. The developers were complaining about intermittent page load issues, so I immediately considered a problem with one of the two servers (assuming that the devs were clearing cookies and cache between attempts). The LB is using HTTP Cookies for client session persistence, but no matter how many times I cleared my cookies, I kept being sent to the same back-end server. I know I could have added an iRule to the LB, but it seems like bad practice to change a production configuration for debugging something like this.
If your site uses a BigIp with cookies for persistence, it’s no problem to edit the cookies manually to force yourself to another back-end server. Simply look through the cookies for a given site using something like the Web Developer addon for Firefox; the BigIp cookie is named like “BigIpServer<poolname>”. The encoding information is specified by F5 in their knowledge base sol6917: Overview of BIG-IP persistence cookie encoding. I also managed to find a Perl one-liner from Tyler Krpata, Manger of Security Engineering at Constant Contact, in a post on his blog. I built on that work to develop the following perl script, which can both encode and decode BigIP cookie IP/port values.
#!/usr/bin/perl # # Perl script to de/encode F5 BigIp persistence cookies. # # The latest version of this script can always be obtained from: # <http://svn.jasonantman.com/misc-scripts/bigipcookie.pl> via HTTP ot SVN # # Update information and description can be found at: # <http://blog.jasonantman.com/?p=931> # # Copyright 2012 Jason Antman <jason@jasonantman.com> <http://www.jasonantman.com>. # ######################################################################################### # # LICENSE: AGPLv3 <http://www.gnu.org/licenses/agpl-3.0.html> # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU Affero General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU Affero General Public License for more details. # # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. # # If you make any modifications/fixes/feature additions, it would be greatly appreciated # if you send them back to me at the above email address. # ######################################################################################### # # CREDITS: # - F5 itself for the formula: <http://support.f5.com/kb/en-us/solutions/public/6000/900/sol6917.html> # - Tyler Krpata <http://www.tylerkrpata.com/2009/06/decode-f5-bigip-cookie-in-one-line-of.html> # for the Perl one-liner that this logic is based on. # # $HeadURL: http://svn.jasonantman.com/misc-scripts/bigipcookie.pl $ # $LastChangedRevision: 27 $ # # Changelog: # # 2012-02-02 Jason Antman <jason@jasonantman.com>: # - initial version # use strict; use warnings; if ( $#ARGV < 0 ) { print "USAGE: bigipcookie.pl <node IP:port | cookie value>\n"; exit 1; } if ($ARGV[0] =~ m/^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3}):(\d+)$/) { my $ipEnc = $1 + ($2*256) + ($3 * (256**2)) + ($4 * (256**3)); my $portEnc = hex(join "", reverse ((sprintf "%04x", $5) =~ /../g)); print "$ipEnc.$portEnc.0000\n"; } elsif ($ARGV[0] =~ m/^(\d+)\.(\d+)\.0000$/){ # decode a cookie value my $ipEnc = $1; my $portEnc = $2; my $ip = join ".", map {hex} reverse ((sprintf "%08x", split /\./, $ipEnc) =~ /../g); my $portDec = hex(join "", reverse ((sprintf "%04x", $portEnc) =~ /../g)); print "$ip:$portDec\n"; } else { print "USAGE: bigipcookie.pl <node IP:port | cookie value>\n"; exit 1; } |
An example of the usage:
jantman@palantir:pts/8:~/bin/misc-scripts > ./bigipcookie.pl 192.168.23.50:80 840411328.20480.0000 jantman@palantir:pts/8:~/bin/misc-scripts > ./bigipcookie.pl 840411328.20480.0000 192.168.23.50:80 |
On a side note for those of your who are security-conscious: yes, of course, this means that if you’re using BigIp with cookie persistence, it is disclosing the internal IP and port of your server to your end users.