Puppet facter facts for syslog daemon type and version, symantec netbackup

I have a few more custom facts that I’ve added to my puppet-facter-facts github repository:

  • syslog_bin, syslog_type, and syslog_version – tell the absolute path to the running syslog binary, its short name (basename), and its version as a string. Currently only know about /sbin/syslogd and /sbin/rsyslogd.
  • has_netbackup – tests for presence of the /usr/openv/netbackup/bin directory, created by installation of Symantec Netbackup. Useful for making generation of include/exclude files conditional on having NetBackup installed.

Hopefully some of these will be of use to someone else as well.

Petit for Log Analysis

I recently discovered the petit program for log analysis. It’s a simple tool to pull out useful information from syslog logs in a variety of ways. I’ve only used it a few times so far, mainly on logs from problems I’ve already solved but didn’t know the cause of at first. So far, it’s proven quite useful. Here are a few examples:

  • petit --wordcount /var/log/messages – displays ordered count of words appearing in the log. My first step, especially if “warning”, “error” or “fatal” shows up near the top…
  • petit --hash --fingerprint /var/log/messages – hashes the log, removes filters (such as numerics, datestamp), and displays count of matching lines. Absolutely wonderful for web error logs, as it removes client IP addresses, line numbers, etc.
  • petit --mgraph /var/log/messages – graph messages per minute for the first hour of the log (ASCII of course)
  • petit --hgraph /var/log/messages – same as above, but messages per hour for the first day
  • Petit will also read from stdin with the –Xgraph options, so you can cat logfile | grep word | petit --mgraph

Just one note – this tool appears to work only on standard syslog formatted logs. If some non-datestamped lines managed to work their way into the log (i.e. someone used echo >> logfile instead of logger), it will choke.

Many thanks to Scott McCarty for this wonderful tool!

php-suhosin syslog issues

I just installed php-suhosin 0.9.29 from EPEL on a CentOS 5.6 box. I’m running a whole bunch of name-based vhosts in Apache, and have a bunch of web apps, so I opted to run suhosin in simulation mode (don’t actually block anything, but log errors) and have it log via syslog to a single file. Unfortunately, when I configured this, the syslog messages started showing up in the wrong place, apparently with the wrong facility and priority. After some roundabout debugging (at first assuming syslogd to be the problem), I determined that, for whatever really strange reason (perhaps an incorrect syslog.h on the EPEL box that built the suhosin package?) the LOG_* constants were incorrect. I looked up the correct integer values in /usr/include/sys/syslog.h and the following configuration directives accomplished the task correctly:

suhosin.log.syslog.facility = 128
; 128 = LOG_LOCAL0
 
suhosin.log.syslog.priority = 5
; 5 = LOG_NOTICE

This one line puts suhosin into simulation mode, where it only logs errors instead of enforcing on them:

suhosin.simulation = On

Linux Memory Usage and Disk Caching

I recently added some Cacti-based graphing to a number of Linux-based servers prior to rolling out a new service. When I was looking over the performance graphs of the initial testing, I noticed that memory usage on our rsyslog server was near 98%. Looking at top(1), I saw numbers that agreed, though processor usage was around 99% idle, and no process appeared to be using more than 1% of memory. It took me a minute or two to open my eyes and see past the panic of memory usage, and finally look at the complete output from free(1):

             total       used       free     shared    buffers     cached
Mem:       8171508    8032632     138876          0     162084    7253716
-/+ buffers/cache:     616832    7554676
Swap:      4192956        152    4192804

The pertinent part is the last column: “cached”. It slipped my mind that while rsyslog is writing vast amounts of data to disk, which may or may not ever be read back, the kernel is using free memory to cache as much of that as it reliably can. Hence the difference between what the kernel and userland tools call “free”, and what most human beings (or at least sysadmins) would consider “free” – or, more correctly, “available for use”.

When I get a chance, maybe I’ll submit patches to the Cacti Memory Usage Percent (SNMP) template to either graph cache separately, or remove it from the total.

Interestingly, I also found a somewhat cute page entitled “Help! Linux ate my RAM!” at http://www.linuxatemyram.com/.

Vyatta VC5 – Snort alerts to syslog

I’m running a Vyatta vyatta router at home – in my opinion it’s pretty near “enterprise grade”, and I’m running the Community/Core (read: no-cost and almost all Free) on commodity hardware with great performance. Granted, I’m still on the older version (5 as opposed to the current 6) since an upgrade will require total downtime and a spare set of SCSI disks for the machine it’s on, but it still works quite well. Today I decided to enable the Snort IDS on the box. It actually worked quite well (albeit stuck at older rules until I upgrade to Vyatta VC6) and didn’t put much more load on the box. For the time being I decided to just have it alert on problems and not drop anything, as I’m getting pretty high false positives and the older Vyatta version doesn’t seem to have a way to disregard rules.

My biggest complaint was that Vyatta didn’t see fit to allow alerts by syslog. I’m not a big fan of keeping information like IDS logs stuck on the router – I don’t like logging in to it any more than I have to, it doesn’t have much storage, and I’d also like to keep stuff like this in a more secure location. Through a bit of digging, I found the /opt/vyatta/share/perl5/VyattaSnortConfig.pm Perl module which generates the Snort config file from Vyatta’s CLI stuff. Looking through the Perl code, I found the definition of the snort.conf output statements:

my $output_def =<<EOD;
  output alert_unified: alert, limit $log_limit
  output log_null
EOD

I simply added a line output alert_syslog: log_local3 log_notice after the output alert_unified line.

my $output_def =<<EOD;
  output alert_unified: alert, limit $log_limit
  output alert_syslog: log_local3 log_notice
  output log_null
EOD

I then went into Vyatta configuration and changed a rule from alert to pass, committed, changed back, committed, and /etc/snort/snort.conf now had my syslog lines. I’m now getting snort alerts to local3 in syslog, which is fed to my centralized log server. My next project is to find or write something which will parse these logs, generate a daily summary email, and maybe check them hourly and alert on any big changes. I also might just cron an emailing of the output from Vyatta’s show ips summary command. So far I have over 11,000 events logged in about 12 hours.

Nagios check scripts

Last week I added some of my Nagios check scripts to my nagios-scripts GitHub repository. Perhaps they’ll be of some use to some other people…

  • check_1wire_temps.php – quick and dirty, built for one specific application, but a good starting place for checking Dallas 1-wire temperatures via OWFS.
  • check_802dot11.php – A script to check various things in the IEEE-802DOT11 MIB, written for Ubiquiti APs (SNMP).
  • check_frogfoot.php – A script to check some stuff from FROGFOOT-MIB, also written for Ubiquiti APs (SNMP).
  • check_asterisk_iaxpeers – a Python check script to parse the output of rasterisk for IAX peer status and latency (includes perf data output).
  • check_bacula_job.php – A script to connect to the Bacula database and make sure a specified job terminated OK and was run on schedule.
  • check_docsis – A script to check status and various metrics for cable modems implementing the DOCSIS MIB (SNMP). Works with (at least) the Motorola SurfBoard modems used by Cablevision (which use 192.168.100.1 on the LAN side).
  • check_syslog_age.php – A PHP script which checks (recursively) that the newest file under a directory is no more than X seconds old. I use this for checking my centralized syslog server, which has logs separated out in /var/log/HOSTS/hostname.

Update 2011-01-31 – the check_syslog_age.php script was updated today to handle an error condition where stat() calls in PHP fail on files larger than 2GB on 32-bit systems.

Project Announcement – PHPsa

So, here’s the “official” scoop on the new project that I’m planning/starting to work on. I’m calling it PHPsa for now, and it’s going to (hopefully) be an integrated dashboard/portal for SysAdmins. While there are a number of tools that fit into this general category (perhaps with OSSIM being the closest, though it’s security-minded), I feel that there’s a real gap in terms of tool integration. My daily workflow, which includes multiple trips to and correlation among Nagios, Cacti, DNS, DHCP, Puppet, logs, and other tools really leaves something to be desired. So, I’m setting out to create a modular SysAdmin dashboard that unifies many of the common SysAdmin-related tools into a modular dashboard.

The first overall design goals that I’ve set are:

  1. A modular, plugin-based architecture that allows admins to select which features/tools they want, and allows easy development of new modules.
  2. Design with legacy tools in mind – easy ways to tie in to tools that weren’t written with PHPsa in mind, both in terms of linking to information and gathering/unifying information.
  3. RBAC, including per-module rules and the possibility for a limited read-only view (client/user mode).
  4. Use of data sources, specifically web-based/REST APIs where available, and databases otherwise, from existing tools with as little modification as possible.
  5. Support for database abstraction, though I’ll be using MySQL.
  6. Eventually, implement RSS feeds of pertinent information.
  7. Balance Ajax/DHTML with the desire for important things to have canonical, static, bookmark-able URLs.

So, here are some of the things that I’m planning on integrating, with obvious bias towards getting my own projects done before I integrate pre-existing tools:

  • MultiBindAdmin, my DNS and DHCP administration tool (specifically geared towards split-view DNS with the inside view behind NAT).
  • RackMan, my tool for mapping devices’ physical locations in racks (and tacking patching).
  • My simple config tool for Puppet.
  • Nagios.
  • Cacti.
  • Nathan Hubbard’s MachDB.
  • Bacula (monitoring/status only).
  • Syslog via rsyslog (or any other syslog-to-SQL solution).
  • Possibly a front-end to Google Analytics.
  • Some of my custom scripts for graphing SpamAssassin, DNS queries, etc.
  • Some sort of Apache log analysis, like Webalizer.
  • Mail log analysis, possibly AWstats.

So, the first big issues that I’m going to tackle:

  1. General layout. Specifically, how to handle a more-or-less consistent layout while integrating tools that weren’t designed for PHPsa. I’ll probably end up using iFrames (or even a frameset) for tools that don’t integrate well.
  2. How to correlate data/objects between different tools (i.e. how to display information from Nagios, Cacti, MultiBindAdmin and MachDB for a given host?).
  3. Do I want to use a templating engine like Smarty or hand-code all of the HTML?
  4. How will I handle plugins?
  5. How much code do I want to re-write and how much can I use as-is from other tools? And, on a related note, how much existing data can I access easily from other tools, vs having to use grabber scripts that dump data in MySQL?

Update 2010-02-03: I think this may become a semi-official project for me at $work, which means that I’ll be able to dedicate quite a bit more time to it. Unfortunately, it also means that I will, most likely, have to give up Nathan Hubbard’s MachDB in favor of OCS Inventory NG, a more mature project that already includes inventory support for Linux, Windows and Mac.

rsyslog on CentOS5

Update July 2011 – We’ve been using rsyslog for our centralized syslog infrastructure at work (site being redesigned at the moment) for about a year now. As a result, our colleagues at Rutgers University Open System Solutions have been nice enough to include rsyslog in their koji build system. Updated packages of rsyslog for CentOS 5 x86 and x86_64 are available here. Please be aware that they have some dependencies of specific RU versions. It’s probably best if you download the source RPM and build it yourself using rpmbuild. The current 5.6.2 source package includes a CentOS 5 spec file and other related scripts. This RPM repository is only open to the public as a courtesy, so please download once at most and distribute it to your servers yourself.

Having finally setup my storage server (I know it’s not much, but for me starting with 1TB is wonderful), I actually got around to redoing my centralized logging infrastructure. Here’s a small summary of what I have to handle:

  • logs from 15 hosts at 2 locations, including Cisco devices and a mix of syslog and syslog-ng Linux boxen.
  • Remote location logs forwarded to one server at remote location, then to centralized log server via SSH port forwarding.
  • 48-hour retention of full iptables border firewall logs (~ 3GB/day).
  • Future plans to have all logs stored in MySQL.

I’d previously had most of the boxes logging to an older host with low disk space, but had to discontinue this due to lack of storage. Having assessed the options, and with definite plans to log to a database, I decided to go with rsyslog for the centralized host.

Unfortunately, stable rsyslog is up to 4.4.x, with 5.x in development, and the newest package I could find for CentOS was 2.something. So, I set about building it from source. It was a *very* difficult build on my machine (CentOS 5.3, 2.6.18-128.el5 #1 SMP i686). Unfortunately, I don’t have an RPM build environment setup, but here’s how I accomplished it:

  1. I f not already done, yum install rpmforge-release
  2. yum install gnutls-devel gnutls libatomic_ops-devel gcc43 java-1.6.0-openjdk-devel
  3. Download rsyslog-4.5.2 (currently Beta). Extract, cd into the directory.
  4. To test the performance difference from MySQL, yum install php-pgsql postgresql postgresql-devel postgresql-libs postgresql-server
  5. Edit the “configure” script, add -DHAVE_ATOMIC_BUILTINS to the DEFS line (28936 in this version).
  6. export CC=/usr/bin/gcc43
  7. export CCDEPMODE=”depmode=gcc4″
  8. export CFLAGS=”-O3 -march=i686″
  9. ./configure –enable-mysql –enable-omtemplate –enable-gnutls –enable-pgsql
  10. make && make install

For my system, I used the /etc/sysconfig/rsyslog and /etc/init.d/rsyslog from the 2.x rsyslog RPMs, with some modifications as follows:

/etc/sysconfig/rsyslog (comments have been removed to save space):

# -c4   version 4 compatibility mode
# -x     disable DNS for remote messages (don't want it to hang if DNS is down
# -4     IPv4 only
SYSLOGD_OPTIONS="-c4 -x -4"

/etc/init.d/rsyslog (comments have been removed to save space):

#!/bin/bash
#
# rsyslog        Starts rsyslogd.
#
#
# chkconfig: - 12 88
# description: Syslog is the facility by which many daemons use to log \
# messages to various system log files.  It is a good idea to always \
# run rsyslog.
### BEGIN INIT INFO
# Provides: $syslog
# Required-Start: $local_fs $network $remote_fs
# Required-Stop: $local_fs $network $remote_fs
# Default-Stop: 0 1 2 3 4 5 6
# Short-Description: Enhanced system logging and kernel message trapping daemons
# Description: Rsyslog is an enhanced multi-threaded syslogd supporting,
#              among others, MySQL, syslog/tcp, RFC 3195, permitted
#              sender lists, filtering on any message part, and fine
#              grain output format control.
### END INIT INFO
 
# Source function library.
. /etc/init.d/functions
 
RETVAL=0
 
start() {
        [ -x /sbin/rsyslogd ] || exit 5
 
        # Do not start rsyslog when sysklogd is running
        if [ -e /var/run/syslogd.pid ] ; then
                echo $"Shut down sysklogd before you run rsyslog";
                exit 1;
        fi
 
        # Source config
        if [ -f /etc/sysconfig/rsyslog ] ; then
                . /etc/sysconfig/rsyslog
        else
                SYSLOGD_OPTIONS="-m 0"
        fi
 
        if [ -z "$SYSLOG_UMASK" ] ; then
              SYSLOG_UMASK=077;
        fi
        umask $SYSLOG_UMASK
 
        echo -n $"Starting system logger: "
        daemon rsyslogd $SYSLOGD_OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch /var/lock/subsys/rsyslog
        return $RETVAL
}
stop() {
        echo -n $"Shutting down system logger: "
        killproc rsyslogd
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/rsyslog
        return $RETVAL
}
reload()  {
    RETVAL=1
    syslog=`cat /var/run/rsyslogd.pid 2>/dev/null`
    echo -n "Reloading system logger..."
    if [ -n "${syslog}" ] && [ -e /proc/"${syslog}" ]; then
        kill -HUP "$syslog";
        RETVAL=$?
    fi
    if [ $RETVAL -ne 0 ]; then
        failure
    else
        success
    fi
    echo
    return $RETVAL
}
rhstatus() {
        status rsyslogd
}
restart() {
        stop
        start
}
 
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        restart
        ;;
  reload|force-reload)
        reload
        ;;
  status)
        rhstatus
        ;;
  condrestart)
        [ -f /var/lock/subsys/rsyslog ] && restart || :
        ;;
  *)
        echo $"Usage: $0 {start|stop|restart|reload|force-reload|condrestart}"
        exit 2
esac
 
exit $?

My rsyslog.conf file:

# uncomment next line for debugging, use graphvis to see the graph
#$GenerateConfigGraph /root/rsyslog-graph.dot
$ModLoad imklog
$ModLoad imtcp
$ModLoad imudp
$ModLoad imuxsock
 
# template
$template RemoteHost,"/var/log/HOSTS/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%syslogfacility-text%.log"
# used for Cisco, vanilla syslog when we can't parse host name
$template RemoteFromHost,"/var/log/HOSTS/%FROMHOST%/%$YEAR%/%$MONTH%/%$DAY%/%syslogfacility-text%.log"
 
# NOTE - we can't bind UDP to a ruleset, so it enters the local RuleSet
#   and has to be dealt with here
 
$RuleSet local
 
# for cisco, vyatta - doesn't send hostname, need to use IP manually
:fromhost, isequal, "192.168.0.99" ?RemoteFromHost
:fromhost, isequal, "192.168.0.99" ~
:fromhost, isequal, "192.168.0.103" ?RemoteFromHost
:fromhost, isequal, "192.168.0.103" ~
:fromhost, isequal, "192.168.0.97" ?RemoteFromHost
:fromhost, isequal, "192.168.0.97" ~
:fromhost, isequal, "192.168.0.111" ?RemoteFromHost
:fromhost, isequal, "192.168.0.111" ~
# anything from a remote host gets logged as such
:source, isequal, "" ?RemoteHost
:source, isequal, "" ~
 
#
# LOCAL LOGGING
#
 
kern.*                                                 /var/log/messages
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 *
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
 
# use the local RuleSet as default
$DefaultRuleset local
 
#
# BEGIN centralized logging stuff added 2009-09-16 by jantman
#
 
# define ruleset for remote logging
$RuleSet remote
 
*.* ?RemoteHost
 
# bind ruleset to tcp listener
$InputTCPServerBindRuleset remote
# and activate it:
$InputTCPServerRun 5000
 
$UDPServerRun 514
$UDPServerRun 5000
 
#
# END remote logging
#