Jason Antman’s Blog

So, here’s the “official” scoop on the new project that I’m planning/starting to work on. I’m calling it PHPsa for now, and it’s going to (hopefully) be an integrated dashboard/portal for SysAdmins. While there are a number of tools that fit into this general category (perhaps with being the closest, though it’s security-minded), I feel that there’s a real gap in terms of tool integration. My daily workflow, which includes multiple trips to and correlation among Nagios, Cacti, DNS, DHCP, Puppet, logs, and other tools really leaves something to be desired. So, I’m setting out to create a modular SysAdmin dashboard that unifies many of the common SysAdmin-related tools into a modular dashboard.

The first overall design goals that I’ve set are:

1. A modular, plugin-based architecture that allows admins to select which features/tools they want, and allows easy development of new modules.
2. Design with legacy tools in mind – easy ways to tie in to tools that weren’t written with PHPsa in mind, both in terms of linking to information and gathering/unifying information.
3. RBAC, including per-module rules and the possibility for a limited read-only view (client/user mode).
4. Use of data sources, specifically databases, from existing tools with as little modification as possible.
5. Support for database abstraction, though I’ll be using MySQL.
6. Eventually, implement RSS feeds of pertinent information.
7. Balance Ajax/DHTML with the desire for important things to have canonical, static, bookmark-able URLs.

So, here are some of the things that I’m planning on integrating, with obvious bias towards getting my own projects done before I integrate pre-existing tools:

• MultiBindAdmin, my DNS and DHCP administration tool (specifically geared towards split-view DNS with the inside view behind NAT).
• RackMan, my tool for mapping devices’ physical locations in racks (and tacking patching).
• My simple config tool for Puppet.
• Nagios.
• Cacti.
• Nathan Hubbard’s MachDB.
• Bacula (monitoring/status only).
• Syslog via rsyslog (or any other syslog-to-SQL solution).
• Possibly a front-end to Google Analytics.
• Some of my custom scripts for graphing SpamAssassin, DNS queries, etc.
• Some sort of Apache log analysis, like Webalizer.
• Mail log analysis, possibly AWstats.

So, the first big issues that I’m going to tackle:

1. General layout. Specifically, how to handle a more-or-less consistent layout while integrating tools that weren’t designed for PHPsa. I’ll probably end up using iFrames (or even a frameset) for tools that don’t integrate well.
2. How to correlate data/objects between different tools (i.e. how to display information from Nagios, Cacti, MultiBindAdmin and MachDB for a given host?).
3. Do I want to use a templating engine like Smarty or hand-code all of the HTML?
4. How will I handle plugins?
5. How much code do I want to re-write and how much can I use as-is from other tools? And, on a related note, how much existing data can I access easily from other tools, vs having to use grabber scripts that dump data in MySQL?

Update 2010-02-03: I think this may become a semi-official project for me at $work, which means that I’ll be able to dedicate quite a bit more time to it. Unfortunately, it also means that I will, most likely, have to give up Nathan Hubbard’s MachDB in favor of OCS Inventory NG, a more mature project that already includes inventory support for Linux, Windows and Mac.

PHPsa, Projects bind, cacti, dns, Nagios, PHPsa, Projects, puppet, rsyslog, sysadmin, syslog

Well, I wanted to call this a $#!^storm, but I don’t think it’s grown to those proportions yet. But any new platform will have its hiccups, and Google is relatively new to the OS world.

So, here’s the news. Google issued a legal Cease and Desist order to a developer Steve Kondik (known as Cyanogen). The story goes something like this… Steve is an active Android developer, doing a lot of work on the lower-level stuff (lower level than apps), including multi-touch and more home screens. His changes are essentially at the OS-level, and Android doesn’t have a full-fledged package management/patching mechanism like Linux distros, so making use of them requires recompiling stuff and re-flashing the device with a new ROM image (since the installation apps can’t handle stuff this low-level). Here’s the rub: in order to work on a device, the ROM image needs to include both closed-source Google apps and proprietary (device manufacturer) drivers. While these apps and drivers are available for download, the license terms prohibit redistribution. But in order for Steve to create a fully-functional ROM image, he has to include the closed code.

There are some writeups on this at Linux Magazine and a good, timely analysis at Linux Insider.

There’s also a clarification by Google’s Dan Morrill on the Android Developers blog.

So, what’s my take on all this (not that another guy taking about this is needed)?

Firstly, I think this is relatively minor. The community will work around it, whether with Google’s blessing or not. The bigger issue that’s coming to light is the fact that Google isn’t simply altruistic, they’re a for-profit entity. They have every right to be, and they have every right to exercise some amount of control over Android. The community needs to realize that Android isn’t a silver bullet, and isn’t even the Linux of the phone world. On the other hand, Google needs to realize two important facts: 1) the openness of Android is what’s driving developers to it, and they need to do all they can to continue that, and 2) most of those developers are flat-out used to running Linux with an all-GPL system, and aren’t used to the concept of not being able to roll their own distribution.

So what’s my advice?

1. Google should further decouple their Apps from the Android platform. Specifically, instead of requiring users to back things up, they should provide a redistributable application that installs their other apps. Allow a user to flash a bare-bones community ROM image, and then pull whatever else they want from Google. If Google intends on toeing the line that Android is Free but the (Google) Apps aren’t, then Google should provide an acceptable means for users of community ROM images to easily and painlessly re-install the closed Google apps on that image.
2. Google should require that handset manufacturers do the same. Create a redistributable application that can be part of community ROM images, which will (via tethering or whatever) download and install any proprietary device-specific drivers that are needed.

Bottom line of my opinion – it’s fine if Google exercises their full control over their own closed apps. But they should provide an avenue for non-technical end-users to easily upgrade a community (i.e. Free) ROM image with the expected Google Apps and device manufacturer software.

Tech News android, google

Having finally setup my storage server (I know it’s not much, but for me starting with 1TB is wonderful), I actually got around to redoing my centralized logging infrastructure. Here’s a small summary of what I have to handle:

• logs from 15 hosts at 2 locations, including Cisco devices and a mix of syslog and syslog-ng Linux boxen.
• Remote location logs forwarded to one server at remote location, then to centralized log server via SSH port forwarding.
• 48-hour retention of full iptables border firewall logs (~ 3GB/day).
• Future plans to have all logs stored in MySQL.

I’d previously had most of the boxes logging to an older host with low disk space, but had to discontinue this due to lack of storage. Having assessed the options, and with definite plans to log to a database, I decided to go with rsyslog for the centralized host.

Unfortunately, stable rsyslog is up to 4.4.x, with 5.x in development, and the newest package I could find for CentOS was 2.something. So, I set about building it from source. It was a *very* difficult build on my machine (CentOS 5.3, 2.6.18-128.el5 #1 SMP i686). Unfortunately, I don’t have an RPM build environment setup, but here’s how I accomplished it:

1. I f not already done, yum install rpmforge-release
2. yum install gnutls-devel gnutls libatomic_ops-devel gcc43 java-1.6.0-openjdk-devel
3. Download rsyslog-4.5.2 (currently Beta). Extract, cd into the directory.
4. To test the performance difference from MySQL, yum install php-pgsql postgresql postgresql-devel postgresql-libs postgresql-server
5. Edit the “configure” script, add -DHAVE_ATOMIC_BUILTINS to the DEFS line (28936 in this version).
6. export CC=/usr/bin/gcc43
7. export CCDEPMODE=”depmode=gcc4″
8. export CFLAGS=”-O3 -march=i686″
9. ./configure –enable-mysql –enable-omtemplate –enable-gnutls –enable-pgsql
10. make && make install

For my system, I used the /etc/sysconfig/rsyslog and /etc/init.d/rsyslog from the 2.x rsyslog RPMs, with some modifications as follows:

/etc/sysconfig/rsyslog (comments have been removed to save space):

# -c4   version 4 compatibility mode
# -x     disable DNS for remote messages (don't want it to hang if DNS is down
# -4     IPv4 only
SYSLOGD_OPTIONS="-c4 -x -4"

/etc/init.d/rsyslog (comments have been removed to save space):

#!/bin/bash
#
# rsyslog        Starts rsyslogd.
#
#
# chkconfig: - 12 88
# description: Syslog is the facility by which many daemons use to log \
# messages to various system log files.  It is a good idea to always \
# run rsyslog.
### BEGIN INIT INFO
# Provides: $syslog
# Required-Start: $local_fs $network $remote_fs
# Required-Stop: $local_fs $network $remote_fs
# Default-Stop: 0 1 2 3 4 5 6
# Short-Description: Enhanced system logging and kernel message trapping daemons
# Description: Rsyslog is an enhanced multi-threaded syslogd supporting,
#              among others, MySQL, syslog/tcp, RFC 3195, permitted
#              sender lists, filtering on any message part, and fine
#              grain output format control.
### END INIT INFO
 
# Source function library.
. /etc/init.d/functions
 
RETVAL=0
 
start() {
        [ -x /sbin/rsyslogd ] || exit 5
 
        # Do not start rsyslog when sysklogd is running
        if [ -e /var/run/syslogd.pid ] ; then
                echo $"Shut down sysklogd before you run rsyslog";
                exit 1;
        fi
 
        # Source config
        if [ -f /etc/sysconfig/rsyslog ] ; then
                . /etc/sysconfig/rsyslog
        else
                SYSLOGD_OPTIONS="-m 0"
        fi
 
        if [ -z "$SYSLOG_UMASK" ] ; then
              SYSLOG_UMASK=077;
        fi
        umask $SYSLOG_UMASK
 
        echo -n $"Starting system logger: "
        daemon rsyslogd $SYSLOGD_OPTIONS
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch /var/lock/subsys/rsyslog
        return $RETVAL
}
stop() {
        echo -n $"Shutting down system logger: "
        killproc rsyslogd
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/rsyslog
        return $RETVAL
}
reload()  {
    RETVAL=1
    syslog=`cat /var/run/rsyslogd.pid 2>/dev/null`
    echo -n "Reloading system logger..."
    if [ -n "${syslog}" ] && [ -e /proc/"${syslog}" ]; then
        kill -HUP "$syslog";
        RETVAL=$?
    fi
    if [ $RETVAL -ne 0 ]; then
        failure
    else
        success
    fi
    echo
    return $RETVAL
}
rhstatus() {
        status rsyslogd
}
restart() {
        stop
        start
}
 
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        restart
        ;;
  reload|force-reload)
        reload
        ;;
  status)
        rhstatus
        ;;
  condrestart)
        [ -f /var/lock/subsys/rsyslog ] && restart || :
        ;;
  *)
        echo $"Usage: $0 {start|stop|restart|reload|force-reload|condrestart}"
        exit 2
esac
 
exit $?

My rsyslog.conf file:

# uncomment next line for debugging, use graphvis to see the graph
#$GenerateConfigGraph /root/rsyslog-graph.dot
$ModLoad imklog
$ModLoad imtcp
$ModLoad imudp
$ModLoad imuxsock
 
# template
$template RemoteHost,"/var/log/HOSTS/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%syslogfacility-text%.log"
# used for Cisco, vanilla syslog when we can't parse host name
$template RemoteFromHost,"/var/log/HOSTS/%FROMHOST%/%$YEAR%/%$MONTH%/%$DAY%/%syslogfacility-text%.log"
 
# NOTE - we can't bind UDP to a ruleset, so it enters the local RuleSet
#   and has to be dealt with here
 
$RuleSet local
 
# for cisco, vyatta - doesn't send hostname, need to use IP manually
:fromhost, isequal, "192.168.0.99" ?RemoteFromHost
:fromhost, isequal, "192.168.0.99" ~
:fromhost, isequal, "192.168.0.103" ?RemoteFromHost
:fromhost, isequal, "192.168.0.103" ~
:fromhost, isequal, "192.168.0.97" ?RemoteFromHost
:fromhost, isequal, "192.168.0.97" ~
:fromhost, isequal, "192.168.0.111" ?RemoteFromHost
:fromhost, isequal, "192.168.0.111" ~
# anything from a remote host gets logged as such
:source, isequal, "" ?RemoteHost
:source, isequal, "" ~
 
#
# LOCAL LOGGING
#
 
kern.*                                                 /var/log/messages
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 *
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log
 
# use the local RuleSet as default
$DefaultRuleset local
 
#
# BEGIN centralized logging stuff added 2009-09-16 by jantman
#
 
# define ruleset for remote logging
$RuleSet remote
 
*.* ?RemoteHost
 
# bind ruleset to tcp listener
$InputTCPServerBindRuleset remote
# and activate it:
$InputTCPServerRun 5000
 
$UDPServerRun 514
$UDPServerRun 5000
 
#
# END remote logging
#

Tech HowTos centos, logging, rsyslog, syslog

Yesterday I finally began moving DNS for my sites from GoDaddy to my own in-house system of master/slave BIND9. While both DNS servers are currently at the same location and on the same WAN connection (heck, they’re beind the same router, too), so is all of the rest of my infrastructure. Migrating jasonantman.com was definitely the most critical task, this has allowed me to easily use my new project, MultiBIND Admin to manage DNS. In addition to just being simpler than using GoDaddy’s tool, it allows me to manage DNS for both the external view and the NATed internal view in one tool. I did have a brief mail outage thanks to some incorrect MX records being served by the slave, and a few other issues with the caching DNS servers at work not expiring the old records, but all seems to be well now. It was a relatively smooth transition, though I haven’t yet moved over some of my older less used domains.

The next part of my project, when I move the ambulance corps hosted services in-house, will be trying to find a decently-priced DNS hosting company that will just act as a slave, to keep DNS up if my WAN connection goes down.

Projects bind, dns, multibindadmin

I know I haven’t been posting a lot, but here’s an update on some of my projects:

• PHP EMS Tools – I’ve done quite a bit of work for the ambulance corps, and intend on rolling this into the main distribution. I’ve also added an Asterisk/AGI module to handle crew call-ins. It’s going to be a long road, as I have to manually diff the ambulance corps version to the trunk version and merge the changes (leaving out anything specific to our organization), but I plan on doing it. The next version will also include historical tracking of roster information (member information, status, positions, committees, etc.) and LDAP integration for authentication.
• PHPsa – My new project, tentatively called PHPsa, is an integrated dashboard for sysadmins. The idea is to develop a plugin-based portal for SA tools. Currently, I will be including some of my own projects – MultiBindAdmin (a tool to administer BIND and DHCPd, specifically geared towards split-view DNS with the inside behind NAT) and RackMan (a tool to track and visualize the location of devices within racks, including ability to temporarily move devices around) – as well as my updates to Nathan Hubbard’s MachDB.

I’ve also done quite a bit of customization of the current version of Nathan Hubbard’s MachDB. My local version is in subversion. It adds detailed network interface information, information on expansion slots, and some extra details for the system and storage. I plan on developing a patch and contacting Nathan once I get a chance. It also includes a Python collector script that I developed.

PHPsa, Projects machdb, php ems tools, PHPsa, Projects, rackman

While I’ve read and really appreciate Tom Limoncelli’s Time Management for System Administrators, the current state of my life (mainly that it’s split between work, personal projects, a freelance client, administering the systems of a the ambulance corps and real people, and that my “work day” is whenever I’m awake) has prevented me from really implementing most of the advice. However, I do try to be as productive as I can.

Without getting into details, a few weeks ago, $WORK suffered a major electrical failure that required everything in the data center to be powered down. This happened around 10:30 AM, and the majority of groups simply powered down their machines and left, planning to return around 2 AM (the estimated power restoration time). After getting our machines down and stopping for pizza, I remembered how much of a pain it was to work in the racks bringing everything down. While my group only has two racks, we’ve had a lot of changeover lately, and the cabling had gotten quote messy. Noting this, I mentioned it to my two higher-ups, remembering that we had a stock of assorted length patch cables. We were able to make an “emergency” run to our cable vendor and pick up a box of 1- 2- and 3-foot power cables.

While everyone else was home or in their offices dodging the pieces of falling sky (everything was down including VoIP and mail), we were the only group getting real productive work done in the data center. The power failure, rather than a catastrophic event, was a great opportunity – the only time we could pull every cable in a production rack and re-do all power and patches.

So, here’s my SA tip for the day – everyone has some big projects that they’d like to do, require downtime, but aren’t critical enough to schedule something. So, keep a list of these and have the parts on hand. Whether it’s a “just in case” hardware swap-out, re-patching, or anything else, eventually you (depending on the environment that you work in) might have one of those times when the solution to the problem is out of your hands and there’s nothing else to do. Use this time productively.

Miscellaneous Geek Stuff sysadmin, time management