As a side note, this was all done pretty quickly (moving the database and tarball for the vhost, installing the plugin, changing DNS), so please give me a heads-up if you experience any problems.

The numbers are rather impressive:

All tests were performed on my workstation, a Dell Precision 470, two dual-core Xeons at 2.8 GHz, 2GB RAM, 16GB swap, OpenSuSE 11.1 64-bit. This was on the same LAN and subnet as the servers, with the workstation connected via a 1Gbps copper Ethernet link and the web-serving interfaces of the servers connected via 100Mbps (There’s a trunk in between, from the gigabit aggregation switch to the 100Mbps distribution switch).

A while ago (back in late 2008), I wrote a PHP script that parses the Nagios status.dat file into an associative array. My original use was to output XML which was then read by another script on another server and used for a small custom GUI. It’s a very simple PHP script that just takes the path of the status.dat file (which, obviously, must be readable by the user running the script).

At that time, I was using Nagios v2. Since then, I’ve moved to Nagios v3, and have updated the script to include the ability to parse v3 status.dat files, as well as a function to detect the version of a status file. I also refactored the code so that the parsing functions are all contained in a single file (statusXML.php.inc) which is safe to include in other scripts. The actual statusXML.php file now just includes examples of how to call all of the functions and output XML (though it is equally useful to output the serialized array, or use it directly).

Since I posted my script online, two people have been kind enough to send back their modifications:

• Artur Krzywański modified the original (r4) version of statusXML.php to allow selection of the keys to be returned.
• Whitham D. Reeve II of General Communication, Inc., who needed higher performance for a very large status file, rewrote my script in C as a PHP module.

Both of these generous contributions have been included in my Subversion repository as of the current revision, 5. Unfortunately, due to my delay in putting my Nagios3 code into svn, both of these contributions are Nagios v2 only.

As time permits, I plan on merging Artur’s changes into the current version of statusXML.php.inc. Unfortunately, C isn’t one of my strong points, but I plan on also updating Whitham’s PHP module code to work with Nagios3 as soon as possible.

Stay tuned for updates, and thanks to both gentlemen for contributing their work. I’m always interested in hearing how people are using my code, and how they are making it better.

Also: While I added this project to Nagios Exchange, and plan on adding it to Monitoring Exchange, I don’t always keep those sites up to date (I can’t access Nagios Exchange right now, and who knows if I’ll have time to update it tomorrow). I strongly recommend directly checking out from Subversion at http://svn.jasonantman.com/nagios-xml/ or taking a look at the code through ViewVC at http://viewvc.jasonantman.com/cgi-bin/viewvc.cgi/nagios-xml/.

• MultiBindAdmin, my DNS and DHCP administration tool (specifically geared towards split-view DNS with the inside view behind NAT).
• RackMan, my tool for mapping devices’ physical locations in racks (and tacking patching).
• My simple config tool for Puppet.
• Nagios.
• Cacti.
• Nathan Hubbard’s MachDB.
• Bacula (monitoring/status only).
• Syslog via rsyslog (or any other syslog-to-SQL solution).
• Possibly a front-end to Google Analytics.
• Some of my custom scripts for graphing SpamAssassin, DNS queries, etc.
• Some sort of Apache log analysis, like Webalizer.
• Mail log analysis, possibly AWstats.

So, the first big issues that I’m going to tackle:

1. General layout. Specifically, how to handle a more-or-less consistent layout while integrating tools that weren’t designed for PHPsa. I’ll probably end up using iFrames (or even a frameset) for tools that don’t integrate well.
2. How to correlate data/objects between different tools (i.e. how to display information from Nagios, Cacti, MultiBindAdmin and MachDB for a given host?).
3. Do I want to use a templating engine like Smarty or hand-code all of the HTML?
4. How will I handle plugins?
5. How much code do I want to re-write and how much can I use as-is from other tools? And, on a related note, how much existing data can I access easily from other tools, vs having to use grabber scripts that dump data in MySQL?

Update 2010-02-03: I think this may become a semi-official project for me at $work, which means that I’ll be able to dedicate quite a bit more time to it. Unfortunately, it also means that I will, most likely, have to give up Nathan Hubbard’s MachDB in favor of OCS Inventory NG, a more mature project that already includes inventory support for Linux, Windows and Mac.

The next part of my project, when I move the ambulance corps hosted services in-house, will be trying to find a decently-priced DNS hosting company that will just act as a slave, to keep DNS up if my WAN connection goes down.

• PHP EMS Tools – I’ve done quite a bit of work for the ambulance corps, and intend on rolling this into the main distribution. I’ve also added an Asterisk/AGI module to handle crew call-ins. It’s going to be a long road, as I have to manually diff the ambulance corps version to the trunk version and merge the changes (leaving out anything specific to our organization), but I plan on doing it. The next version will also include historical tracking of roster information (member information, status, positions, committees, etc.) and LDAP integration for authentication.
• PHPsa – My new project, tentatively called PHPsa, is an integrated dashboard for sysadmins. The idea is to develop a plugin-based portal for SA tools. Currently, I will be including some of my own projects – MultiBindAdmin (a tool to administer BIND and DHCPd, specifically geared towards split-view DNS with the inside behind NAT) and RackMan (a tool to track and visualize the location of devices within racks, including ability to temporarily move devices around) – as well as my updates to Nathan Hubbard’s MachDB.

I’ve also done quite a bit of customization of the current version of Nathan Hubbard’s MachDB. My local version is in subversion. It adds detailed network interface information, information on expansion slots, and some extra details for the system and storage. I plan on developing a patch and contacting Nathan once I get a chance. It also includes a Python collector script that I developed.

I’ve begun to move DNS for all of my domains in-house, mostly because since everything is behind NAT, it’s a real pain to manage DNS entries in two places (one of them being GoDaddy’s web interface). Because of the NAT issue, I’m also writing my own BIND configuration tool, currently named MultiBIND Admin. In addition to managing multiple zones in a sane way, it stores all configuration in MySQL. Among other things, it can store different IP addresses for A records for the inside and outside views. Zone files can either be pulled by a script on the name server (push capability is being worked on) or downloaded (for uploading to a DNS hosting provider like GoDaddy).

For my final project for my XML web design class, I’m going to be making some “mashup” with RackMan, Google Maps, Google Visualizer, Nagios, and a few other tools…

Stay tuned…

I’m running into some problems with the old management card for the Tripp Lite UPS, and I have a few other issues to sort out, but here’s a photo that I took this weekend after the upgrades (yes, it’s a bit blurry – that happens handheld at 1/10 sec).

ACHTUNG! ALLES LOOKENSPEEPERS!
Alles touristen und non-technischen looken peepers! Das computermachine ist nicht fuer gefingerpoken und mittengrabben.
Ist easy schnappen der springenwerk, blowenfusen und poppencorken mit spitzensparken. Ist nicht fuer gewerken bei das dumpkopfen. Das rubbernecken sichtseeren keepen das cotten-pickenen hans in das pockets muss; relaxen und watchen das blinkenlichten.

(For those of you who aren’t familiar with it, blinkenlights).

So, being the closest thing to a SysAdmin we have (my official title is still Student Systems Programmer), it’s my job to build a new installation, configuration and backup infrastructure. We’ve already standardized on CentOS as a University-wide distro, and have a local full mirror, so I don’t need to choose a distro. I do, however, have to plan the installation and backup architecture. The main requirements are:

1. Lowest overall time for bare-metal recovery to a working system.
2. Ease of use, as people other than myself will need to administer it (so they should be able to do so from a cheat sheet in the wiki).
3. Repeatability – it should be easy and intuitive to make an almost-exact-copy of a machine.

I started a thread a few days ago on the SAGE mailing list, which you can find here.

At the moment, it looks like the general idea that I’m going with is to use Kickstart to install the systems, using a basic and minimal Kickstart file. Basic package selection (minimalist) with just what’s needed to configure the system with a hostname and network settings for the management VLAN. I’ll then have Kickstart install and configure a configuration management package – I’m leaning towards Puppet over Cfengine and am starting testing. The config management software will handle all of the customization for the system (everything different from the base generic Kickstart install) so it’s all kept under the control of config management from step 1.

The final part is a backup system, mainly for whatever eventually – whether out of human error or simple laziness – ends up out of the config management system’s control. Our previous SA had settled on Zmanda, the paid version of Amanda, which comes with specific plugins for MySQL and MSSQL. I’m also looking at Bacula, mainly because of its’ advanced features, scheduling (especially the new scheduling in Bacula 3) and scalability.

The beauty that I see in having Kickstart do something minimal and then letting Puppet handle the rest is that (especially since we’ve standardized on SunFire X4100’s with identical configurations) I can kickstart and rack up a few spare machines, and to get them up and running all I need to do is power them up (iLOM) and tell Puppet what to make them.

I’m currently starting testing of both Puppet itself and getting Kickstart to start the puppet install and daemon (instructions from David Lutterkort’s blog (Red Hat software engineer)). We’ll see how everything goes…

Yesterday, I also had Cablevision show up to install the new Optimum Business with 5 static IPs.

So, last night around 9:00, I started the arduous task of (for the first time ever) powering down ALL of my machines, moving them to the new rack, and re-cabling. That took about 2-1/2 hours, after which my intent was to bring up the new Optimum connection, configure the Vyatta router, and roll over mail and web. From what I’d read of the Vyatta docs it seemed a relatively straightforward task, and being the stubborn jackass that I am, I decied, “hey, it’s my personal site, it’s low traffic, and I want it up before I go to sleep. I’ll roll over DNS before I bring everything up.”

That was a very bad idea. Vyatta isn’t nearly as simple as it seems – especially for someone who isn’t really a network (or at least router/firewall) guy. When they say Enterprise, they mean robust. They also mean that week-long bootcamps aren’t for naught. It took me about half an hour to figure out that even if no “firewall” ruleset is associated with an interface, it still has an implicit drop all. And if you only want to firewall what’s coming in from the outside world, and let everything out, you need to add explicit allow all rules to the in and out sides of the LAN inteface and the out side of the WAN interface.

To top all this off, I had some serious still-unexplained DHCP problems on the LAN, a serious issue since I just set all my hosts to DHCP (which I’ll probably undo soon). So, Yesterday was network work from 7:30 AM to 3 AM today (including driving to pickup the rack). By the time 3 AM rolled around, I was quite unhappy that I decided to roll over DNS in order to force myself to get things working, as I ended up going back to FiOS for client access only. Today started around 10 AM, and here I am – 6:30 PM, and I just got things working partially right. I have mail working – arguably the most important – for jasonantman.com only, though I have yet to setup any aliases.

On the web side, I’m working to setup name-based vhosts for all of the subdomains, but for some reason, blog is showing up for everything. Luckily it works right. So we’ll see….