Jason Antman’s Blog

If you’re just looking for the script or PHP module, you can get them via Subversion at: http://svn.jasonantman.com/nagios-xml/.

A while ago (back in late 2008), I wrote a PHP script that parses the Nagios status.dat file into an associative array. My original use was to output XML which was then read by another script on another server and used for a small custom GUI. It’s a very simple PHP script that just takes the path of the status.dat file (which, obviously, must be readable by the user running the script).

At that time, I was using Nagios v2. Since then, I’ve moved to Nagios v3, and have updated the script to include the ability to parse v3 status.dat files, as well as a function to detect the version of a status file. I also refactored the code so that the parsing functions are all contained in a single file (statusXML.php.inc) which is safe to include in other scripts. The actual statusXML.php file now just includes examples of how to call all of the functions and output XML (though it is equally useful to output the serialized array, or use it directly).

Since I posted my script online, two people have been kind enough to send back their modifications:

• Artur Krzywański modified the original (r4) version of statusXML.php to allow selection of the keys to be returned.
• Whitham D. Reeve II of General Communication, Inc., who needed higher performance for a very large status file, rewrote my script in C as a PHP module.

Both of these generous contributions have been included in my Subversion repository as of the current revision, 5. Unfortunately, due to my delay in putting my Nagios3 code into svn, both of these contributions are Nagios v2 only.

As time permits, I plan on merging Artur’s changes into the current version of statusXML.php.inc. Unfortunately, C isn’t one of my strong points, but I plan on also updating Whitham’s PHP module code to work with Nagios3 as soon as possible.

Stay tuned for updates, and thanks to both gentlemen for contributing their work. I’m always interested in hearing how people are using my code, and how they are making it better.

Also: While I added this project to Nagios Exchange, and plan on adding it to Monitoring Exchange, I don’t always keep those sites up to date (I can’t access Nagios Exchange right now, and who knows if I’ll have time to update it tomorrow). I strongly recommend directly checking out from Subversion at http://svn.jasonantman.com/nagios-xml/ or taking a look at the code through ViewVC at http://viewvc.jasonantman.com/cgi-bin/viewvc.cgi/nagios-xml/.

Projects Nagios, PHP, xml

So, here’s the “official” scoop on the new project that I’m planning/starting to work on. I’m calling it PHPsa for now, and it’s going to (hopefully) be an integrated dashboard/portal for SysAdmins. While there are a number of tools that fit into this general category (perhaps with being the closest, though it’s security-minded), I feel that there’s a real gap in terms of tool integration. My daily workflow, which includes multiple trips to and correlation among Nagios, Cacti, DNS, DHCP, Puppet, logs, and other tools really leaves something to be desired. So, I’m setting out to create a modular SysAdmin dashboard that unifies many of the common SysAdmin-related tools into a modular dashboard.

• MultiBindAdmin, my DNS and DHCP administration tool (specifically geared towards split-view DNS with the inside view behind NAT).
• RackMan, my tool for mapping devices’ physical locations in racks (and tacking patching).
• My simple config tool for Puppet.
• Nagios.
• Cacti.
• Nathan Hubbard’s MachDB.
• Bacula (monitoring/status only).
• Syslog via rsyslog (or any other syslog-to-SQL solution).
• Possibly a front-end to Google Analytics.
• Some of my custom scripts for graphing SpamAssassin, DNS queries, etc.
• Some sort of Apache log analysis, like Webalizer.
• Mail log analysis, possibly AWstats.

So, the first big issues that I’m going to tackle:

1. General layout. Specifically, how to handle a more-or-less consistent layout while integrating tools that weren’t designed for PHPsa. I’ll probably end up using iFrames (or even a frameset) for tools that don’t integrate well.
2. How to correlate data/objects between different tools (i.e. how to display information from Nagios, Cacti, MultiBindAdmin and MachDB for a given host?).
3. Do I want to use a templating engine like Smarty or hand-code all of the HTML?
4. How will I handle plugins?
5. How much code do I want to re-write and how much can I use as-is from other tools? And, on a related note, how much existing data can I access easily from other tools, vs having to use grabber scripts that dump data in MySQL?

Update 2010-02-03: I think this may become a semi-official project for me at $work, which means that I’ll be able to dedicate quite a bit more time to it. Unfortunately, it also means that I will, most likely, have to give up Nathan Hubbard’s MachDB in favor of OCS Inventory NG, a more mature project that already includes inventory support for Linux, Windows and Mac.

PHPsa, Projects bind, cacti, dns, Nagios, PHPsa, Projects, puppet, rsyslog, sysadmin, syslog

Yesterday I finally began moving DNS for my sites from GoDaddy to my own in-house system of master/slave BIND9. While both DNS servers are currently at the same location and on the same WAN connection (heck, they’re beind the same router, too), so is all of the rest of my infrastructure. Migrating jasonantman.com was definitely the most critical task, this has allowed me to easily use my new project, MultiBIND Admin to manage DNS. In addition to just being simpler than using GoDaddy’s tool, it allows me to manage DNS for both the external view and the NATed internal view in one tool. I did have a brief mail outage thanks to some incorrect MX records being served by the slave, and a few other issues with the caching DNS servers at work not expiring the old records, but all seems to be well now. It was a relatively smooth transition, though I haven’t yet moved over some of my older less used domains.

The next part of my project, when I move the ambulance corps hosted services in-house, will be trying to find a decently-priced DNS hosting company that will just act as a slave, to keep DNS up if my WAN connection goes down.

Projects bind, dns, multibindadmin

I know I haven’t been posting a lot, but here’s an update on some of my projects:

• PHP EMS Tools – I’ve done quite a bit of work for the ambulance corps, and intend on rolling this into the main distribution. I’ve also added an Asterisk/AGI module to handle crew call-ins. It’s going to be a long road, as I have to manually diff the ambulance corps version to the trunk version and merge the changes (leaving out anything specific to our organization), but I plan on doing it. The next version will also include historical tracking of roster information (member information, status, positions, committees, etc.) and LDAP integration for authentication.
• PHPsa – My new project, tentatively called PHPsa, is an integrated dashboard for sysadmins. The idea is to develop a plugin-based portal for SA tools. Currently, I will be including some of my own projects – MultiBindAdmin (a tool to administer BIND and DHCPd, specifically geared towards split-view DNS with the inside behind NAT) and RackMan (a tool to track and visualize the location of devices within racks, including ability to temporarily move devices around) – as well as my updates to Nathan Hubbard’s MachDB.

I’ve also done quite a bit of customization of the current version of Nathan Hubbard’s MachDB. My local version is in subversion. It adds detailed network interface information, information on expansion slots, and some extra details for the system and storage. I plan on developing a patch and contacting Nathan once I get a chance. It also includes a Python collector script that I developed.

PHPsa, Projects machdb, php ems tools, PHPsa, Projects, rackman

In terms of ongoing projects, I should be updating RackMan sometime soon, and also adding the demo site.

I’ve begun to move DNS for all of my domains in-house, mostly because since everything is behind NAT, it’s a real pain to manage DNS entries in two places (one of them being GoDaddy’s web interface). Because of the NAT issue, I’m also writing my own BIND configuration tool, currently named MultiBIND Admin. In addition to managing multiple zones in a sane way, it stores all configuration in MySQL. Among other things, it can store different IP addresses for A records for the inside and outside views. Zone files can either be pulled by a script on the name server (push capability is being worked on) or downloaded (for uploading to a DNS hosting provider like GoDaddy).

For my final project for my XML web design class, I’m going to be making some “mashup” with RackMan, Google Maps, Google Visualizer, Nagios, and a few other tools…

Stay tuned…

PHPsa, Projects bind, dns, google, PHPsa, rackman

I’ll be posting more on this in the next few days, but I did a few more upgrades at home, including a Proliant DL380G2 to replace my aged ML370 (G1) storage box (array is failing badly) and a Proliant DL360 G2 as a second web server (and possibly moving Nagios over to that box).

I’m running into some problems with the old management card for the Tripp Lite UPS, and I have a few other issues to sort out, but here’s a photo that I took this weekend after the upgrades (yes, it’s a bit blurry – that happens handheld at 1/10 sec).

ACHTUNG! ALLES LOOKENSPEEPERS!
Alles touristen und non-technischen looken peepers! Das computermachine ist nicht fuer gefingerpoken und mittengrabben.
Ist easy schnappen der springenwerk, blowenfusen und poppencorken mit spitzensparken. Ist nicht fuer gewerken bei das dumpkopfen. Das rubbernecken sichtseeren keepen das cotten-pickenen hans in das pockets muss; relaxen und watchen das blinkenlichten.

(For those of you who aren’t familiar with it, blinkenlights).

Projects

At $WORK, my group runs about two dozen servers that provide services for over 60,000 users. They’re a mix of Windows and Linux, with some old Solaris stuff thrown in there. The one thing they have in common is they’re all hand-built, hand-configured, and old. They’ve been around for a while. At the moment, we don’t even have an adequate backup system.

So, being the closest thing to a SysAdmin we have (my official title is still Student Systems Programmer), it’s my job to build a new installation, configuration and backup infrastructure. We’ve already standardized on CentOS as a University-wide distro, and have a local full mirror, so I don’t need to choose a distro. I do, however, have to plan the installation and backup architecture. The main requirements are:

1. Lowest overall time for bare-metal recovery to a working system.
2. Ease of use, as people other than myself will need to administer it (so they should be able to do so from a cheat sheet in the wiki).
3. Repeatability – it should be easy and intuitive to make an almost-exact-copy of a machine.

I started a thread a few days ago on the SAGE mailing list, which you can find here.

At the moment, it looks like the general idea that I’m going with is to use Kickstart to install the systems, using a basic and minimal Kickstart file. Basic package selection (minimalist) with just what’s needed to configure the system with a hostname and network settings for the management VLAN. I’ll then have Kickstart install and configure a configuration management package – I’m leaning towards Puppet over Cfengine and am starting testing. The config management software will handle all of the customization for the system (everything different from the base generic Kickstart install) so it’s all kept under the control of config management from step 1.

The final part is a backup system, mainly for whatever eventually – whether out of human error or simple laziness – ends up out of the config management system’s control. Our previous SA had settled on Zmanda, the paid version of Amanda, which comes with specific plugins for MySQL and MSSQL. I’m also looking at Bacula, mainly because of its’ advanced features, scheduling (especially the new scheduling in Bacula 3) and scalability.

The beauty that I see in having Kickstart do something minimal and then letting Puppet handle the rest is that (especially since we’ve standardized on SunFire X4100’s with identical configurations) I can kickstart and rack up a few spare machines, and to get them up and running all I need to do is power them up (iLOM) and tell Puppet what to make them.

I’m currently starting testing of both Puppet itself and getting Kickstart to start the puppet install and daemon (instructions from David Lutterkort’s blog (Red Hat software engineer)). We’ll see how everything goes…

Projects backup, configuration, kickstart, linux, puppet, recovery, work

Well, I don’t have time to go into a lot of detail, but I thought I’d give a recap of what’s going on. I went down to Mount Holly, NJ yestserday morning – about a 2-/12 hour drive each way, and picked up a 41U rack for the basement. Pretty damn heavy, took me two hours to disassemble it, wrangle it down the stairs, and get it back together. It’s an old round-hole rack, which didn’t seem to matter much until I found that the tabs of the Dell Rapid Rails are just a bit too big to fit in it, so neither my rack KMM nor the rails for my mail server will fit. A bigger problem, though, is that the guy told me it was the standard HP 29″ deep, and I found it to be 28-1/4″ deep when I started racking things up. So, though I just spent $200 on rails for old Proliants, they’re all about 1/2″ too long to fit.

Yesterday, I also had Cablevision show up to install the new Optimum Business with 5 static IPs.

So, last night around 9:00, I started the arduous task of (for the first time ever) powering down ALL of my machines, moving them to the new rack, and re-cabling. That took about 2-1/2 hours, after which my intent was to bring up the new Optimum connection, configure the Vyatta router, and roll over mail and web. From what I’d read of the Vyatta docs it seemed a relatively straightforward task, and being the stubborn jackass that I am, I decied, “hey, it’s my personal site, it’s low traffic, and I want it up before I go to sleep. I’ll roll over DNS before I bring everything up.”

That was a very bad idea. Vyatta isn’t nearly as simple as it seems – especially for someone who isn’t really a network (or at least router/firewall) guy. When they say Enterprise, they mean robust. They also mean that week-long bootcamps aren’t for naught. It took me about half an hour to figure out that even if no “firewall” ruleset is associated with an interface, it still has an implicit drop all. And if you only want to firewall what’s coming in from the outside world, and let everything out, you need to add explicit allow all rules to the in and out sides of the LAN inteface and the out side of the WAN interface.

To top all this off, I had some serious still-unexplained DHCP problems on the LAN, a serious issue since I just set all my hosts to DHCP (which I’ll probably undo soon). So, Yesterday was network work from 7:30 AM to 3 AM today (including driving to pickup the rack). By the time 3 AM rolled around, I was quite unhappy that I decided to roll over DNS in order to force myself to get things working, as I ended up going back to FiOS for client access only. Today started around 10 AM, and here I am – 6:30 PM, and I just got things working partially right. I have mail working – arguably the most important – for jasonantman.com only, though I have yet to setup any aliases.

On the web side, I’m working to setup name-based vhosts for all of the subdomains, but for some reason, blog is showing up for everything. Luckily it works right. So we’ll see….

Projects firewall, optimum, rack, router, wan

Well, I finally broke down and ordered Optimum Business. Come tomorrow, I’ll be moving from Verizon FiOS residential with a dynamic IP, much blocked (hence jantman.dyndns.org:10011) and 10Mbps down/2Mbps up to Optimum Business with 30 down/5 up, a block of 5 static IPs, and no blocked ports.

It’s going to be a crazy weekend. Probably not the best thing the week before midterms, but oh well. Tomorrow morning I’m picking up a 42U rack for home to replace the Sears shelving unit my boxes are currently on. Cablevision is supposed to be here between 2-5 PM to do the install (yes, they insist that for Business they do the install, even though it’s only a 4-foot coax run from the first splitter to the demarc). I’ve got Vyatta CE5 Beta installed on a Proliant DL360G2 as the new router, ready to go (after some configuration). I’ll probably keep FiOS up until I know the new router is working correctly (I’ll do a test on my management VLAN).

Once Optimum and the new router is up, the fun starts:

1. Forward the appropriate ports on the new router, including 80 (in addition to 10011).
2. Bring the old router down and make sure the new one is up, operational, and forwarding all the right ports.
3. Update DynDNS to point to the first IP, used as a catch-all for old DynDNS links.
4. Begin assignment of the 5 IPs (everything will be behind NAT) based on a list of what hosts need valid reverse DNS, and then adding other ports (NATed) as needed.
5. Update DNS for JasonAntman.com and the other domains.
6. Update Optimum reverse DNS.
7. Ensure that everything works as planned, DNS is up, ports are forwarded, and everything is as before (at least in terms of HTTP).
8. Once DNS is up, reconfigure Apache to have a vhost handling any legacy requests to port 10011 and rewrite them to www.jasonantman.com.
9. Setup a vhost for ‘www’ that takes URLs that used to be subdirectories (i.e. www.jasonantman.com/blog) and rewrites them to requests for the appropriate subdomain. Simultaneously move everything from the default vhost to name-based vhosts.
10. Ensure that old jantman.dyndns.org:10011 requests are being redirected properly, and requests for subdirectories under the web root are going to the right subdomain.
11. Check that this all works acceptably with the existing blogger-to-wordpress rewrite script.
12. Finally start rolling out some of the new services that I had waiting for the new connection.
13. Start the arduous process of reconfiguring my mail server, moving from Fetchmail from Verizon to an actual mail server, make everything work, and make sure my IPs aren’t blacklisted.
14. Ugh. Find anywhere in the entire ‘net where my old @verizon.net address appeared (especially GoDaddy, DynDNS, other important stuff) and change it to the new jasonantman.com address.
15. Since this is all in my mother’s basement (there’s nothing like a mother’s love, especially when it comes to a constant hum emanating from the ground level of a house), figure out what to do for her when the verizon.net email goes away.

So I might have some downtime this weekend, but when things come back up, I’ll be done with this DynDNS and Port 10011 crap.

Projects fios, optimum, vyatta

I’m part-way through the major overhaul of my home network (hosting this blog and everything else jasonantman.com) that I’ve been planning for quite some time. The current hardware is… uh… currently… described on my Hardware page, but I soon plan on ditching the wiki and moving to a CMS for my entire site.

Anyway, so far I’ve decommissioned my aged HP ProCurve 2424M switch and replaced it with used but less-aged Cisco 2948G from Horizon Datacom (purchased on Ebay). Quite an upgrade. In order to handle network backups a little better, I’m also adding a Cisco 4912G 12-port Gigabit (GBIC) aggregation switch for the administrative/backup VLAN – though this was purchased via ebay from RedApe Technologies in PA. The switch came with 12 1000BASE-SX GBICs, and I plan to do a mix of copper (1000BASE-T) where it’s already available (onboard NICs) and 1000BASE-SX where there’s enough room in the box for a card.

On the hardware side, I also have 2 new boxes – a set of HP Proliant DL360 G2’s from MJS Global, who I’ve done business with before. The prices were great, and though one of them showed up with a faulty temperature sensor that prevents boot, MJS has been wonderful and is shipping me a replacement motherboard. One of the boxes will be running Vyatta (vee-AH-tha) VC5 router/firewall software, and the other will be a new services box running internal DNS, DHCP, NTP, and whatever else.

On the hardware side, I’m also planning some extended downtime a few weekends from now, when I should finally have a 42U rack to replace the Sears shelves my equipment is now on. It’ll be a fun-filled evening of racking equipment and re-patching everything. Also, hopefully within a few weeks, I’ll be moving my WAN pipe from Verizon FiOS residential to Optimum Business, which is essentially re-packaged residential but provides 5 static IPs, no blocked ports, and 30 Mbps down/5 Mbps up.

Vyatta

When planning this upgrade, I think I looked at every open source router package out there, as well as some of the lower-end or older Cisco models. I’m currently running IPcop, which does everything I need except it doesn’t handle multiple WAN IPs, and all configuration is via a web interface – which means every time I want to make a change remotely (and during the week I’m not home) I have to forward HTTPS over SSH. After doing an extensive feature comparison, I ended up narrowing it down to a relative newcomer – Vyatta. Though I don’t know how much of it is marketing hype, they are targeted squarely at Cisco, and provide relatively enterprise-level features; a JunOS-based CLI, BGP, OSPF, and all of the other important stuff.

Yesterday I attempted an install of Vyatta CE 5 Beta on one of the DL360G2’s. The only real problem that I found was the install script doesn’t support CCISS drives, as found in the Proliants, but a few manual hacks to the script fixed that. By far the best thing about Vyatta is it’s based on vanilla Debian Lenny, and full root shell access is available, so modifying the install script – or even adding non-Vyatta packages – is a cinch. I haven’t really played around with it too much, but it appears to be a wonderful mix of Linux and an enterprise router CLI. While root has a full BASH shell, and the Vyata commands are all done as shell aliases (so users still have access to shell primitives and OS commands), configuration is accomplished via a JunOS-like command set. You still get “commit” and “rollback” in config mode, and can still do fun things like save and load configs to/from tftp, ftp and http. On the other hand, I doubt I’ll do config backups that way since I can just use scp or sftp.

The Vyatta box will probably go home this weekend, and get hooked up to the network for config-only use (and I can always get in via iLO on the hardware) and hopefully come up sometime in the next few weeks.

At this point, the most daunting task is figuring out how to get all of the existing links to my site to work – since jantman.dyndns.org will be legacy, and most of the site structure will probably change to use name-based vhosts. Lately I’ve been trying to use the real subdomains in all of my public links, so the transition (planned for a while) will work, but I’m sure there are still plenty of links out there that will need dealing with (maybe keep port 10011 serving HTTP with a massive mod_rewrite script to redirect to the right place???), as well as checking everything on the web server to make sure there aren’t any absolute URLs (like WordPress).

Projects cisco, networking, router, vyatta