Jason Antman’s Blog

Tonight, more or less on a whim, I moved my blog from my older (dual 1GHz Pentium III Coppermine, 1GB RAM, 10k RPM SCSI disks, Compaq Proliant DL360 G1, OpenSuSE 10.2 32-bit) web server to my newer one (dual 1.4GHz Pentium III, 2GB RAM, 10k RPM SCSI disks, HP Proliant DL360 G2, CentOS 5.3 32-bit). I did some profiling with ab (ApacheBench), and just moving from one server to the other got some serious performance gains (I was profiling with runs of 1000 requests total, 10 concurrent requests). I also added the W3 Total Cache Wordpress plugin, which got the numbers to look even better!

As a side note, this was all done pretty quickly (moving the database and tarball for the vhost, installing the plugin, changing DNS), so please give me a heads-up if you experience any problems.

The numbers are rather impressive:

All tests were performed on my workstation, a Dell Precision 470, two dual-core Xeons at 2.8 GHz, 2GB RAM, 16GB swap, OpenSuSE 11.1 64-bit. This was on the same LAN and subnet as the servers, with the workstation connected via a 1Gbps copper Ethernet link and the web-serving interfaces of the servers connected via 100Mbps (There’s a trunk in between, from the gigabit aggregation switch to the 100Mbps distribution switch).

Projects apache, optimization, performance, wordpress

If you’re just looking for the script or PHP module, you can get them via Subversion at: http://svn.jasonantman.com/nagios-xml/.

A while ago (back in late 2008), I wrote a PHP script that parses the Nagios status.dat file into an associative array. My original use was to output XML which was then read by another script on another server and used for a small custom GUI. It’s a very simple PHP script that just takes the path of the status.dat file (which, obviously, must be readable by the user running the script).

At that time, I was using Nagios v2. Since then, I’ve moved to Nagios v3, and have updated the script to include the ability to parse v3 status.dat files, as well as a function to detect the version of a status file. I also refactored the code so that the parsing functions are all contained in a single file (statusXML.php.inc) which is safe to include in other scripts. The actual statusXML.php file now just includes examples of how to call all of the functions and output XML (though it is equally useful to output the serialized array, or use it directly).

Since I posted my script online, two people have been kind enough to send back their modifications:

• Artur Krzywański modified the original (r4) version of statusXML.php to allow selection of the keys to be returned.
• Whitham D. Reeve II of General Communication, Inc., who needed higher performance for a very large status file, rewrote my script in C as a PHP module.

Both of these generous contributions have been included in my Subversion repository as of the current revision, 5. Unfortunately, due to my delay in putting my Nagios3 code into svn, both of these contributions are Nagios v2 only.

As time permits, I plan on merging Artur’s changes into the current version of statusXML.php.inc. Unfortunately, C isn’t one of my strong points, but I plan on also updating Whitham’s PHP module code to work with Nagios3 as soon as possible.

Stay tuned for updates, and thanks to both gentlemen for contributing their work. I’m always interested in hearing how people are using my code, and how they are making it better.

Also: While I added this project to Nagios Exchange, and plan on adding it to Monitoring Exchange, I don’t always keep those sites up to date (I can’t access Nagios Exchange right now, and who knows if I’ll have time to update it tomorrow). I strongly recommend directly checking out from Subversion at http://svn.jasonantman.com/nagios-xml/ or taking a look at the code through ViewVC at http://viewvc.jasonantman.com/cgi-bin/viewvc.cgi/nagios-xml/.

Projects Nagios, PHP, xml

So, here’s the “official” scoop on the new project that I’m planning/starting to work on. I’m calling it PHPsa for now, and it’s going to (hopefully) be an integrated dashboard/portal for SysAdmins. While there are a number of tools that fit into this general category (perhaps with being the closest, though it’s security-minded), I feel that there’s a real gap in terms of tool integration. My daily workflow, which includes multiple trips to and correlation among Nagios, Cacti, DNS, DHCP, Puppet, logs, and other tools really leaves something to be desired. So, I’m setting out to create a modular SysAdmin dashboard that unifies many of the common SysAdmin-related tools into a modular dashboard.

The first overall design goals that I’ve set are:

1. A modular, plugin-based architecture that allows admins to select which features/tools they want, and allows easy development of new modules.
2. Design with legacy tools in mind – easy ways to tie in to tools that weren’t written with PHPsa in mind, both in terms of linking to information and gathering/unifying information.
3. RBAC, including per-module rules and the possibility for a limited read-only view (client/user mode).
4. Use of data sources, specifically databases, from existing tools with as little modification as possible.
5. Support for database abstraction, though I’ll be using MySQL.
6. Eventually, implement RSS feeds of pertinent information.
7. Balance Ajax/DHTML with the desire for important things to have canonical, static, bookmark-able URLs.

So, here are some of the things that I’m planning on integrating, with obvious bias towards getting my own projects done before I integrate pre-existing tools:

• MultiBindAdmin, my DNS and DHCP administration tool (specifically geared towards split-view DNS with the inside view behind NAT).
• RackMan, my tool for mapping devices’ physical locations in racks (and tacking patching).
• My simple config tool for Puppet.
• Nagios.
• Cacti.
• Nathan Hubbard’s MachDB.
• Bacula (monitoring/status only).
• Syslog via rsyslog (or any other syslog-to-SQL solution).
• Possibly a front-end to Google Analytics.
• Some of my custom scripts for graphing SpamAssassin, DNS queries, etc.
• Some sort of Apache log analysis, like Webalizer.
• Mail log analysis, possibly AWstats.

So, the first big issues that I’m going to tackle:

1. General layout. Specifically, how to handle a more-or-less consistent layout while integrating tools that weren’t designed for PHPsa. I’ll probably end up using iFrames (or even a frameset) for tools that don’t integrate well.
2. How to correlate data/objects between different tools (i.e. how to display information from Nagios, Cacti, MultiBindAdmin and MachDB for a given host?).
3. Do I want to use a templating engine like Smarty or hand-code all of the HTML?
4. How will I handle plugins?
5. How much code do I want to re-write and how much can I use as-is from other tools? And, on a related note, how much existing data can I access easily from other tools, vs having to use grabber scripts that dump data in MySQL?

Update 2010-02-03: I think this may become a semi-official project for me at $work, which means that I’ll be able to dedicate quite a bit more time to it. Unfortunately, it also means that I will, most likely, have to give up Nathan Hubbard’s MachDB in favor of OCS Inventory NG, a more mature project that already includes inventory support for Linux, Windows and Mac.

PHPsa, Projects bind, cacti, dns, Nagios, PHPsa, Projects, puppet, rsyslog, sysadmin, syslog

Yesterday I finally began moving DNS for my sites from GoDaddy to my own in-house system of master/slave BIND9. While both DNS servers are currently at the same location and on the same WAN connection (heck, they’re beind the same router, too), so is all of the rest of my infrastructure. Migrating jasonantman.com was definitely the most critical task, this has allowed me to easily use my new project, MultiBIND Admin to manage DNS. In addition to just being simpler than using GoDaddy’s tool, it allows me to manage DNS for both the external view and the NATed internal view in one tool. I did have a brief mail outage thanks to some incorrect MX records being served by the slave, and a few other issues with the caching DNS servers at work not expiring the old records, but all seems to be well now. It was a relatively smooth transition, though I haven’t yet moved over some of my older less used domains.

The next part of my project, when I move the ambulance corps hosted services in-house, will be trying to find a decently-priced DNS hosting company that will just act as a slave, to keep DNS up if my WAN connection goes down.

Projects bind, dns, multibindadmin

I know I haven’t been posting a lot, but here’s an update on some of my projects:

• PHP EMS Tools – I’ve done quite a bit of work for the ambulance corps, and intend on rolling this into the main distribution. I’ve also added an Asterisk/AGI module to handle crew call-ins. It’s going to be a long road, as I have to manually diff the ambulance corps version to the trunk version and merge the changes (leaving out anything specific to our organization), but I plan on doing it. The next version will also include historical tracking of roster information (member information, status, positions, committees, etc.) and LDAP integration for authentication.
• PHPsa – My new project, tentatively called PHPsa, is an integrated dashboard for sysadmins. The idea is to develop a plugin-based portal for SA tools. Currently, I will be including some of my own projects – MultiBindAdmin (a tool to administer BIND and DHCPd, specifically geared towards split-view DNS with the inside behind NAT) and RackMan (a tool to track and visualize the location of devices within racks, including ability to temporarily move devices around) – as well as my updates to Nathan Hubbard’s MachDB.

I’ve also done quite a bit of customization of the current version of Nathan Hubbard’s MachDB. My local version is in subversion. It adds detailed network interface information, information on expansion slots, and some extra details for the system and storage. I plan on developing a patch and contacting Nathan once I get a chance. It also includes a Python collector script that I developed.

PHPsa, Projects machdb, php ems tools, PHPsa, Projects, rackman

In terms of ongoing projects, I should be updating RackMan sometime soon, and also adding the demo site.

I’ve begun to move DNS for all of my domains in-house, mostly because since everything is behind NAT, it’s a real pain to manage DNS entries in two places (one of them being GoDaddy’s web interface). Because of the NAT issue, I’m also writing my own BIND configuration tool, currently named MultiBIND Admin. In addition to managing multiple zones in a sane way, it stores all configuration in MySQL. Among other things, it can store different IP addresses for A records for the inside and outside views. Zone files can either be pulled by a script on the name server (push capability is being worked on) or downloaded (for uploading to a DNS hosting provider like GoDaddy).

For my final project for my XML web design class, I’m going to be making some “mashup” with RackMan, Google Maps, Google Visualizer, Nagios, and a few other tools…

Stay tuned…

PHPsa, Projects bind, dns, google, PHPsa, rackman

I’ll be posting more on this in the next few days, but I did a few more upgrades at home, including a Proliant DL380G2 to replace my aged ML370 (G1) storage box (array is failing badly) and a Proliant DL360 G2 as a second web server (and possibly moving Nagios over to that box).

I’m running into some problems with the old management card for the Tripp Lite UPS, and I have a few other issues to sort out, but here’s a photo that I took this weekend after the upgrades (yes, it’s a bit blurry – that happens handheld at 1/10 sec).

ACHTUNG! ALLES LOOKENSPEEPERS!
Alles touristen und non-technischen looken peepers! Das computermachine ist nicht fuer gefingerpoken und mittengrabben.
Ist easy schnappen der springenwerk, blowenfusen und poppencorken mit spitzensparken. Ist nicht fuer gewerken bei das dumpkopfen. Das rubbernecken sichtseeren keepen das cotten-pickenen hans in das pockets muss; relaxen und watchen das blinkenlichten.

(For those of you who aren’t familiar with it, blinkenlights).

Projects

At $WORK, my group runs about two dozen servers that provide services for over 60,000 users. They’re a mix of Windows and Linux, with some old Solaris stuff thrown in there. The one thing they have in common is they’re all hand-built, hand-configured, and old. They’ve been around for a while. At the moment, we don’t even have an adequate backup system.

So, being the closest thing to a SysAdmin we have (my official title is still Student Systems Programmer), it’s my job to build a new installation, configuration and backup infrastructure. We’ve already standardized on CentOS as a University-wide distro, and have a local full mirror, so I don’t need to choose a distro. I do, however, have to plan the installation and backup architecture. The main requirements are:

1. Lowest overall time for bare-metal recovery to a working system.
2. Ease of use, as people other than myself will need to administer it (so they should be able to do so from a cheat sheet in the wiki).
3. Repeatability – it should be easy and intuitive to make an almost-exact-copy of a machine.

I started a thread a few days ago on the SAGE mailing list, which you can find here.

At the moment, it looks like the general idea that I’m going with is to use Kickstart to install the systems, using a basic and minimal Kickstart file. Basic package selection (minimalist) with just what’s needed to configure the system with a hostname and network settings for the management VLAN. I’ll then have Kickstart install and configure a configuration management package – I’m leaning towards Puppet over Cfengine and am starting testing. The config management software will handle all of the customization for the system (everything different from the base generic Kickstart install) so it’s all kept under the control of config management from step 1.

The final part is a backup system, mainly for whatever eventually – whether out of human error or simple laziness – ends up out of the config management system’s control. Our previous SA had settled on Zmanda, the paid version of Amanda, which comes with specific plugins for MySQL and MSSQL. I’m also looking at Bacula, mainly because of its’ advanced features, scheduling (especially the new scheduling in Bacula 3) and scalability.

The beauty that I see in having Kickstart do something minimal and then letting Puppet handle the rest is that (especially since we’ve standardized on SunFire X4100’s with identical configurations) I can kickstart and rack up a few spare machines, and to get them up and running all I need to do is power them up (iLOM) and tell Puppet what to make them.

I’m currently starting testing of both Puppet itself and getting Kickstart to start the puppet install and daemon (instructions from David Lutterkort’s blog (Red Hat software engineer)). We’ll see how everything goes…

Projects backup, configuration, kickstart, linux, puppet, recovery, work

Well, I don’t have time to go into a lot of detail, but I thought I’d give a recap of what’s going on. I went down to Mount Holly, NJ yestserday morning – about a 2-/12 hour drive each way, and picked up a 41U rack for the basement. Pretty damn heavy, took me two hours to disassemble it, wrangle it down the stairs, and get it back together. It’s an old round-hole rack, which didn’t seem to matter much until I found that the tabs of the Dell Rapid Rails are just a bit too big to fit in it, so neither my rack KMM nor the rails for my mail server will fit. A bigger problem, though, is that the guy told me it was the standard HP 29″ deep, and I found it to be 28-1/4″ deep when I started racking things up. So, though I just spent $200 on rails for old Proliants, they’re all about 1/2″ too long to fit.

Yesterday, I also had Cablevision show up to install the new Optimum Business with 5 static IPs.

So, last night around 9:00, I started the arduous task of (for the first time ever) powering down ALL of my machines, moving them to the new rack, and re-cabling. That took about 2-1/2 hours, after which my intent was to bring up the new Optimum connection, configure the Vyatta router, and roll over mail and web. From what I’d read of the Vyatta docs it seemed a relatively straightforward task, and being the stubborn jackass that I am, I decied, “hey, it’s my personal site, it’s low traffic, and I want it up before I go to sleep. I’ll roll over DNS before I bring everything up.”

That was a very bad idea. Vyatta isn’t nearly as simple as it seems – especially for someone who isn’t really a network (or at least router/firewall) guy. When they say Enterprise, they mean robust. They also mean that week-long bootcamps aren’t for naught. It took me about half an hour to figure out that even if no “firewall” ruleset is associated with an interface, it still has an implicit drop all. And if you only want to firewall what’s coming in from the outside world, and let everything out, you need to add explicit allow all rules to the in and out sides of the LAN inteface and the out side of the WAN interface.

To top all this off, I had some serious still-unexplained DHCP problems on the LAN, a serious issue since I just set all my hosts to DHCP (which I’ll probably undo soon). So, Yesterday was network work from 7:30 AM to 3 AM today (including driving to pickup the rack). By the time 3 AM rolled around, I was quite unhappy that I decided to roll over DNS in order to force myself to get things working, as I ended up going back to FiOS for client access only. Today started around 10 AM, and here I am – 6:30 PM, and I just got things working partially right. I have mail working – arguably the most important – for jasonantman.com only, though I have yet to setup any aliases.

On the web side, I’m working to setup name-based vhosts for all of the subdomains, but for some reason, blog is showing up for everything. Luckily it works right. So we’ll see….

Projects firewall, optimum, rack, router, wan

Well, I finally broke down and ordered Optimum Business. Come tomorrow, I’ll be moving from Verizon FiOS residential with a dynamic IP, much blocked (hence jantman.dyndns.org:10011) and 10Mbps down/2Mbps up to Optimum Business with 30 down/5 up, a block of 5 static IPs, and no blocked ports.

It’s going to be a crazy weekend. Probably not the best thing the week before midterms, but oh well. Tomorrow morning I’m picking up a 42U rack for home to replace the Sears shelving unit my boxes are currently on. Cablevision is supposed to be here between 2-5 PM to do the install (yes, they insist that for Business they do the install, even though it’s only a 4-foot coax run from the first splitter to the demarc). I’ve got Vyatta CE5 Beta installed on a Proliant DL360G2 as the new router, ready to go (after some configuration). I’ll probably keep FiOS up until I know the new router is working correctly (I’ll do a test on my management VLAN).

Once Optimum and the new router is up, the fun starts:

1. Forward the appropriate ports on the new router, including 80 (in addition to 10011).
2. Bring the old router down and make sure the new one is up, operational, and forwarding all the right ports.
3. Update DynDNS to point to the first IP, used as a catch-all for old DynDNS links.
4. Begin assignment of the 5 IPs (everything will be behind NAT) based on a list of what hosts need valid reverse DNS, and then adding other ports (NATed) as needed.
5. Update DNS for JasonAntman.com and the other domains.
6. Update Optimum reverse DNS.
7. Ensure that everything works as planned, DNS is up, ports are forwarded, and everything is as before (at least in terms of HTTP).
8. Once DNS is up, reconfigure Apache to have a vhost handling any legacy requests to port 10011 and rewrite them to www.jasonantman.com.
9. Setup a vhost for ‘www’ that takes URLs that used to be subdirectories (i.e. www.jasonantman.com/blog) and rewrites them to requests for the appropriate subdomain. Simultaneously move everything from the default vhost to name-based vhosts.
10. Ensure that old jantman.dyndns.org:10011 requests are being redirected properly, and requests for subdirectories under the web root are going to the right subdomain.
11. Check that this all works acceptably with the existing blogger-to-wordpress rewrite script.
12. Finally start rolling out some of the new services that I had waiting for the new connection.
13. Start the arduous process of reconfiguring my mail server, moving from Fetchmail from Verizon to an actual mail server, make everything work, and make sure my IPs aren’t blacklisted.
14. Ugh. Find anywhere in the entire ‘net where my old @verizon.net address appeared (especially GoDaddy, DynDNS, other important stuff) and change it to the new jasonantman.com address.
15. Since this is all in my mother’s basement (there’s nothing like a mother’s love, especially when it comes to a constant hum emanating from the ground level of a house), figure out what to do for her when the verizon.net email goes away.

So I might have some downtime this weekend, but when things come back up, I’ll be done with this DynDNS and Port 10011 crap.

Projects fios, optimum, vyatta