Jason Antman’s Blog

So it’s been a very busy day. I was up until 5 AM or so working on implementing Puppet at home. I’m building two new boxes – a storage (centralized home directory)/syslog (to MySQL) server and a second web server (possibly also to handle Nagios) – and I decided that they’ll be totally built by Puppet. The only thing I had to give up on was setting up the NFS share for my home directory on the new storage box and installing and testing rsyslog on it.

This afternoon around 7, I started on my weekend projects for the ambulance corps – setting up Nagios to receive SNMP traps from the APC UPS and moving over to the new Vyatta-based router (from m0n0wall). I’d attempted the router before, but had to rollback – I’m using an old BlueSocket controller for hardware – it’s just a nice black 1U enclosure with a stock Intel motherboard, 20GB HDD, 512MB RAM and three 10/100 NICs. The first time, I was unable to get link on either of the two NICs I was using, so I decided to rollback.

Nagios SNMP Traps

I found a good starting point for Nagios SNMP traps on the OpsView blog. I setup `snmptrapd` on the Nagios server and hacked together a little Python script to just write all of the traps to a file. After some testing with `snmptrap` on my laptop, I did a test by pulling the power plug of the UPS, waiting about 30 seconds, and then plugging it back in. Sure enough, the little old AP9605 PowerNet SNMP card generated two SNMP traps – one for power loss and one for power regained – both of which showed up in the test file

The next step will be deciding how to get the traps into Nagios – specifically whether I want to go with something heavy-weight, like SNMPtt that can handle other devices, or whether I want to code a simple script myself just to deal with the APC cards.

Router

The main reason why I wanted to make the switch from m0n0 to Vyatta was to ease the setup and maintenance of an IPsec tunnel from the ambulance HQ to my house, so I could push backups (relatively small) over the WAN to my infrastructure (or, rather, have Bacula pull the backups). Another big bonus was finally having a way of configuring and checking things through SSH without having to port-forward a web GUI. Another bonus of having a real Linux system under the router is the ability to make custom Nagios check scripts and easily execute them. Something I hadn’t thought of – but became obvious during the switchover – is the ability to run full-fledged `tcpdump` on the router itself.

After building the new config myself, and confirming that the system ran in isolation, I moved it over to production. The first issue was a bit of a thinko on my part – the interfaces on the BSC are actually arranged on the back of the box like eth0—–eth2—–eth1, so I originally had the LAN uplink in the wrong interface. After correcting that and waiting for the network to stabilize, I noticed a total external connectivity failure. After some troubleshooting – thanks to tcpdump on the router – it occurred to me that the (ancient) cable modem needs to be rebooted when the router MAC changes.

I honestly don’t remember the other problems that I ran into, but eventually I ended up getting almost-full functionality – and then a total network outage. A tcpdump on my laptop showed some really really weird BOOTP traffic with addresses of 255.255.255.255. After doing some troubleshooting and monitoring port counters on the switch, I narrowed it down to coming from a single Windows box and the wireless access point. After shutting off both ports, things seemed to stabilize. I also had some “martian address” issues with one of the boxes, but decided to roll the box and that solved it.

Over the next day or so, I’ll be reconfiguring Nagios both at home and at the ambulance corps to cope with the changes and add in the requisite monitoring, and keep an eye on things. Assuming all goes well, I’ll power down the old router on Sunday.

On the home front, I’ve moved over from my old storage machine to the old one – essentially just the NFS mount, and moved over a tarball of everything else. I also added a 1000Base-SX card to the new box, though it appears that I’m out of fiber patch cords. The old storage box was brought down for the first time in about 3 years (aside from brief outages for hardware upgrades or array rebuilds). Assuming I got everything off of it, it will be relegated to the spares pile.

I’m going to make a serious effort to post on a daily basis, if only for my own future reference. I should have the demo of RackMan out soon, and I’m also about to start on integrating it with Nathan Hubbard’s MachDB as well as a PHP script I wrote to pull port names and MACs from Cisco switches and associate them with NICs in machines. Hopefully I’ll also have some interesting Puppet stuff out soon.

Miscellaneous Geek Stuff apc, bluesocket, Nagios, snmp, vyatta

Well, I finally broke down and ordered Optimum Business. Come tomorrow, I’ll be moving from Verizon FiOS residential with a dynamic IP, much blocked (hence jantman.dyndns.org:10011) and 10Mbps down/2Mbps up to Optimum Business with 30 down/5 up, a block of 5 static IPs, and no blocked ports.

It’s going to be a crazy weekend. Probably not the best thing the week before midterms, but oh well. Tomorrow morning I’m picking up a 42U rack for home to replace the Sears shelving unit my boxes are currently on. Cablevision is supposed to be here between 2-5 PM to do the install (yes, they insist that for Business they do the install, even though it’s only a 4-foot coax run from the first splitter to the demarc). I’ve got Vyatta CE5 Beta installed on a Proliant DL360G2 as the new router, ready to go (after some configuration). I’ll probably keep FiOS up until I know the new router is working correctly (I’ll do a test on my management VLAN).

Once Optimum and the new router is up, the fun starts:

1. Forward the appropriate ports on the new router, including 80 (in addition to 10011).
2. Bring the old router down and make sure the new one is up, operational, and forwarding all the right ports.
3. Update DynDNS to point to the first IP, used as a catch-all for old DynDNS links.
4. Begin assignment of the 5 IPs (everything will be behind NAT) based on a list of what hosts need valid reverse DNS, and then adding other ports (NATed) as needed.
5. Update DNS for JasonAntman.com and the other domains.
6. Update Optimum reverse DNS.
7. Ensure that everything works as planned, DNS is up, ports are forwarded, and everything is as before (at least in terms of HTTP).
8. Once DNS is up, reconfigure Apache to have a vhost handling any legacy requests to port 10011 and rewrite them to www.jasonantman.com.
9. Setup a vhost for ‘www’ that takes URLs that used to be subdirectories (i.e. www.jasonantman.com/blog) and rewrites them to requests for the appropriate subdomain. Simultaneously move everything from the default vhost to name-based vhosts.
10. Ensure that old jantman.dyndns.org:10011 requests are being redirected properly, and requests for subdirectories under the web root are going to the right subdomain.
11. Check that this all works acceptably with the existing blogger-to-wordpress rewrite script.
12. Finally start rolling out some of the new services that I had waiting for the new connection.
13. Start the arduous process of reconfiguring my mail server, moving from Fetchmail from Verizon to an actual mail server, make everything work, and make sure my IPs aren’t blacklisted.
14. Ugh. Find anywhere in the entire ‘net where my old @verizon.net address appeared (especially GoDaddy, DynDNS, other important stuff) and change it to the new jasonantman.com address.
15. Since this is all in my mother’s basement (there’s nothing like a mother’s love, especially when it comes to a constant hum emanating from the ground level of a house), figure out what to do for her when the verizon.net email goes away.

So I might have some downtime this weekend, but when things come back up, I’ll be done with this DynDNS and Port 10011 crap.

Projects fios, optimum, vyatta

I’m part-way through the major overhaul of my home network (hosting this blog and everything else jasonantman.com) that I’ve been planning for quite some time. The current hardware is… uh… currently… described on my Hardware page, but I soon plan on ditching the wiki and moving to a CMS for my entire site.

Anyway, so far I’ve decommissioned my aged HP ProCurve 2424M switch and replaced it with used but less-aged Cisco 2948G from Horizon Datacom (purchased on Ebay). Quite an upgrade. In order to handle network backups a little better, I’m also adding a Cisco 4912G 12-port Gigabit (GBIC) aggregation switch for the administrative/backup VLAN – though this was purchased via ebay from RedApe Technologies in PA. The switch came with 12 1000BASE-SX GBICs, and I plan to do a mix of copper (1000BASE-T) where it’s already available (onboard NICs) and 1000BASE-SX where there’s enough room in the box for a card.

On the hardware side, I also have 2 new boxes – a set of HP Proliant DL360 G2’s from MJS Global, who I’ve done business with before. The prices were great, and though one of them showed up with a faulty temperature sensor that prevents boot, MJS has been wonderful and is shipping me a replacement motherboard. One of the boxes will be running Vyatta (vee-AH-tha) VC5 router/firewall software, and the other will be a new services box running internal DNS, DHCP, NTP, and whatever else.

On the hardware side, I’m also planning some extended downtime a few weekends from now, when I should finally have a 42U rack to replace the Sears shelves my equipment is now on. It’ll be a fun-filled evening of racking equipment and re-patching everything. Also, hopefully within a few weeks, I’ll be moving my WAN pipe from Verizon FiOS residential to Optimum Business, which is essentially re-packaged residential but provides 5 static IPs, no blocked ports, and 30 Mbps down/5 Mbps up.

Vyatta

When planning this upgrade, I think I looked at every open source router package out there, as well as some of the lower-end or older Cisco models. I’m currently running IPcop, which does everything I need except it doesn’t handle multiple WAN IPs, and all configuration is via a web interface – which means every time I want to make a change remotely (and during the week I’m not home) I have to forward HTTPS over SSH. After doing an extensive feature comparison, I ended up narrowing it down to a relative newcomer – Vyatta. Though I don’t know how much of it is marketing hype, they are targeted squarely at Cisco, and provide relatively enterprise-level features; a JunOS-based CLI, BGP, OSPF, and all of the other important stuff.

Yesterday I attempted an install of Vyatta CE 5 Beta on one of the DL360G2’s. The only real problem that I found was the install script doesn’t support CCISS drives, as found in the Proliants, but a few manual hacks to the script fixed that. By far the best thing about Vyatta is it’s based on vanilla Debian Lenny, and full root shell access is available, so modifying the install script – or even adding non-Vyatta packages – is a cinch. I haven’t really played around with it too much, but it appears to be a wonderful mix of Linux and an enterprise router CLI. While root has a full BASH shell, and the Vyata commands are all done as shell aliases (so users still have access to shell primitives and OS commands), configuration is accomplished via a JunOS-like command set. You still get “commit” and “rollback” in config mode, and can still do fun things like save and load configs to/from tftp, ftp and http. On the other hand, I doubt I’ll do config backups that way since I can just use scp or sftp.

The Vyatta box will probably go home this weekend, and get hooked up to the network for config-only use (and I can always get in via iLO on the hardware) and hopefully come up sometime in the next few weeks.

At this point, the most daunting task is figuring out how to get all of the existing links to my site to work – since jantman.dyndns.org will be legacy, and most of the site structure will probably change to use name-based vhosts. Lately I’ve been trying to use the real subdomains in all of my public links, so the transition (planned for a while) will work, but I’m sure there are still plenty of links out there that will need dealing with (maybe keep port 10011 serving HTTP with a massive mod_rewrite script to redirect to the right place???), as well as checking everything on the web server to make sure there aren’t any absolute URLs (like WordPress).

Projects cisco, networking, router, vyatta