Jason Antman’s Blog

Just found this today on one of my all-time favorite sites, XKCD. I swear it’s about me:

Miscellaneous Geek Stuff humor, sysadmin, XKCD

At $WORK, the subnet we use for some of our workstations and test boxes was only recently setup with DHCP. Previously, we’d used IP-by-Whiteboard in the office. As a result, most of the recent machines use DHCP, but there are a few older ones still around using static addresses. I recently had to add a new machine, so I had to go through the process of finding out which IPs are in use and which aren’t (since some aren’t in DHCP).

I decided to be good and update DHCP with records for all machines in the subnet, whether they’re actually using DHCP or not. There’s a quick way to do this with nmap using the options for ping scan (-sP) and always resolve DNS (-R):

nmap -sP -R 172.16.43.129-159

Host ar01-hill-hill.example.com (172.16.43.129) appears to be up.
MAC Address: 00:11:BC:7D:28:0A (Cisco Systems)
Host ccf-hill019-1.example.com (172.16.43.130) appears to be up.
MAC Address: 00:00:AA:63:54:BB (Xerox)
Host ccf-hill019-2.example.com (172.16.43.131) appears to be down.
Host ccf-hill019-3.example.com (172.16.43.132) appears to be down.
Host ccf-hill019-4.example.com (172.16.43.133) appears to be down.
Host ccf-hill019-5.example.com (172.16.43.134) appears to be down.
Host ccf-hill019-6.example.com (172.16.43.135) appears to be down.
Host ccf-hill019-7.example.com (172.16.43.136) appears to be up.
Host speakeasy.example.com (172.16.43.137) appears to be up.
MAC Address: 00:17:A4:13:EB:57 (Global Data Services)
Host ccf-hill019-9.example.com (172.16.43.138) appears to be up.
MAC Address: 00:17:A4:13:E8:17 (Global Data Services)
Host ccf-hill019-10.example.com (172.16.43.139) appears to be down.
Host testmac01.example.com (172.16.43.140) appears to be down.
Host ccf-hill019-12.example.com (172.16.43.141) appears to be down.
Host ccf-hill019-13.example.com (172.16.43.142) appears to be up.
MAC Address: 00:0D:29:59:58:00 (Cisco)
Host ccf-hill019-14.example.com (172.16.43.143) appears to be down.
Host ccf-hill019-15.example.com (172.16.43.144) appears to be down.
Host ccf-hill019-16.example.com (172.16.43.145) appears to be down.
Host ccf-hill019-17.example.com (172.16.43.146) appears to be down.
Host ccf-hill019-18.example.com (172.16.43.147) appears to be up.
MAC Address: 00:1E:C2:0D:C1:98 (Unknown)
Host ccf-hill019-19.example.com (172.16.43.148) appears to be down.
Host ccf-hill019-20.example.com (172.16.43.149) appears to be down.
Host ccf-hill019-21.example.com (172.16.43.150) appears to be down.
Host lordkris.example.com (172.16.43.151) appears to be down.
Host ccf-hill019-23.example.com (172.16.43.152) appears to be down.
Host ccf-hill019-24.example.com (172.16.43.153) appears to be down.
Host ccf-hill019-25.example.com (172.16.43.154) appears to be down.
Host ccf-hill019-26.example.com (172.16.43.155) appears to be down.
Host ccf-hill019-27.example.com (172.16.43.156) appears to be down.
Host ccf-hill019-28.example.com (172.16.43.157) appears to be down.
Host ccf-hill019-29.example.com (172.16.43.158) appears to be down.
Host ccf-hill019-30.example.com (172.16.43.159) appears to be down.
Nmap finished: 31 IP addresses (7 hosts up) scanned in 0.892 seconds

As you can see, the results also (very usefully) include MAC addresses, so it’s pretty easy to update DHCP as needed.

Tech HowTos dhcp, nmap, ping, sysadmin

So, here’s the “official” scoop on the new project that I’m planning/starting to work on. I’m calling it PHPsa for now, and it’s going to (hopefully) be an integrated dashboard/portal for SysAdmins. While there are a number of tools that fit into this general category (perhaps with being the closest, though it’s security-minded), I feel that there’s a real gap in terms of tool integration. My daily workflow, which includes multiple trips to and correlation among Nagios, Cacti, DNS, DHCP, Puppet, logs, and other tools really leaves something to be desired. So, I’m setting out to create a modular SysAdmin dashboard that unifies many of the common SysAdmin-related tools into a modular dashboard.

The first overall design goals that I’ve set are:

1. A modular, plugin-based architecture that allows admins to select which features/tools they want, and allows easy development of new modules.
2. Design with legacy tools in mind – easy ways to tie in to tools that weren’t written with PHPsa in mind, both in terms of linking to information and gathering/unifying information.
3. RBAC, including per-module rules and the possibility for a limited read-only view (client/user mode).
4. Use of data sources, specifically databases, from existing tools with as little modification as possible.
5. Support for database abstraction, though I’ll be using MySQL.
6. Eventually, implement RSS feeds of pertinent information.
7. Balance Ajax/DHTML with the desire for important things to have canonical, static, bookmark-able URLs.

So, here are some of the things that I’m planning on integrating, with obvious bias towards getting my own projects done before I integrate pre-existing tools:

• MultiBindAdmin, my DNS and DHCP administration tool (specifically geared towards split-view DNS with the inside view behind NAT).
• RackMan, my tool for mapping devices’ physical locations in racks (and tacking patching).
• My simple config tool for Puppet.
• Nagios.
• Cacti.
• Nathan Hubbard’s MachDB.
• Bacula (monitoring/status only).
• Syslog via rsyslog (or any other syslog-to-SQL solution).
• Possibly a front-end to Google Analytics.
• Some of my custom scripts for graphing SpamAssassin, DNS queries, etc.
• Some sort of Apache log analysis, like Webalizer.
• Mail log analysis, possibly AWstats.

So, the first big issues that I’m going to tackle:

1. General layout. Specifically, how to handle a more-or-less consistent layout while integrating tools that weren’t designed for PHPsa. I’ll probably end up using iFrames (or even a frameset) for tools that don’t integrate well.
2. How to correlate data/objects between different tools (i.e. how to display information from Nagios, Cacti, MultiBindAdmin and MachDB for a given host?).
3. Do I want to use a templating engine like Smarty or hand-code all of the HTML?
4. How will I handle plugins?
5. How much code do I want to re-write and how much can I use as-is from other tools? And, on a related note, how much existing data can I access easily from other tools, vs having to use grabber scripts that dump data in MySQL?

Update 2010-02-03: I think this may become a semi-official project for me at $work, which means that I’ll be able to dedicate quite a bit more time to it. Unfortunately, it also means that I will, most likely, have to give up Nathan Hubbard’s MachDB in favor of OCS Inventory NG, a more mature project that already includes inventory support for Linux, Windows and Mac.

PHPsa, Projects bind, cacti, dns, Nagios, PHPsa, Projects, puppet, rsyslog, sysadmin, syslog

While I’ve read and really appreciate Tom Limoncelli’s Time Management for System Administrators, the current state of my life (mainly that it’s split between work, personal projects, a freelance client, administering the systems of a the ambulance corps and real people, and that my “work day” is whenever I’m awake) has prevented me from really implementing most of the advice. However, I do try to be as productive as I can.

Without getting into details, a few weeks ago, $WORK suffered a major electrical failure that required everything in the data center to be powered down. This happened around 10:30 AM, and the majority of groups simply powered down their machines and left, planning to return around 2 AM (the estimated power restoration time). After getting our machines down and stopping for pizza, I remembered how much of a pain it was to work in the racks bringing everything down. While my group only has two racks, we’ve had a lot of changeover lately, and the cabling had gotten quote messy. Noting this, I mentioned it to my two higher-ups, remembering that we had a stock of assorted length patch cables. We were able to make an “emergency” run to our cable vendor and pick up a box of 1- 2- and 3-foot power cables.

While everyone else was home or in their offices dodging the pieces of falling sky (everything was down including VoIP and mail), we were the only group getting real productive work done in the data center. The power failure, rather than a catastrophic event, was a great opportunity – the only time we could pull every cable in a production rack and re-do all power and patches.

So, here’s my SA tip for the day – everyone has some big projects that they’d like to do, require downtime, but aren’t critical enough to schedule something. So, keep a list of these and have the parts on hand. Whether it’s a “just in case” hardware swap-out, re-patching, or anything else, eventually you (depending on the environment that you work in) might have one of those times when the solution to the problem is out of your hands and there’s nothing else to do. Use this time productively.

Miscellaneous Geek Stuff sysadmin, time management