Vyatta – Showing ISC dhcpd fixed-address leases

Posted on December 24, 2011 by Jason Antman

ISC dhcpd has the ability to always give a specific MAC address the same IP address “lease” using the fixed-address configuration option. This is configured in Vyatta using the static-mapping configuration statement. Unfortunately, since dhcpd doesn’t store fixed-address leases in the dhcpd.leases file, the Vyatta show dhcp leases command doesn’t show anything about them – which makes it difficult to debug anything dhcp-related if all of the hosts on your network are setup for fixed addresses. I found a mention of this on the vyatta forum, and also an open bug (1990) to fix it, but no proposed resolution. Since I came across this problem and happen to know a bit about ISC dhcpd, I developed both a workaround for users (including a perl script) and a possible solution for the Vyatta developers to implement.

Workaround for Users:

There’s no way to get dhcpd to store fixed-address hosts in the dhcpd.leases file (though it’s been discussed on the dhcpd-users mailing list a few times). There is, however, a way to get dhcpd to log every time it sends an ACK to a client. The following Vyatta configuration commands will get dhcpd to log all transactions to syslog, will have rsyslog put that in /var/log/user/dhcpd. Since this log can fill up very quickly on a busy server, the latter two commands will tell logrotate to rotate the log file when it reaches 3000k in size, and keep 5 copies (feel free to adjust to your needs):

set service dhcp-server global-parameters "log-facility local2;"
set system syslog file dhcpd facility local2 level debug
set system syslog file dhcpd archive files 5
set system syslog file dhcpd archive size 3000

Once this is done, you can tail -f /var/log/user/dhcpd to watch DHCP discover/request/offer/ack in realtime, or grep through the log file for a specific IP or MAC. If you want an easier method, I’ve written a perl script (latest version will always live in my subversion repo) to grep through /var/log/user/dhcpd and show the most recent DHCPACK for each IP address, sorted by IP. Here’s the code of the simple script, which is more than half comments. To use it, after performing the above steps, all you need to do is login to your Vyatta box, wget http://svn.jasonantman.com/misc-scripts/show_dhcp_fixed_ACKs.pl and then perl show_dhcp_fixed_ACKs.pl.

#!/usr/bin/perl
 
#
# show_dhcp_fixed_ACKs.pl - script to show the most recent DHCP ACKs per IP address for ISC DHCPd,
#   from a log file. Originally written for Vyatta routers that just show the dynamic leases.
#
# To use this, you need to have dhcpd logging to syslog, and your syslog server putting the log file at
# /var/log/user/dhcpd (or a file path specified by the $logfile variable below.
#
# To accomplish this on Vyatta 6.3, run:
# set service dhcp-server global-parameters "log-facility local2;"
# set system syslog file dhcpd facility local2 level debug
# set system syslog file dhcpd archive files 5
# set system syslog file dhcpd archive size 3000
# commit
#
# Copyright 2011 Jason Antman <jason@jasonantman.com> All Rights Reserved.
# This script is free for use by anyone anywhere, provided that you comply with the following terms:
# 1) Keep this notice and copyright statement intact.
# 2) Send any substantial changes, improvements or bog fixes back to me at the above address.
# 3) If you include this in a product or redistribute it, you notify me, and include my name in the credits or changelog.
#
# The following URL always points to the newest version of this script. If you obtained it from another source, you should
# check here:
# $HeadURL$
# $LastChangedRevision$
#
# CHANGELOG:
# 2011-12-24 jason@jasonantman.com:
#    initial version of script
#
#
 
use strict;
use warnings;
 
my $logfile = "/var/log/user/dhcpd";
 
my %data = ();
 
open DF, $logfile or die $!;
while ( my $line = <DF> ) {
    if ( $line !~ m/dhcpd: DHCPACK/) { next;}
    $line =~ m/([A-Za-z]+ [0-9]+ [0-9]{1,2}:[0-9]{2}:[0-9]{2}) [^\/x]+ dhcpd: DHCPACK on (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) to ((?:[0-9a-f]{2}[:-]){5}[0-9a-f]{2}) via (.+)/;
    #print "$1==$2==$3==$4==\n" ;
    $data{"$2"}->{'mac'} = "$3";
    $data{"$2"}->{'date'} = "$1";
    $data{"$2"}->{'if'} = "$4";
    $data{"$2"}->{'ip'} = "$2";    
}
 
printf("%-18s %-20s %-18s %-10s\n", "IP Address", "Hardware Address", "Date", "Interface");
printf("%-18s %-20s %-18s %-10s\n", "----------", "----------------", "----", "---------");
 
# begin sort by IP address
my @keys =
  map  substr($_, 4) =>
  sort
  map  pack('C4' =>
    /(\d+)\.(\d+)\.(\d+)\.(\d+)/)
    . $_ => (keys %data);
# end sort by IP address
 
foreach my $key (@keys) {
    printf("%-18s %-20s %-18s %-10s\n", $data{$key}{'ip'}, $data{$key}{'mac'}, $data{$key}{'date'}, $data{$key}{'if'});
}

A solution for Vyatta:

I suggested this to Vyatta in a reply to bug 1990. Since they already use rsyslog which has very powerful processing capabilities, it would be easy to have rsyslog parse the DHCPACK messages in real time and update some data store (flat files or a simple database) with the information. While how to store this would be up to the Vyatta guys, I have some rsyslog configuration to parse DHCPACK messages and update a MySQL database (with two tables; one for most recent ACK per IP address and one for most recent ACK per MAC address) that might be of some use:

$template DHCPACKonIP, "INSERT INTO dhcplog_ip \
        SET \
        server_ip=inet_aton('%fromhost-ip%'), \
        msg_type='DHCPACK', \
        date='%timereported:::date-mysql%', \
        mac_addr='%msg:R,ERE,2,BLANK:DHCPACK on ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) to (([0-9a-f]{2}:){5}[0-9a-f]{2})( \(([^)]+)\))? via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        client_ip=inet_aton('%msg:R,ERE,1,BLANK:DHCPACK on ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) to (([0-9a-f]{2}:){5}[0-9a-f]{2})( \(([^)]+)\))? via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%'), \
        gateway='%msg:R,ERE,6,BLANK:DHCPACK on ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) to (([0-9a-f]{2}:){5}[0-9a-f]{2})( \(([^)]+)\))? via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        fullmsg='%msg%' \
        ON DUPLICATE KEY UPDATE \
        server_ip=inet_aton('%fromhost-ip%'), \
        date='%timereported:::date-mysql%', \
        mac_addr='%msg:R,ERE,2,BLANK:DHCPACK on ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) to (([0-9a-f]{2}:){5}[0-9a-f]{2})( \(([^)]+)\))? via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        gateway='%msg:R,ERE,6,BLANK:DHCPACK on ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) to (([0-9a-f]{2}:){5}[0-9a-f]{2})( \(([^)]+)\))? via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        fullmsg='%msg%'\
",SQL
 
$template DHCPACKonMAC, "INSERT INTO dhcplog_mac \
        SET \
        server_ip=inet_aton('%fromhost-ip%'), \
        msg_type='DHCPACK', \
        date='%timereported:::date-mysql%', \
        mac_addr='%msg:R,ERE,2,BLANK:DHCPACK on ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) to (([0-9a-f]{2}:){5}[0-9a-f]{2})( \(([^)]+)\))? via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        client_ip=inet_aton('%msg:R,ERE,1,BLANK:DHCPACK on ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) to (([0-9a-f]{2}:){5}[0-9a-f]{2})( \(([^)]+)\))? via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%'), \
        gateway='%msg:R,ERE,6,BLANK:DHCPACK on ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) to (([0-9a-f]{2}:){5}[0-9a-f]{2})( \(([^)]+)\))? via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        fullmsg='%msg%' \
        ON DUPLICATE KEY UPDATE \
        server_ip=inet_aton('%fromhost-ip%'), \
        date='%timereported:::date-mysql%', \
        client_ip=inet_aton('%msg:R,ERE,1,BLANK:DHCPACK on ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) to (([0-9a-f]{2}:){5}[0-9a-f]{2})( \(([^)]+)\))? via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%'), \
        gateway='%msg:R,ERE,6,BLANK:DHCPACK on ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) to (([0-9a-f]{2}:){5}[0-9a-f]{2})( \(([^)]+)\))? via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        fullmsg='%msg%'\
",SQL
 
$template DHCPACKtoIP, "INSERT INTO dhcplog_ip \
        SET \
        server_ip=inet_aton('%fromhost-ip%'), \
        msg_type='DHCPACK', \
        date='%timereported:::date-mysql%', \
        mac_addr='%msg:R,ERE,2,BLANK:DHCPACK to ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) \((([0-9a-f]{2}:){5}[0-9a-f]{2})\) via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        client_ip=inet_aton('%msg:R,ERE,1,BLANK:DHCPACK to ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) \((([0-9a-f]{2}:){5}[0-9a-f]{2})\) via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%'), \
        gateway='%msg:R,ERE,4,BLANK:DHCPACK to ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) \((([0-9a-f]{2}:){5}[0-9a-f]{2})\) via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        fullmsg='%msg%' \
        ON DUPLICATE KEY UPDATE \
        server_ip=inet_aton('%fromhost-ip%'), \
        date='%timereported:::date-mysql%', \
        mac_addr='%msg:R,ERE,2,BLANK:DHCPACK to ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) \((([0-9a-f]{2}:){5}[0-9a-f]{2})\) via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        gateway='%msg:R,ERE,4,BLANK:DHCPACK to ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) \((([0-9a-f]{2}:){5}[0-9a-f]{2})\) via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        fullmsg='%msg%'\
",SQL
 
$template DHCPACKtoMAC, "INSERT INTO dhcplog_mac \
        SET \
        server_ip=inet_aton('%fromhost-ip%'), \
        msg_type='DHCPACK', \
        date='%timereported:::date-mysql%', \
        mac_addr='%msg:R,ERE,2,BLANK:DHCPACK to ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) \((([0-9a-f]{2}:){5}[0-9a-f]{2})\) via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        client_ip=inet_aton('%msg:R,ERE,1,BLANK:DHCPACK to ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) \((([0-9a-f]{2}:){5}[0-9a-f]{2})\) via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%'), \
        gateway='%msg:R,ERE,4,BLANK:DHCPACK to ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) \((([0-9a-f]{2}:){5}[0-9a-f]{2})\) via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        fullmsg='%msg%' \
        ON DUPLICATE KEY UPDATE \
        server_ip=inet_aton('%fromhost-ip%'), \
        date='%timereported:::date-mysql%', \
        client_ip=inet_aton('%msg:R,ERE,1,BLANK:DHCPACK to ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) \((([0-9a-f]{2}:){5}[0-9a-f]{2})\) via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%'), \
        gateway='%msg:R,ERE,4,BLANK:DHCPACK to ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}) \((([0-9a-f]{2}:){5}[0-9a-f]{2})\) via (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})|eth0)--end%', \
        fullmsg='%msg%'\
",SQL
 
:msg, startswith, " DHCPACK on" :ommysql:hostname,database,dbuser,dbpass;DHCPACKonIP
& :ommysql:hostname,database,dbuser,dbpass;DHCPACKonMAC
& ~ ### DISCARD
 
if $msg startswith ' DHCPACK to' and ( not ( $msg contains 'no client hardware address' ) ) \
then :ommysql:hostname,database,dbuser,dbpass;DHCPACKtoMAC
& :ommysql:hostname,database,dbuser,dbpass;DHCPACKtoIP
& ~ ### DISCARD

Vyatta NetworkOS router/firewall on Alix board / Compact Flash

Posted on September 30, 2011 by Jason Antman

With the impending move to an apartment in Georgia and the migration of my rack full of servers to a hosting provider, there’s no longer a need for me to run my Vyatta VC router on a beefy dual-CPU RAIDed DL360 G3 HP Proliant server chassis. I found an older PCEngines Alix 2c1 single board computer (433 MHz AMD Geode LX700 , 128MB DDR DRAM, CompactFlash (CF) card socket, MiniPCI, 3x 10/100 ethernet) lying around, and decided to turn that into the new router. But I’ve been so spoiled by Vyatta’s good performance (well, at least on an x86 server) and the real CLI, so I don’t think I can go back to something like m0n0wall or pfSense, and since it’s going to be my only network services box (also doing DNS, DHCP, firewalling, NAT, and maybe IPsec VPN) it’s not viable to use the type of older Cisco or Juniper hardware that I can afford.

The down side is that Vyatta isn’t really designed or tuned for small systems, let alone CF media that doesn’t take too well to lots of writes. So, I’m going to begin experimentation with doing a CF install of the current Vyatta Core 6.3, and we’ll see how it goes and what tuning I do over time.

I found two relatively good references; a post on the vyatta.org forum from 2008, relating to Vyatta version 4 (also on the author’s blog), and a blog post detailing a more complex SquashFS/tmpfs/UnionFS read-only Vyatta install. Given my relatively short timeframe and little free time, I decided to try the former approach for now, and plan to make a more customized and tuned CF version of Vyatta in the future.

Creating the actual disk image:

My development platform at the moment is an intel-based MacBook Pro, running MacOS X 10.6.4 and VirtualBox 4.0.12. As much of a Linux fan as I am, my work laptop runs Mac (like everyone else in the office) and lately I can’t guarantee that I’ll be at my desktop long enough to finish anything. The target is an Alix2c1 with a 2GB SanDisk Ultra CF card (yes, I know an industrial card would be better, but I couldn’t get my hands on one). For starters, I created a new VirtualBox VM with the following settings:

OS Type: Linux 2.6
Base Memory: 128MB
Boot Order: Floppy, CD-ROM, Hard Disk
IDE Controller Primary: mounted vyatta-livecd_VC6.3 ISO image
IDE Controller Secondary: RAW VMDK image (created below)
Audio: None
Network: Disabled (this is important, as Vyatta saves the interfaces by hardware address, and it would require some config editing and reboots to change them)
Serial Port: disconnected (but present)

One difficulty I ran into on Mac is mounting the raw CF card in the VirtualBox guest. I plugged it in via a USB reader, and of course it automatically mounted in MacOS. I ejected it and the /dev/disk1 device disappeared. It turns out that the full procedure (as far as I could tell) for Mac is:

Plug in the CF card and reader.
It should automount. Run mount to see what the actual device is – in my case, the /dev/disk1s1 partition was mounted, so the disk is /dev/disk1.
Run sudo umount -f /dev/disk1. It seems that the MacOS automounter has a god complex, so you may need to re-run this command quite a few times throughout the process if you get device or resource busy errors.
In an appropriate directory, create the raw VMDK image with: VBoxManage internalcommands createrawvmdk -filename rawdisk.vmdk -rawdisk /dev/disk1.
When creating your VM, you’ll have an option to select Use an Existing Virtual Disk. Use that option, and select the file created in the last step.

Once that’s done, and you’ve setup the VM with the raw disk, boot the VM (should boot to the Vyatta LiveCD), login as usual for an install (vyatta:vyatta), and the the fun begins:

At the prompt after logging in, sudo su -
Edit /opt/vyatta/sbin/install-system (hint: Vyatta has nano and vi installed. nano -c filename shows line numbers) and change the ROOT_FSTYPE variable (line 78 in VC6.3) from “ext4″ to “ext2″.
Run install-system. I used all default options (including one partition) and it seemed to work fine. It took a minute or two to create the ext2 filesystem on my 2GB CF card.
The file copy took even longer… so be patient, or have a book handy.
When system-install finishes and you get the root prompt back, before rebooting, continue with some minor tweaks:
mkdir /mnt/temp
mount /dev/sda1 /mnt/temp
cd /mnt/temp
Edit boot/grub/grub.cfg and change all occurrences of “root=UUID=…” entries for the “linux” lines (lines 13, 18, 23, 28 in my grub.cfg) to “root=/dev/sda1″. My only real reason for this change is so that I can move my altered config files (config.boot, fstab and grub.cfg) with minimal changes when I upgrade or make a different vyatta CF card, without having to update the UUID for the new partition.
Edit etc/fstab and change the “UUID=…” device to “/dev/sda1″.
shutdown. Once the VM is stopped, you can remove the CF card.
The PCEngines Alix.2 boards use a default serial console speed of 38400 baud. Pretty much every network device, plus Linux and Vyatta, use a default speed of 9600 baud. Once I got the CF card installed in the Alix board and hooked it up to my laptop (null modem cable to a PL-2303 based USB to serial adapter, minicom for terminal emulation), I set my terminal emulator to 38400 8N1, powered the board, and then pressed ‘s’ during POST to get into BIOS settings. Option ’9′ sets the Alix to 9600 baud, ‘Q’ to quit, and ‘Y’ to save changes permanently. The board will reboot, and once the terminal emulator is set back to 9600 baud, serial console should work fine both in BIOS and in the OS.

If all worked well, you should be able to boot into Vyatta and login as the default “vyatta” user (which you set a password for during the install). Assuming you know your way around Vyatta, it’s pretty standard from here, though there are a few things you may want to check or configure right away:

In configuration mode (configure) run show interfaces. All of your physical ethernet interfaces should appear, along with their MAC addresses.
Some changes to reduce the number of log writes to the CF card: set system syslog console facility all level notice and set system syslog global facility protocols level notice.
Configure interfaces. with firewalls, IP addresses or DHCP, etc.
Do whatever other configuration you need for a minimal system – dhcp, dns, nat, etc.

And that’s it – this should give you a working Vyatta system on CF on an Alix board. Stay tuned, hopefully in a month or so I’ll get around to customizing it a bit more, based on the second blog entry linked above.

Vyatta VC5 – Snort alerts to syslog

Posted on January 30, 2011 by Jason Antman

I’m running a Vyatta vyatta router at home – in my opinion it’s pretty near “enterprise grade”, and I’m running the Community/Core (read: no-cost and almost all Free) on commodity hardware with great performance. Granted, I’m still on the older version (5 as opposed to the current 6) since an upgrade will require total downtime and a spare set of SCSI disks for the machine it’s on, but it still works quite well. Today I decided to enable the Snort IDS on the box. It actually worked quite well (albeit stuck at older rules until I upgrade to Vyatta VC6) and didn’t put much more load on the box. For the time being I decided to just have it alert on problems and not drop anything, as I’m getting pretty high false positives and the older Vyatta version doesn’t seem to have a way to disregard rules.

My biggest complaint was that Vyatta didn’t see fit to allow alerts by syslog. I’m not a big fan of keeping information like IDS logs stuck on the router – I don’t like logging in to it any more than I have to, it doesn’t have much storage, and I’d also like to keep stuff like this in a more secure location. Through a bit of digging, I found the /opt/vyatta/share/perl5/VyattaSnortConfig.pm Perl module which generates the Snort config file from Vyatta’s CLI stuff. Looking through the Perl code, I found the definition of the snort.conf output statements:

my $output_def =<<EOD;
  output alert_unified: alert, limit $log_limit
  output log_null
EOD

I simply added a line output alert_syslog: log_local3 log_notice after the output alert_unified line.

my $output_def =<<EOD;
  output alert_unified: alert, limit $log_limit
  output alert_syslog: log_local3 log_notice
  output log_null
EOD

I then went into Vyatta configuration and changed a rule from alert to pass, committed, changed back, committed, and /etc/snort/snort.conf now had my syslog lines. I’m now getting snort alerts to local3 in syslog, which is fed to my centralized log server. My next project is to find or write something which will parse these logs, generate a daily summary email, and maybe check them hourly and alert on any big changes. I also might just cron an emailing of the output from Vyatta’s show ips summary command. So far I have over 11,000 events logged in about 12 hours.

Daily Work – Nagios SNMP traps, Vyatta, JasonAntman.com upgrades

Posted on June 27, 2009 by Jason Antman

So it’s been a very busy day. I was up until 5 AM or so working on implementing Puppet at home. I’m building two new boxes – a storage (centralized home directory)/syslog (to MySQL) server and a second web server (possibly also to handle Nagios) – and I decided that they’ll be totally built by Puppet. The only thing I had to give up on was setting up the NFS share for my home directory on the new storage box and installing and testing rsyslog on it.

This afternoon around 7, I started on my weekend projects for the ambulance corps – setting up Nagios to receive SNMP traps from the APC UPS and moving over to the new Vyatta-based router (from m0n0wall). I’d attempted the router before, but had to rollback – I’m using an old BlueSocket controller for hardware – it’s just a nice black 1U enclosure with a stock Intel motherboard, 20GB HDD, 512MB RAM and three 10/100 NICs. The first time, I was unable to get link on either of the two NICs I was using, so I decided to rollback.

Nagios SNMP Traps

I found a good starting point for Nagios SNMP traps on the OpsView blog. I setup `snmptrapd` on the Nagios server and hacked together a little Python script to just write all of the traps to a file. After some testing with `snmptrap` on my laptop, I did a test by pulling the power plug of the UPS, waiting about 30 seconds, and then plugging it back in. Sure enough, the little old AP9605 PowerNet SNMP card generated two SNMP traps – one for power loss and one for power regained – both of which showed up in the test file

The next step will be deciding how to get the traps into Nagios – specifically whether I want to go with something heavy-weight, like SNMPtt that can handle other devices, or whether I want to code a simple script myself just to deal with the APC cards.

Router

The main reason why I wanted to make the switch from m0n0 to Vyatta was to ease the setup and maintenance of an IPsec tunnel from the ambulance HQ to my house, so I could push backups (relatively small) over the WAN to my infrastructure (or, rather, have Bacula pull the backups). Another big bonus was finally having a way of configuring and checking things through SSH without having to port-forward a web GUI. Another bonus of having a real Linux system under the router is the ability to make custom Nagios check scripts and easily execute them. Something I hadn’t thought of – but became obvious during the switchover – is the ability to run full-fledged `tcpdump` on the router itself.

After building the new config myself, and confirming that the system ran in isolation, I moved it over to production. The first issue was a bit of a thinko on my part – the interfaces on the BSC are actually arranged on the back of the box like eth0—–eth2—–eth1, so I originally had the LAN uplink in the wrong interface. After correcting that and waiting for the network to stabilize, I noticed a total external connectivity failure. After some troubleshooting – thanks to tcpdump on the router – it occurred to me that the (ancient) cable modem needs to be rebooted when the router MAC changes.

I honestly don’t remember the other problems that I ran into, but eventually I ended up getting almost-full functionality – and then a total network outage. A tcpdump on my laptop showed some really really weird BOOTP traffic with addresses of 255.255.255.255. After doing some troubleshooting and monitoring port counters on the switch, I narrowed it down to coming from a single Windows box and the wireless access point. After shutting off both ports, things seemed to stabilize. I also had some “martian address” issues with one of the boxes, but decided to roll the box and that solved it.

Over the next day or so, I’ll be reconfiguring Nagios both at home and at the ambulance corps to cope with the changes and add in the requisite monitoring, and keep an eye on things. Assuming all goes well, I’ll power down the old router on Sunday.

On the home front, I’ve moved over from my old storage machine to the old one – essentially just the NFS mount, and moved over a tarball of everything else. I also added a 1000Base-SX card to the new box, though it appears that I’m out of fiber patch cords. The old storage box was brought down for the first time in about 3 years (aside from brief outages for hardware upgrades or array rebuilds). Assuming I got everything off of it, it will be relegated to the spares pile.

I’m going to make a serious effort to post on a daily basis, if only for my own future reference. I should have the demo of RackMan out soon, and I’m also about to start on integrating it with Nathan Hubbard’s MachDB as well as a PHP script I wrote to pull port names and MACs from Cisco switches and associate them with NICs in machines. Hopefully I’ll also have some interesting Puppet stuff out soon.

Big Changes to JasonAntman.com

Posted on March 5, 2009 by Jason Antman

Well, I finally broke down and ordered Optimum Business. Come tomorrow, I’ll be moving from Verizon FiOS residential with a dynamic IP, much blocked (hence jantman.dyndns.org:10011) and 10Mbps down/2Mbps up to Optimum Business with 30 down/5 up, a block of 5 static IPs, and no blocked ports.

It’s going to be a crazy weekend. Probably not the best thing the week before midterms, but oh well. Tomorrow morning I’m picking up a 42U rack for home to replace the Sears shelving unit my boxes are currently on. Cablevision is supposed to be here between 2-5 PM to do the install (yes, they insist that for Business they do the install, even though it’s only a 4-foot coax run from the first splitter to the demarc). I’ve got Vyatta CE5 Beta installed on a Proliant DL360G2 as the new router, ready to go (after some configuration). I’ll probably keep FiOS up until I know the new router is working correctly (I’ll do a test on my management VLAN).

Once Optimum and the new router is up, the fun starts:

Forward the appropriate ports on the new router, including 80 (in addition to 10011).
Bring the old router down and make sure the new one is up, operational, and forwarding all the right ports.
Update DynDNS to point to the first IP, used as a catch-all for old DynDNS links.
Begin assignment of the 5 IPs (everything will be behind NAT) based on a list of what hosts need valid reverse DNS, and then adding other ports (NATed) as needed.
Update DNS for JasonAntman.com and the other domains.
Update Optimum reverse DNS.
Ensure that everything works as planned, DNS is up, ports are forwarded, and everything is as before (at least in terms of HTTP).
Once DNS is up, reconfigure Apache to have a vhost handling any legacy requests to port 10011 and rewrite them to www.jasonantman.com.
Setup a vhost for ‘www’ that takes URLs that used to be subdirectories (i.e. www.jasonantman.com/blog) and rewrites them to requests for the appropriate subdomain. Simultaneously move everything from the default vhost to name-based vhosts.
Ensure that old jantman.dyndns.org:10011 requests are being redirected properly, and requests for subdirectories under the web root are going to the right subdomain.
Check that this all works acceptably with the existing blogger-to-wordpress rewrite script.
Finally start rolling out some of the new services that I had waiting for the new connection.
Start the arduous process of reconfiguring my mail server, moving from Fetchmail from Verizon to an actual mail server, make everything work, and make sure my IPs aren’t blacklisted.
Ugh. Find anywhere in the entire ‘net where my old @verizon.net address appeared (especially GoDaddy, DynDNS, other important stuff) and change it to the new jasonantman.com address.
Since this is all in my mother’s basement (there’s nothing like a mother’s love, especially when it comes to a constant hum emanating from the ground level of a house), figure out what to do for her when the verizon.net email goes away.

So I might have some downtime this weekend, but when things come back up, I’ll be done with this DynDNS and Port 10011 crap.

Vyatta Initial Impressions

Posted on February 26, 2009 by Jason Antman

I’m part-way through the major overhaul of my home network (hosting this blog and everything else jasonantman.com) that I’ve been planning for quite some time. The current hardware is… uh… currently… described on my Hardware page, but I soon plan on ditching the wiki and moving to a CMS for my entire site.

Anyway, so far I’ve decommissioned my aged HP ProCurve 2424M switch and replaced it with used but less-aged Cisco 2948G from Horizon Datacom (purchased on Ebay). Quite an upgrade. In order to handle network backups a little better, I’m also adding a Cisco 4912G 12-port Gigabit (GBIC) aggregation switch for the administrative/backup VLAN – though this was purchased via ebay from RedApe Technologies in PA. The switch came with 12 1000BASE-SX GBICs, and I plan to do a mix of copper (1000BASE-T) where it’s already available (onboard NICs) and 1000BASE-SX where there’s enough room in the box for a card.

On the hardware side, I also have 2 new boxes – a set of HP Proliant DL360 G2‘s from MJS Global, who I’ve done business with before. The prices were great, and though one of them showed up with a faulty temperature sensor that prevents boot, MJS has been wonderful and is shipping me a replacement motherboard. One of the boxes will be running Vyatta (vee-AH-tha) VC5 router/firewall software, and the other will be a new services box running internal DNS, DHCP, NTP, and whatever else.

On the hardware side, I’m also planning some extended downtime a few weekends from now, when I should finally have a 42U rack to replace the Sears shelves my equipment is now on. It’ll be a fun-filled evening of racking equipment and re-patching everything. Also, hopefully within a few weeks, I’ll be moving my WAN pipe from Verizon FiOS residential to Optimum Business, which is essentially re-packaged residential but provides 5 static IPs, no blocked ports, and 30 Mbps down/5 Mbps up.

Vyatta

When planning this upgrade, I think I looked at every open source router package out there, as well as some of the lower-end or older Cisco models. I’m currently running IPcop, which does everything I need except it doesn’t handle multiple WAN IPs, and all configuration is via a web interface – which means every time I want to make a change remotely (and during the week I’m not home) I have to forward HTTPS over SSH. After doing an extensive feature comparison, I ended up narrowing it down to a relative newcomer – Vyatta. Though I don’t know how much of it is marketing hype, they are targeted squarely at Cisco, and provide relatively enterprise-level features; a JunOS-based CLI, BGP, OSPF, and all of the other important stuff.

Yesterday I attempted an install of Vyatta CE 5 Beta on one of the DL360G2′s. The only real problem that I found was the install script doesn’t support CCISS drives, as found in the Proliants, but a few manual hacks to the script fixed that. By far the best thing about Vyatta is it’s based on vanilla Debian Lenny, and full root shell access is available, so modifying the install script – or even adding non-Vyatta packages – is a cinch. I haven’t really played around with it too much, but it appears to be a wonderful mix of Linux and an enterprise router CLI. While root has a full BASH shell, and the Vyata commands are all done as shell aliases (so users still have access to shell primitives and OS commands), configuration is accomplished via a JunOS-like command set. You still get “commit” and “rollback” in config mode, and can still do fun things like save and load configs to/from tftp, ftp and http. On the other hand, I doubt I’ll do config backups that way since I can just use scp or sftp.

The Vyatta box will probably go home this weekend, and get hooked up to the network for config-only use (and I can always get in via iLO on the hardware) and hopefully come up sometime in the next few weeks.

At this point, the most daunting task is figuring out how to get all of the existing links to my site to work – since jantman.dyndns.org will be legacy, and most of the site structure will probably change to use name-based vhosts. Lately I’ve been trying to use the real subdomains in all of my public links, so the transition (planned for a while) will work, but I’m sure there are still plenty of links out there that will need dealing with (maybe keep port 10011 serving HTTP with a massive mod_rewrite script to redirect to the right place???), as well as checking everything on the web server to make sure there aren’t any absolute URLs (like WordPress).

Jason Antman's Blog

Information, thoughts, tips and tricks from a professional system administrator / monitoring and automation nut, open source junkie, troubleshooter, sometime software developer and ceaseless tinkerer. And some occasional commentary on my hobbies and non-tech interests.

Tag Archives: vyatta