Jason Antman’s Blog

Here’s a little tidbit that I never knew until I had an Apache2 name-based virtual host problem: httpd -S lists the vhosts that are being served by Apache, and how they were parsed from the config files.

The output on one of my servers looks something like:

[root@web2 vhosts.d]# httpd -S
VirtualHost configuration:
wildcard NameVirtualHosts and _default_ servers:
_default_:443          web2.jasonantman.com (/etc/httpd/vhosts.d/ssl-host.conf:7)
*:80                   is a NameVirtualHost
         default server www.jasonantman.com (/etc/httpd/vhosts.d/000-default.conf:1)
         port 80 namevhost www.jasonantman.com (/etc/httpd/vhosts.d/000-default.conf:1)
         port 80 namevhost rackman.jasonantman.com (/etc/httpd/vhosts.d/rackman.jasonantman.com.conf:1)
         port 80 namevhost whatismyip.jasonantman.com (/etc/httpd/vhosts.d/whatismyip.jasonantman.com.conf:1)
Syntax OK

This is quite useful in debugging vhost problems, especially those pesky times when a request that should go to a specific vhost is being served by the default (in my case at this time, I had two ServerName directives instead of a ServerName and a ServerAlias).

Uncategorized apache, configuration, linux

At $WORK, my group runs about two dozen servers that provide services for over 60,000 users. They’re a mix of Windows and Linux, with some old Solaris stuff thrown in there. The one thing they have in common is they’re all hand-built, hand-configured, and old. They’ve been around for a while. At the moment, we don’t even have an adequate backup system.

So, being the closest thing to a SysAdmin we have (my official title is still Student Systems Programmer), it’s my job to build a new installation, configuration and backup infrastructure. We’ve already standardized on CentOS as a University-wide distro, and have a local full mirror, so I don’t need to choose a distro. I do, however, have to plan the installation and backup architecture. The main requirements are:

1. Lowest overall time for bare-metal recovery to a working system.
2. Ease of use, as people other than myself will need to administer it (so they should be able to do so from a cheat sheet in the wiki).
3. Repeatability – it should be easy and intuitive to make an almost-exact-copy of a machine.

I started a thread a few days ago on the SAGE mailing list, which you can find here.

At the moment, it looks like the general idea that I’m going with is to use Kickstart to install the systems, using a basic and minimal Kickstart file. Basic package selection (minimalist) with just what’s needed to configure the system with a hostname and network settings for the management VLAN. I’ll then have Kickstart install and configure a configuration management package – I’m leaning towards Puppet over Cfengine and am starting testing. The config management software will handle all of the customization for the system (everything different from the base generic Kickstart install) so it’s all kept under the control of config management from step 1.

The final part is a backup system, mainly for whatever eventually – whether out of human error or simple laziness – ends up out of the config management system’s control. Our previous SA had settled on Zmanda, the paid version of Amanda, which comes with specific plugins for MySQL and MSSQL. I’m also looking at Bacula, mainly because of its’ advanced features, scheduling (especially the new scheduling in Bacula 3) and scalability.

The beauty that I see in having Kickstart do something minimal and then letting Puppet handle the rest is that (especially since we’ve standardized on SunFire X4100’s with identical configurations) I can kickstart and rack up a few spare machines, and to get them up and running all I need to do is power them up (iLOM) and tell Puppet what to make them.

I’m currently starting testing of both Puppet itself and getting Kickstart to start the puppet install and daemon (instructions from David Lutterkort’s blog (Red Hat software engineer)). We’ll see how everything goes…

Projects backup, configuration, kickstart, linux, puppet, recovery, work

On the same thread as the last post, some thoughts on my ideal network, or the hosts on that network:

• One “gold master” installation/kickstart file of a single chosen distro, with a base set of packages, including site-specific packages. (or something like this implemented in a configuration management system)
• All new installations performed over the network in an automated fashion, and from a local repository.
• Software updates are automated (either through a configuration management tool or something like the behemoth I talked about here and pulled from a local repository (perhaps one which mirrors the mainline repos, but only downloads a package the first time it’s requested?).
• Puppet or CFengine on each machine. Better than just having them is having each machine automatically added when it’s created. Even better yet would be to have Puppet or CFengine combined with something like Cobbler, so I can define a new machine in {puppet|cfengine}, list its’ MAC address, then netboot the box and come back in a few hours to have an OS installed, packages installed and the machine configured, monitored in Nagios, monitored for security and backed up.
• Tripwire or some other sort of security software, as well as centralized logging and auditing, on every box.
• A small number of additional “package groups” to add to the “gold master” via config management – something like “web server” (Apache2, PHP, MySQL, log analysis for them, etc.), “development server” (CVS, debuggers, etc.). These would also update the backup system to include appropriate directories, update Nagios configs, etc.
• A good way – if even a human making notes in a per-machine text file – of tracking the “little stuff” like that one cron script that makes everything work right, the location of that hacked-together Python script, etc. A way to easily remember the things needed to recreate a box which aren’t found in rpm -qa or any obvious overviews.
• Bacula or AMANDA setup to backup every box, perhaps with some sort of template system for server types – every machine gets /root and /etc backed up, but web servers get /srv/www and mail servers get /var/mail.
• Nagios setup to monitor everything logical on every box. Perhaps this would use a configuration management engine to handle Nagios configs, so that for example if any Proliant hardware is used, {config management program} will figure this out, install the HPASM packages, put the appropriate check scripts on the box, and update the Nagios configs. Likewise, adding Apache to a machine should cause it to be monitored in the Nagios configs.

Unfortunately, as I’m not independently wealthy, I don’t have the time to quit my job, wipe every machine I own, and start from scratch. But it sure would be nice to be able to, one day, start a server farm from scratch and be able to implement some of these cool things…

Ideas and Rants configuration, dream network, ideal network, management

After a few crazy weeks at work, the pay check is finally here, and I’ve decided to allocate some of the money to hardware upgrades of my personal machines. While Nagios gives me a good idea of where performance is a problem, I’m still running a large amount of “legacy” hardware (my home router/firewall is a 350MHz P-II desktop) and hardware age is a significant factor in my upgrade plans.

So, I set out looking for a program (most likely some backend scripts that dump data to MySQL, and then a PHP front-end) to perform a hardware inventory – essentially, run a script on each box, find out the system details, and dump it in a DB. Now that my pool of upgrade candidates is above a dozen machines, at multiple locations, some of which are single-use boxes often neglected/forgotten, doing this in my head isn’t the easiest.

So, while I’ve been googling and searching some mailing list archives, I’m also developing a set of requirements.

The chief requirements:

• Ability to run a script on a remote machine and have the results returned in a meaningful format. Most likely, a single script, run as root (SUID or sudo) that returns nice, formatted, SQL-ready results (so the parsing of platform-specific command output will happen on the client, with every client returning a normalized data set).
• Ability to track hardware changes – i.e. disks swapped from one box to another, NIC replacement, processor upgrade, etc.
• Each piece of hardware tracked individually, allowing future support of fully tracking components, spares, etc.
• Support for future barcoding of components and physically-performed inventory.

Some of the data that I’d like collected:

• Data on a machine stored by chassis vendor name and serial numer/service tag.
• Hostname associated with each box.
• Architecture, number of CPUs/cores and type, model, speed, socket (for upgrade planning/ordering).
• Motherboard model/manufacturer, serial number, HW and SW/FW revisions, manufacture date.
• Storage (internal & directly connected external) – type, interface, capacity, specifications/manufacturer and model.
• Amount fo RAM, type of RAM, and configuration of cards (number of empty slots).
• PCI card configuration – number and type/mfr/model of cards, as well as number of empty slots
• For all NICs – MAC addresses, current IPs, as well as link type/speed and connected switch/port.

I’d also like some level of software inventory, especially for those machines that may be running “forgotten” services:

• nmap scan results.
• OS-generated list of running services, with GUI including a blacklist of “default” services not to be displayed, and possibly also cross-links to Nagios status.
• Possibly a parsed output from ps, using a blacklist as shown above.
• If a firewall is running on the system, a list of all open ports.

I’ll be checking out some options today. Unfortunately, I have a feeling that there’s most likely nothing that supports my requirements, and I’ll probably end up implementing a lot of this myself.

Projects configuration, inventory