Puppet facter facts for syslog daemon type and version, symantec netbackup

Posted on August 25, 2012 by Jason Antman

I have a few more custom facts that I’ve added to my puppet-facter-facts github repository:

syslog_bin, syslog_type, and syslog_version – tell the absolute path to the running syslog binary, its short name (basename), and its version as a string. Currently only know about /sbin/syslogd and /sbin/rsyslogd.
has_netbackup – tests for presence of the /usr/openv/netbackup/bin directory, created by installation of Symantec Netbackup. Useful for making generation of include/exclude files conditional on having NetBackup installed.

Hopefully some of these will be of use to someone else as well.

Puppet facter fact for all applied classes, returned as a CSV list

Posted on August 22, 2012 by Jason Antman

I’m unfortunatey stuck, at least for the time being, using flat-file manifests to configure my puppet nodes. Without an ENC, it’s pretty difficult to get a good ovewview of what classes are used on each node, and what nodes use a given class. I know I could write up a simple web tool to do this (unfortunately, given my limited Ruby knowledge, it would have to be in PHP or Perl, not a real modification to Dashboard in Ruby). But where to get the data from?

After some research, I found a puppet fact for puppet classes on Matthew Nicholson’s Coffee & Beer blog. It parses /var/lib/puppet/classes.txt and returns the list of classes found as a JSON array. Great base, but I wanted something easier, that would be more easily parsed from its direct storage in MySQL. My modification to his code is onlty a few characters; I dropped out the JSON require, and return the classes as a CSV list. This lets me to easy LIKE '%,classname,%' SELECTs in MySQL, and also gives me the fact value stored in the puppet DB, so I can build a separate tool around that data. Thanks, Matt.

#
# facter fact for puppet classes on node, pulled from /var/lib/puppet/classes.txt
# from <http://sjoeboo.github.com/blog/2012/07/31/updated-puppet-facts-for-puppet-classes/>
#
 
require 'facter'
begin
        Facter.hostname
        Facter.fqdn
rescue
        Facter.loadfacts()
end
hostname = Facter.value('hostname')
fqdn = Facter.value('fqdn')
 
classes_txt = "/var/lib/puppet/classes.txt"
 
if File.exists?(classes_txt) then
        f = File.new(classes_txt)
        classes = Array.new()
        f.readlines.each do |line|
                line = line.chomp.to_s
                line = line.sub(" ","_")
                classes.push(line)
        end
        classes.delete("settings")
        classes.delete("#{hostname}")
        classes.delete("#{fqdn}")
        Facter.add("puppet_classes_csv") do
                setcode do
                        classes.join(",")
                end
        end
end

All of my facts are now available in a GitHub repository: https://github.com/jantman/puppet-facter-facts.

Puppet facter fact for last applied configuration version

Posted on August 21, 2012 by Jason Antman

For anyone else who sets the Puppet config_version paramater to return the current SVN or Git version of your configuration, here’s a fact that grabs that version (by parsing the cached YAML catalog) and sets it as a fact called “catalog_config_version”. It can then be used for sanity-checking your nodes, looking up via the Inventory Service, or you can display it in the Dashboard using my patch: Patch to Puppet Dashboard 1.2.10 to show arbitrary facts in the main node table.

#
# facter fact for last applied config version, skeleton from /var/lib/puppet/client_yaml/catalog/fqdn.yaml
#
 
require 'puppet'
require 'yaml'
require 'facter'
 
localconfig = ARGV[0] || "#{Puppet[:clientyamldir]}/catalog/#{ Facter.fqdn }.yaml"
 
unless File.exist?(localconfig)
  puts("Can't find #{ Facter.fqdn }.yaml")
  exit 1
end
 
lc = File.read(localconfig)
 
begin
  pup = Marshal.load(lc)
rescue TypeError
  pup = YAML.load(lc)
rescue Exception => e
  raise
end
 
if pup.class == Puppet::Resource::Catalog
        Facter.add("catalog_config_version") do
                setcode do
                        pup.version
                end
        end
else
        Facter.add("catalog_config_version") do
                setcode do
                        "unknown"
                end
        end
end

All of my facts are now available in a GitHub repository: https://github.com/jantman/puppet-facter-facts.

Patch to Puppet Dashboard 1.2.10 to show arbitrary facts in the main node table

Posted on August 11, 2012 by Jason Antman

We use Puppet Dashboard at work to view the status of our puppet nodes. While it’s very handy, there’s one feature I really wanted: the ability to show the value of arbitrary puppet facts in the main node table on the home page. Specifically, the facts we use for environment (we have eng/dev, qa, prod, and test puppet environments), zone (physical location) and last applied configuration version. I’m not terribly experience with Ruby, but I managed to muddle my way through a working patch to do this, along with options in the settings file to enable it and configure the facts. You’ll need to restart dashboard (or your web server) to change the facts, of course. The commit is currently available on github, but it doesn’t strictly follow the puppet-dashboard contributing checklist so I may have to redo it.

Here’s a screenshot:

And here’s that the configuration section added to settings.yml looks like:

# Enables display of arbitrary node facts in "home" page node table, between node name and latest report time
enable_home_facts: true
 
# If enable_home_facts is true, the fact names and column headings to display. Simply repeat the following two line pairs
# as needed:
#- name: 'factname'
#  heading: 'heading text'
home_facts: 
- name: 'environment'
  heading: 'Env'
- name: 'zone'
  heading: 'Zone'
- name: 'catalog_config_version'
  heading: 'Cfg Ver'

If I feel really adventurous, I’d like to implement my other big wish, some sort of pop-up list of links, based on arbitrary facts (mainly hostname and fqdn) for each node – something where I can mouse over the node name/table cell, and see links (static URLs with node name/fqdn/other facts plugged in) to things like Nagios/Icinga, our backup system, etc.

Python script to find dependency cycles in GraphViz dot files

Posted on March 28, 2012 by Jason Antman

Using GraphViz to describe configurations is relatively popular in the software and systems architecture world; the simple text-based format makes it quiet simple, and the directed graph (dot file) is a simple method to store a graph of information flow or component relationships. Puppet includes builtin support for generating dot graphs of its configuration resources and relationships (both those specified by the user, and all relationships including ones generated by Puppet itself).

One of the common uses for puppet’s graphs is to identify dependency cycles, as cyclic dependencies cause an error condition. However, Puppet’s own FAQ only mentions using the dot command to generate a PNG graphical representation of the the graph. When debugging a recent problem with puppet, I ended up with a message like:
Could not apply complete catalog: Found dependency cycles in the following relationships: Service[puppet] => File[/var/lib/puppet/yaml/foreman], File[/var/lib/puppet/yaml] => File[/var/lib/puppet/yaml/foreman], Service[puppet] => File[/etc/puppet/node.rb], File[fileserver.conf] => Service[apache], File[namespaceauth.conf] => Service[apache], File[puppet.conf] => Service[apache], File[puppet.conf] => Service[puppet], Service[puppet] => File[foreman-report.rb], File[/var/lib/puppet/yaml/foreman] => Package[puppet-server], Service[foreman-proxy] => Package[puppet-server], File[foreman-proxy-settings.yml] => Package[puppet-server], Package[foreman-proxy] => Package[puppet-server], User[foreman-proxy] => Package[puppet-server], File[/var/lib/puppet/yaml] => Package[puppet-server], File[foreman-report.rb] => Package[puppet-server], File[/etc/puppet/node.rb] => Package[puppet-server], File[/var/lib/puppet/yaml/foreman] => Exec[create_puppetmaster_certs], Service[foreman-proxy] => Exec[create_puppetmaster_certs], File[foreman-proxy-settings.yml] => Exec[create_puppetmaster_certs], Package[foreman-proxy] => Exec[create_puppetmaster_certs], User[foreman-proxy] => Exec[create_puppetmaster_certs], File[/var/lib/puppet/yaml] => Exec[create_puppetmaster_certs], File[foreman-report.rb] => Exec[create_puppetmaster_certs], File[/etc/puppet/node.rb] => Exec[create_puppetmaster_certs], File[/var/lib/puppet/yaml/foreman] => File[/etc/puppet/environments], Service[foreman-proxy] => File[/etc/puppet/environments], File[foreman-proxy-settings.yml] => File[/etc/puppet/environments], Package[foreman-proxy] => File[/etc/puppet/environments], User[foreman-proxy] => File[/etc/puppet/environments], File[/var/lib/puppet/yaml] => File[/etc/puppet/environments], File[foreman-report.rb] => File[/etc/puppet/environments], File[/etc/puppet/node.rb] => File[/etc/puppet/environments], File[foreman-proxy-settings.yml] => Service[foreman-proxy], Package[foreman-proxy] => Service[foreman-proxy], Service[puppet]
Not exactly easy to follow, or to pull much meaning out of, even if I did some slight reformatting by putting in some newlines. I then generated the png as described in the Puppet FAQs, but even scrolling back and forth on this for 20 minutes didn’t help: (note: link is to the original 14405x665px png)

With a little research, I managed to find the NetworkX package for Python; according to their site, “NetworkX is a Python language software package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.” Among its features are the ability to read dot files using the pydot library, and the ability to find simple cycles within a graph. In about 20 minutes, I hacked together the dead-simple script below.

Given a dot file (of the type generated, for example, by Puppet), this will output all of the cycles found within the graph. I ran this script on the expanded_relationships.dot file from Puppet, which had 99 nodes (nodes on the graph, not puppet clients), and got the following output:

['File[foreman-report.rb]', 'File[puppet.conf]', 'Service[puppet]', 'File[foreman-report.rb]']
['File[foreman-report.rb]', 'Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'File[foreman-report.rb]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'Service[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'File[/etc/puppet/node.rb]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'File[foreman-proxy-settings.yml]', 'Service[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'File[foreman-proxy-settings.yml]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'User[foreman-proxy]', 'File[/etc/puppet/node.rb]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'User[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'Service[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'File[foreman-proxy-settings.yml]', 'Service[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'File[foreman-proxy-settings.yml]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'File[/var/lib/puppet/yaml/foreman]', 'Package[puppet-server]']
['File[/etc/puppet/node.rb]', 'File[puppet.conf]', 'Service[puppet]', 'File[/etc/puppet/node.rb]']
['File[/etc/puppet/node.rb]', 'File[puppet.conf]', 'Service[puppet]', 'User[foreman-proxy]', 'File[/etc/puppet/node.rb]']
['File[puppet.conf]', 'Service[puppet]', 'Service[foreman-proxy]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'Service[foreman-proxy]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'File[foreman-proxy-settings.yml]', 'Service[foreman-proxy]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'File[foreman-proxy-settings.yml]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'File[/var/lib/puppet/yaml/foreman]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'File[foreman-proxy-settings.yml]', 'Service[foreman-proxy]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'File[foreman-proxy-settings.yml]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'User[foreman-proxy]', 'File[puppet.conf]']

It probably would have taken me hours to come up with that by hand, let alone realize that Service[puppet] is the common item in all of them, and hence the problem. With that little tidbit of information, I managed to track down an extraneous “require puppet” that this all originated from. I sincerely hope that this script will save someone else at least as much time as it took me to write (I know it will for me…).

You can always obtain the latest version of this script from dot_find_cycles.py (SVN), or by checking out my misc-scripts SVN repository. It’s free for use and distribution, provided that you leave my copyright/attribution and source URL notice intact, update the changelog, and send any features/fixes back to me. The script is written in Python, and depends on the python-networkx, graphviz-python, and pydot packages (all of which are available as packages in the default repos of Fedora and CentOS at least). For Puppet purposes, I’d recommend running this on expanded_relationships.dot to get the full information.

#!/usr/bin/env python
 
"""
dot_find_cycles.py - uses Pydot and NetworkX to find cycles in a dot file directed graph.
 
Very helpful for 
 
By Jason Antman <jason@jasonantman.com> 2012.
 
Free for all use, provided that you send any changes you make back to me, update the changelog, and keep this comment intact.
 
REQUIREMENTS:
Python
python-networkx - <http://networkx.lanl.gov/>
graphviz-python - <http://www.graphviz.org/>
pydot - <http://code.google.com/p/pydot/>
(all of these are available as native packages at least on CentOS)
 
USAGE:
dot_find_cycles.py /path/to/file.dot
 
The canonical source of this script can always be found from:
<http://blog.jasonantman.com/2012/03/python-script-to-find-dependency-cycles-in-graphviz-dot-files/>
 
$HeadURL: http://svn.jasonantman.com/misc-scripts/dot_find_cycles.py $
$LastChangedRevision: 33 $
 
CHANGELOG:
    Wednesday 2012-03-28 Jason Antman <jason@jasonantman.com>:
        - initial script creation
"""
 
import sys
from os import path, access, R_OK
import networkx as nx
 
def usage():
    sys.stderr.write("dot_find_cycles.py by Jason Antman <http://blog.jasonantman.com>\n")
    sys.stderr.write("  finds cycles in dot file graphs, such as those from Puppet\n\n")
    sys.stderr.write("USAGE: dot_find_cycles.py /path/to/file.dot\n")
 
def main():
 
    path = ""
    if (len(sys.argv) > 1):
        path = sys.argv[1]
    else:
        usage()
        sys.exit(1)
 
    try:
        fh = open(path)
    except IOError as e:
        sys.stderr.write("ERROR: could not read file " + path + "\n")
        usage()
        sys.exit(1)
 
    # read in the specified file, create a networkx DiGraph
    G = nx.DiGraph(nx.read_dot(path))
 
    C = nx.simple_cycles(G)
    if(len(C) < 1):
        sys.exit(0)
    for i in C:
        print i
 
# Run
if __name__ == "__main__":
    main()

The state of Puppet External Node Classifiers

Posted on February 26, 2012 by Jason Antman

At work, we’re in the process of rolling out puppet for configuration management of our servers. It will be an integral part of the provisioning process of all new physical and virtual hosts, and will also be phased in on existing hosts as possible. Right now, we have an initial puppet install that was “development”, but we’re about to move to “production” (new puppetmaster in our production infrastructure, production MySQL, etc.). We’ve been using Dashboard, but just as a report viewer. Up until now, we’ve been using nodes.pp and per-node flat-file manifests. I’ve got a few issues with this, but the biggest is that all of our node definitions (and their classes and parameters) live in the same SVN repository as our modules and other puppet configuration. Not only does this mean a checkout and commit just to change a node parameter or add a module/class to a node, but it also means that for my team members who don’t have previous puppet experience, it greatly blurs the line between administering puppet (developing and maintaining modules) and using puppet (building a node, changing node params or modules/classes), since both tasks are accomplished in the same SVN repository.

So, I’ve been pushing an External Node Classifier (ENC) with a web interface as one of the biggest feature enhancements we need for our puppet install. The complicating factor is that I’ve been given a time frame of approximately 1 week to get the “production” puppetmaster running on our production infrastructure and marked as “done”. That includes the ENC. At my last gig, at Rutgers University, I wrote our ENC in PHP (actually I wrote it for my half-dozen or so boxes at home, and brought it to Rutgers gratis), and it also handled kickstart file distribution and PXE configuration, and was extended to also set DHCP and DNS for the hosts – a one-stop solution. Unfortunately the code is very organization-specific, not terribly solid, and the UI looks awful, so it’s not a fit for the current employer. So I have to find something else that fits the bill. I have a list of initial (“phase 1″) requirements that are a mix of functionality that we require and management requirements:

Must support environments, since we make use of them.
Must support default values for parameters based on environment, “zone” (a custom fact and variable we define), or a combination of both.
For accountability and legal reasons, must have full auditing of all changes by all users (and, obviously, support authentication).
Display node last run time and status.

As well as at least the ability to implement some of our phase 2 requirements:

Ability to show modules and classes applied to a node, including those required/included through other modules/classes/roles.
Should support at least some level of puppet report display.
Ability to trigger a node run (kick) from the UI.
Some level of permission separation, ACL or RBAC so that we could potentially delegate control of a certain module or parameter, on a certain group of nodes, to the development team.
Per-node links to other tools such as Icinga/Nagios or our wiki.
Some way of detecting valid classes and modules (and our “role” module) per-environment (i.e. available modules/classes/roles should be pulled from the configs, not manually entered).
Ability to display puppet docs from modules/classes

Our current situation makes this even more difficult: we’re an operations team of five (hiring at the moment to fill the position of the sixth), and I believe I’m the only member of the team with any real software development experience. And none of us have experience with Ruby (which Puppet and most of its universe is written in). This means that any in-house solution runs the risk of being unmaintainable should I get hit by a bus (some of our team have various levels of experience with Perl and other scripting languages, but not really from an app development perspective). Because of these reasons, there’s a management aversion to anything that we code ourselves (well, these reasons, and the fact that with a shorthanded team we don’t have much time for projects without an immediate impact).

So, I spent hours looking around online trying to find existing web-based ENC projects, and came up with a pretty small list:

Dashboard, the Puppet Labs web UI. It’s the most common web-based puppet ENC as far as I know, and since it’s an official Puppet Labs project (and the basis for their Puppet Enterprise UI), its future is pretty secure. But it’s still very basic (let alone enterprise features), and has a plugin system that is very young.
The Foreman is probably the second-most-common puppet ENC, and has also been around about as long as Dashboard. Its features are nice, and it includes support for Kickstart (management of TFTP and DHCP) and DNS, as well as some virtual machine management. Unfortunately, we already have DHCP and DNS infrastructure so I’m sure it would be quite a bit of effort to integrate it with our environment, and for a non-Ruby shop, it has the same problem with maintainability of custom code.
initr, a Redmine plugin that functions as an ENC and manages modules. It includes RBAC and leverages Redmine. But since we don’t use Redmine, it’s not much of an advantage.
OpenNMS Puppet Node PusherAn ENC script for OpenNMS, which we also don’t use.

I was pretty amazed to see that nobody had written a puppet web UI/ENC in PHP (or Perl or Python), especially since Puppet is now quite popular.

So, I’m essentially left with the following options:

Start from scratch and write my own in PHP. By far the worst option, since we don’t have anyone on our team who’s likely to maintain it, and the Puppet community is Ruby-focused.
Use Foreman, since it’s the only one that appears to offer audit logging, have a bunch of features that don’t work for us, and hopefully deal with it.
Learn Ruby, write plugins for Dashboard, and hope that Puppet Labs or someone else will pick them up and maintain them if I can’t.

At the moment, I’ve decided to investigate Foreman and initr in a bit more depth, and also play around with the Dashboard code and try to pick up some Ruby (as they’re all written in Ruby anyway). I’ll also discuss these options with the team and see how opinions go (keeping in mind that the higher the likelihood of the community picking up/merging my changes, the better).

Puppet Syntax Highlighting with GeSHi

Posted on January 11, 2012 by Jason Antman

This blog is run on wordpress, and I also do quite a bit in PHP, so I’m familiar with the GeSHi syntax highlighter. It’s PHP-based, and can run both as a WordPress plugin (WP-Syntax) and as a PHP module. It also works quite well with the MediaWiki SyntaxHighlight GeSHi extension.

Today I was documenting some Puppet code in a wiki, and realized that I didn’t have syntax highlighting. Well, fellow Linux sysadmin and puppetmaster Jason Hancock was nice enough to post on his blog (Puppet Syntax Highlighting with GeSHi) that he’s developed a GeSHi language file for Puppet, available from GitHub. Many thanks!

Puppet problems with hostname in autosign.conf – Invalid pattern

Posted on October 14, 2009 by Jason Antman

In playing with Puppet (0.24.8 on clients and server) today (well, building a new host) I came by a strange error when I ran puppet on the client:

err: Could not request certificate: Certificate retrieval failed: Invalid pattern css-storemanager

The thing that was so strange is that “css-storemanager” is the name of a host at my site, controlled by Puppet, but it has nothing to do with the host I was building. They’re different boxes, on different subnets, in different rooms. One is a SunFire and the other is an HP desktop.

Google turned up nothing. Running puppetmasterd with --debug --trace yielded:

info: Listening on port 8140
notice: Starting Puppet server version 0.24.8
notice: Allowing unauthenticated client ccf-hill019-12.example.edu(172.x.x.x) access to puppetca.getcert
/usr/lib/ruby/site_ruby/1.8/puppet/network/authstore.rb:289:in `parse'
/usr/lib/ruby/site_ruby/1.8/puppet/network/authstore.rb:170:in `pattern='
/usr/lib/ruby/site_ruby/1.8/puppet/network/authstore.rb:151:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/authstore.rb:80:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet/network/authstore.rb:80:in `store'
/usr/lib/ruby/site_ruby/1.8/puppet/network/authstore.rb:20:in `allow'
/usr/lib/ruby/site_ruby/1.8/puppet/network/handler/ca.rb:54:in `autosign?'
/usr/lib/ruby/site_ruby/1.8/puppet/network/handler/ca.rb:51:in `each'
/usr/lib/ruby/site_ruby/1.8/puppet/network/handler/ca.rb:51:in `autosign?'
/usr/lib/ruby/site_ruby/1.8/puppet/network/handler/ca.rb:50:in `open'
/usr/lib/ruby/site_ruby/1.8/puppet/network/handler/ca.rb:50:in `autosign?'
/usr/lib/ruby/site_ruby/1.8/puppet/network/handler/ca.rb:112:in `getcert'
/usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `to_proc'
/usr/lib/ruby/site_ruby/1.8/puppet/network/xmlrpc/processor.rb:52:in `call'
/usr/lib/ruby/site_ruby/1.8/puppet/network/xmlrpc/processor.rb:52:in `protect_service'
/usr/lib/ruby/site_ruby/1.8/puppet/network/xmlrpc/processor.rb:85:in `setup_processor'
/usr/lib/ruby/1.8/xmlrpc/server.rb:336:in `call'
/usr/lib/ruby/1.8/xmlrpc/server.rb:336:in `dispatch'
/usr/lib/ruby/1.8/xmlrpc/server.rb:323:in `each'
/usr/lib/ruby/1.8/xmlrpc/server.rb:323:in `dispatch'
/usr/lib/ruby/1.8/xmlrpc/server.rb:366:in `call_method'
/usr/lib/ruby/1.8/xmlrpc/server.rb:378:in `handle'
/usr/lib/ruby/site_ruby/1.8/puppet/network/xmlrpc/processor.rb:44:in `process'
/usr/lib/ruby/site_ruby/1.8/puppet/network/xmlrpc/webrick_servlet.rb:68:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet.rb:293:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet.rb:144:in `newthread'
/usr/lib/ruby/site_ruby/1.8/puppet.rb:143:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet.rb:143:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet.rb:143:in `newthread'
/usr/lib/ruby/site_ruby/1.8/puppet.rb:291:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet.rb:290:in `each'
/usr/lib/ruby/site_ruby/1.8/puppet.rb:290:in `start'
/usr/sbin/puppetmasterd:285
err: Invalid pattern css-storemanager

After a bit of investigation into that trace, I found the following code in /usr/lib/ruby/site_ruby/1.8/puppet/network/authstore.rb starting on line 242:

            # Parse our input pattern and figure out what kind of allowal
            # statement it is.  The output of this is used for later matching.
            def parse(value)
                case value
                when /^(\d+\.){1,3}\*$/: # an ip address with a '*' at the end
                    @name = :ip
                    match = $1
                    match.sub!(".", '')
                    ary = value.split(".")
 
                    mask = case ary.index(match)
                    when 0: 8
                    when 1: 16
                    when 2: 24
                    else
                        raise AuthStoreError, "Invalid IP pattern %s" % value
                    end
 
                    @length = mask
 
                    ary.pop
                    while ary.length < 4
                        ary.push("0")
                    end
 
                    begin
                        @pattern = IPAddr.new(ary.join(".") + "/" + mask.to_s)
                    rescue ArgumentError => detail
                        raise AuthStoreError, "Invalid IP address pattern %s" % value
                    end
                when /^([a-zA-Z][-\w]*\.)+[-\w]+$/: # a full hostname
                    @name = :domain
                    @pattern = munge_name(value)
                when /^\*(\.([a-zA-Z][-\w]*)){1,}$/: # *.domain.com
                    @name = :domain
                    @pattern = munge_name(value)
                    @pattern.pop # take off the '*'
                    @length = @pattern.length
                else
                    # Else, use the IPAddr class to determine if we've got a
                    # valid IP address.
                    if value =~ /\/(\d+)$/
                        @length = Integer($1)
                    end
                    begin
                        @pattern = IPAddr.new(value)
                    rescue ArgumentError => detail
                        raise AuthStoreError, "Invalid pattern %s" % value
                    end
                    @name = :ip
                end

Following the trace back, I took a look at /usr/lib/ruby/site_ruby/1.8/puppet/network/handler/ca.rb starting at line 50:

            auth = Puppet::Network::AuthStore.new
            File.open(autosign) { |f|
                f.each { |line|
                    next if line =~ /^\s*#/
                    next if line =~ /^\s*$/
                    auth.allow(line.chomp)
                }
            }

After looking at this, it clicked that it must be what evaluates autosign.conf. Taking a look at mine, one line stood out: a line containing only “css-storemanager”, not a FQDN like all the rest. The parse() function in authstore.rb only accepts IP addresses and FQDNs (or IP addresses ending in a wildcard, or wildcard FQDNs). It appears to choke on hostnames (a string that doesn’t match an FQDN or IP). Interestingly, it also evaluates in order, and stops evaluating autosign.conf once it finds a match. So, if you’re a Bad Person like me, and left autosign turned on for all of your hosts, you wouldn’t notice this until you try and build a new box.

To solve this, just remove any offending lines from autosign.conf.

I’ve filed a bug report on the Puppet Trac: Issue 2723.

Building a Rebuild-able Site

Posted on May 6, 2009 by Jason Antman

At $WORK, my group runs about two dozen servers that provide services for over 60,000 users. They’re a mix of Windows and Linux, with some old Solaris stuff thrown in there. The one thing they have in common is they’re all hand-built, hand-configured, and old. They’ve been around for a while. At the moment, we don’t even have an adequate backup system.

So, being the closest thing to a SysAdmin we have (my official title is still Student Systems Programmer), it’s my job to build a new installation, configuration and backup infrastructure. We’ve already standardized on CentOS as a University-wide distro, and have a local full mirror, so I don’t need to choose a distro. I do, however, have to plan the installation and backup architecture. The main requirements are:

Lowest overall time for bare-metal recovery to a working system.
Ease of use, as people other than myself will need to administer it (so they should be able to do so from a cheat sheet in the wiki).
Repeatability – it should be easy and intuitive to make an almost-exact-copy of a machine.

I started a thread a few days ago on the SAGE mailing list, which you can find here.

At the moment, it looks like the general idea that I’m going with is to use Kickstart to install the systems, using a basic and minimal Kickstart file. Basic package selection (minimalist) with just what’s needed to configure the system with a hostname and network settings for the management VLAN. I’ll then have Kickstart install and configure a configuration management package – I’m leaning towards Puppet over Cfengine and am starting testing. The config management software will handle all of the customization for the system (everything different from the base generic Kickstart install) so it’s all kept under the control of config management from step 1.

The final part is a backup system, mainly for whatever eventually – whether out of human error or simple laziness – ends up out of the config management system’s control. Our previous SA had settled on Zmanda, the paid version of Amanda, which comes with specific plugins for MySQL and MSSQL. I’m also looking at Bacula, mainly because of its’ advanced features, scheduling (especially the new scheduling in Bacula 3) and scalability.

The beauty that I see in having Kickstart do something minimal and then letting Puppet handle the rest is that (especially since we’ve standardized on SunFire X4100′s with identical configurations) I can kickstart and rack up a few spare machines, and to get them up and running all I need to do is power them up (iLOM) and tell Puppet what to make them.

I’m currently starting testing of both Puppet itself and getting Kickstart to start the puppet install and daemon (instructions from David Lutterkort’s blog (Red Hat software engineer)). We’ll see how everything goes…

Jason Antman's Blog

Information, thoughts, tips and tricks from a professional system administrator / monitoring and automation nut, open source junkie, troubleshooter, sometime software developer and ceaseless tinkerer. And some occasional commentary on my hobbies and non-tech interests.

Category Archives: Puppet