Creating RPMs from Perl CPAN Modules

Posted on by

I try my absolute best to always install software on my Linux boxes as RPMs, installed through Yum (yes, I use CentOS on servers and Fedora on my desktops/laptops). Not only is this more-or-less required to sanely manage configuration through Puppet, but it also lets me recreate a machine, or install dependencies for something, in one simple command line. Unfortunately, I run quite a bit of Perl code, and there are a lot of CPAN Perl modules that aren’t in any of the usual CentOS/Fedora repositories.

Enter cpan2rpm: a Perl script that, in its simplest invocation, downloads a specified CPAN module and automatically builds RPMs and SRPMs for it. The original version by Erick Calder hasn’t been touched since 2005, but there’s a newer version from Mediaburst, cpan2rpmmb, that seems to incorporate some nice improvements and worked quite well for me.

Perl script to convert F5 BigIp VIP address to list of internal pool member addresses

Posted on by

I often find myself logging in to the web UI of F5 BigIp load balancers and tracing down a VIP address to the servers that actually back it. This is an arduous, repetitive task of tracing from the VIP list to the VIP details page to find the default pool, then matching up that in the pool list and checking the pool members page. Luckily, the F5 boxes have a web service API that can be used for tasks like this. They have GPL sample code in Perl that uses only SOAP::Lite (as well as Getopt::Long and Pod::Usage) to interact with an F5 BigIp. I wrote a simple script to trace a VIP to the appropriate internal pool member addresses, assuming you have a simple configuration of VIP -> Single default pool -> pool members.

Usage is quite simple:

> ./VipToInternalHosts.pl --host=prod-lb1.example.com --user=myname --pass=mypassword --vip=128.6.30.130:80
VIP 128.6.30.130:80 (f5_vip_name) -> Pool 'pool_name'
Members of Pool 'pool_name':
	10.145.15.10:80
	10.145.15.11:80

The code can be found at http://svn.jasonantman.com/misc-scripts/VipToInternalHosts.pl via either HTTP or SVN. I hope it’s of use to someone else as well.

SysAdmin Links of The Day

Posted on by

A few links that I’ve had in my “mention in a blog post” category for a while:

Instagram – Scaling a Startup

Posted on by

If you keep up with news on the ‘net, you may have heard about photo sharing startup Instagram, which was purchased by Facebook for $1 Billion just 9 days after they released their Android app. Well, not only are they based mostly on open source software (well, yeah, pretty much a given for web startups these days), but they’ve dealt with scaling issues like a million new users in 12 hours, and they’re talking about it. There’s the slide deck to a talk that co-founder Mike Krieger gave on TechCrunch and Scribd, along with a High Scalability article about it, and an earlier High Scalability article that gives an overview of the company and some details of what and how they’re running. Instagram Engineering also has a tumblr account, with a bunch of cool posts like Keeping Instagram up with over a million new users in twelve hours (which specifically mentions statsd, dogslow, PGFouine, node2dm and some database stuff) and What Powers Instagram: Hundreds of Instances, Dozens of Technologies which talks about their OS and hosting (Ubuntu 11.04 on EC2), load balancing (nginx, DNS and Amazon Elastic Load Balancer), Django, Redis, Solr, Munin, etc.

This is a really cool company, doing some really cool stuff, at a really large scale, and growing fast.

On another note, I’m continuing my attempt to read all of the excellent Puppet articles on Brice Figureau’s (aka masterzen) blog. It’s taking a while, as it’s really good, in-depth information that I want to rememeber, but I’d highly recommend it for anyone working with Puppet.

A Collection of Great Links on Monitoring, SysAdmin, Scaling, etc.

Posted on by

I’ve had a bunch of tabs open in my browser for a while – stuff that I read, thought was wonderful, and wanted to comment on. At risk of letting it pile up forever, here’s a collection of links that I thought were really interesting or insightful…

  • MongoDB is Fantastic for Logging – I was looking into some log storage ideas, and came by this post (on the MongoDB blog) about why Mongo is well-suited to storing logs.
  • Sensu – a Ruby-based cloud-oriented monitoring system. It uses AMQP/RabbitMQ to communicate between the clients and server, which is a really big part of what I think monitoring should be.
  • High Scalability – this is one of the few blogs I follow on a regular basis. Some really wonderful stuff, and great food for thought.
  • Everything Sysadmin: Fear of Rebooting – A great article on Tom Limoncelli’s blog about why we fear rebooting machines and why this is bad – moreover, why we should reboot often.
  • The Netflix Tech Blog: Fault Tolerance in a High Volume, Distributed System – This is a really, really cool post NetFlix about how latency increases in a single subsystem can bring down their whole API in seconds, and how they combat this. Really cool stuff.
  • Ars Technica – Exclusive: a behind-the-scenes look at Facebook release engineering – Ars Technical is more or less “mainstream media” to me, but this is a really interesting writeup on Facebook’s release engineering process, albeit at a higher level. Specifically, it talks about their automation, phased rollouts, rollbacks, and how they release the Facebook codebase as a single giant binary, sent out via BitTorrent.
  • Monitoring Sucks blog posts (github) – The “monitoing sucks” movement really speaks to me, having worked extensively with Nagios, Cacti, and similar technologies. Specifically, having rolled out monitoring in a variety of “weird” scenarios (a lot of monitoring devices or whole networks behind NAT, on dynamic IP connections, or otherwise unreachable from a central server), I’ve felt a lot of pain in the current want of doing things. There are a lot of really good thoughts linked here, especially the “wonderland” series by Patrick Debois and the “Latency sucks” series by Lindsay Holmwood. This really got me thinking about my ideal monitoring system, which among other things, would integrate the “alerting” functions of Nagios with graphing/trending and correlation, would be based on some sort of message queue architecture (that supports multiple levels of proxies that could gracefully support NAT and multiple hops), and would be configured almost totally on the originating “client” (unlike the pain of distributed Nagios/Icinga).
  • Mike Brittain – Metrics Driven Engineering at Etsy (3.2MB PDF) – presentation slides. I’d love to see the video. Some really good ideas about putting the science back into being a SysAdmin. Also mentions a few tools I really want to play around with (including ganglia, graphite, logster and StatsD). Also mentions adding PHP memory usage and time to Apache logs, which I don’t believe I never thought of.
  • Some really thoughtful posts from R. I. Pienaar on Thinking about monitoring frameworks and Composable Architectures. Some really good stuff, but what else would you expect from someone like this.

CentOS/Fedora Install on SuperMicro Servers via IPMI Card KVM Over IP

Posted on by

I recently had to setup Fedora 16 to test something on a SuperMicro 6015T-TV 1U “dual” server. This is a 1U enclosure with two separate servers in it – based on the SuperMicro X7DBT motherboards – each with an AOC-SIMSO+ IPMI and KVM-over-IP management card. Every time I tried different options for the kernel parameters (I was using Cobbler), I could get the OS to boot, but once Anaconda started, I’d lose image (“No Signal”) on the IP KVM, and the serial console would go quiet. It took me about a dozen tries before I found a mailing list reference to the “nomodeset” option. This did the trick perfectly, and kept Anaconda working.

Adjusting the VirtualBox F12 BIOS Boot Prompt Timeout

Posted on by

I’m working from home today, connected by VPN. I’m in the process of testing a bunch of Puppet stuff, and needed to re-image a bunch of VirtualBox VMs on my desktop at work, using PXE boot to Cobbler. I’m only connected to the desktop by SSH, and running the VMs with VBoxHeadless and connecting to them via RDP (well, VRDP). The problem with this is that if I start a VM on my console window, then switch to my RDP client and connect, by the time the VM gets keyboard focus, it’s already past the VBox “Press F12 to select boot device” prompt and booting from disk. I could modify the boot order on the VM, but then that becomes a pain when it reboots after the install.

Thanks to some of the guys on the VirtualBox IRC channel, I found out about the --bioslogodisplaytime option for VMs, which controls the length of time (in milliseconds) that the boot splash screen is shown (the default value seems to be 0). It’s included in the reference guide to VBoxManage in the modifyvm section. Setting this to a value of 10 seconds or so, as shown below, is more than enough for me to start the VM, Alt-Tab to my RDP client, connect to the VM, and hit ‘F12′ to select a one-time network boot:

VBoxManage modifyvm VMNAME --bioslogodisplaytime 10000

Rsync on an Android phone

Posted on by

Every once in a while, there are some files that I want kept in sync between my Android-based phone and one of my Linux (or Mac, or any Unix, or maybe Windows too, but I use Linux…) boxes. Yeah, I can copy them over manually via USB or even something a bit simpler like AndFTP (assuming you can SCP to the target machine). But that’s a real pain for anything like my KeePass (well, actually KeePassX and KeePassDroid) password database, that I might add something to at any time and forget to sync. I also try to occasionally (waiting in line?) backup SMS, call logs, etc. on my phone, and like to have those synced back to the desktop automatically.

Enter the solution: rsync backup for Android, a rsync client for Android that includes Tasker plugins (there are a few things about the app that I don’t like, but it seems to be the only option at the moment), and Tasker, an automation framework for Android.Tasker is one of the few Android apps that I’ve actually bought (i.e. not free/no-cost), and is currently selling for $6.49. It’s an incredibly capable task automator, very much akin to Locale on steroids. On the down side, Tasker can eat up battery life if you don’t configure it intelligently, and it’s not always 100% reliable when interacting with the system. On the positive side, Tasker can identify practically any combination of states in the android system (from hardware and software events to GPS location, time, signal status, etc.) and perform almost any task on the system based on this information. Sure, this specific problem could be solved with a cron replacement (which Android lacks, of course), but Tasker can do things like play specific audio files when you get an SMS from a specific number, mute audio at certain GPS locations, or turn WiFi on when I get home and off when I leave the house. It also has a plugin architecture, and rsync backup for Android happens to have a plugin that works with it.

So, our goal is to have a daily, bi-directional, newest-file-wins sync between a directory on our Android phone and a directory on a computer. I’m not going to go into a lot of the computer-side stuff, mainly because that varies quite a bit between operating systems, and also because my personal setup is a bit paranoid in terms of security. For the computer side, we’ll need a machine that can be SSHed to from the Internet (either a static IP or a known hostname/dynamic DNS), a user that can run rsync over SSH, and a directory that’s writable (obviously).

Setup:

  1. Buy and install the Tasker app.
  2. Install the rsync backup for Android app.
  3. Configure the rsync stuff on the computer. In the simplest form, we’ll just need a user that can login and run rsync, and a directory to sync from/to (note: this should be a directory used only for syncing the phone…).
  4. Open the rsync backup for Android app. Use Menu -> Generate Keys to generate a new pair of SSH keys, and then get the public key setup on the target computer. See the developer’s web site for instructions.
  5. Once keys are setup, create a new profile called “PC-droid”. Set the local directory to a new empty directory (I used /sdcard/sync), enter the remote host address, port, username, and remote directory, and select the private SSH key that you created. Check off “rsync on reverse direction”. As this program is just a GUI wrapper around normal rsync binaries, you can specify additional options to the rsync command; my string ended up being -vHrltDuO --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms. If it helps, at the bottom of the screen you can see the actual rsync command line that will be run. Save when done.
  6. Save the profile, then long-press it and select “Duplicate”. Change the name to “droid-PC”, uncheck “rsync in reverse direction”, and change your additional options as needed (mine became -vHrltDu --chmod=Dug+rwx,o-rwx,Fug+rw,o-rw --no-perms). Save when done.
  7. Create a test file in the sync directory on the PC, and a different one in the sync directory on the droid.
  8. One at a time, in the rsync backup app, tap on the profile names. If all goes well, the syncs should run, and both files will now be in both places. If there are any problems, the output should help; the most likely issues are probably permissions, rsync command options, or SSH keys.
  9. Long-press each profile, select “Edit”, and check off “Close log window after job is done”. Save profile.
  10. Now fire up Tasker. Click the “+” at the bottom of the screen to create a new profile, call it “sync”, and click the check mark.
  11. On the First Context panel, tap Time, and select when you want the jobs to run; I chose 03:01. Tap the check mark.
  12. On the Task Selection panel, tap New Task. Give it a name, like “sync2″.
  13. On the Task Edit panel, tap the “+” button at the bottom left, tap Plugin, tap “rsync backup for Android”, click the “Edit” button on the Configuration line, and select the PC-droid rsync profile. Tap the check mark in the lower left to save.
  14. Repeat the last step for the droid-PC rsync profile.
  15. Tap the check box in the lower left. This saves the profile.
  16. In the main Tasker screen, make sure there’s a green check to the right of the profile you just added, and that the button at the bottom right of the screen is set to “On”.

Assuming this all went well, the next time the time you specified rolls around, your sync should run. If you gave the task a name in step 12, you can setup additional profiles to run it at other times (or use the repeat logic builtin).

Python script to find dependency cycles in GraphViz dot files

Posted on by

Using GraphViz to describe configurations is relatively popular in the software and systems architecture world; the simple text-based format makes it quiet simple, and the directed graph (dot file) is a simple method to store a graph of information flow or component relationships. Puppet includes builtin support for generating dot graphs of its configuration resources and relationships (both those specified by the user, and all relationships including ones generated by Puppet itself).

One of the common uses for puppet’s graphs is to identify dependency cycles, as cyclic dependencies cause an error condition. However, Puppet’s own FAQ only mentions using the dot command to generate a PNG graphical representation of the the graph. When debugging a recent problem with puppet, I ended up with a message like:

Could not apply complete catalog: Found dependency cycles in the following relationships: Service[puppet] => File[/var/lib/puppet/yaml/foreman], File[/var/lib/puppet/yaml] => File[/var/lib/puppet/yaml/foreman], Service[puppet] => File[/etc/puppet/node.rb], File[fileserver.conf] => Service[apache], File[namespaceauth.conf] => Service[apache], File[puppet.conf] => Service[apache], File[puppet.conf] => Service[puppet], Service[puppet] => File[foreman-report.rb], File[/var/lib/puppet/yaml/foreman] => Package[puppet-server], Service[foreman-proxy] => Package[puppet-server], File[foreman-proxy-settings.yml] => Package[puppet-server], Package[foreman-proxy] => Package[puppet-server], User[foreman-proxy] => Package[puppet-server], File[/var/lib/puppet/yaml] => Package[puppet-server], File[foreman-report.rb] => Package[puppet-server], File[/etc/puppet/node.rb] => Package[puppet-server], File[/var/lib/puppet/yaml/foreman] => Exec[create_puppetmaster_certs], Service[foreman-proxy] => Exec[create_puppetmaster_certs], File[foreman-proxy-settings.yml] => Exec[create_puppetmaster_certs], Package[foreman-proxy] => Exec[create_puppetmaster_certs], User[foreman-proxy] => Exec[create_puppetmaster_certs], File[/var/lib/puppet/yaml] => Exec[create_puppetmaster_certs], File[foreman-report.rb] => Exec[create_puppetmaster_certs], File[/etc/puppet/node.rb] => Exec[create_puppetmaster_certs], File[/var/lib/puppet/yaml/foreman] => File[/etc/puppet/environments], Service[foreman-proxy] => File[/etc/puppet/environments], File[foreman-proxy-settings.yml] => File[/etc/puppet/environments], Package[foreman-proxy] => File[/etc/puppet/environments], User[foreman-proxy] => File[/etc/puppet/environments], File[/var/lib/puppet/yaml] => File[/etc/puppet/environments], File[foreman-report.rb] => File[/etc/puppet/environments], File[/etc/puppet/node.rb] => File[/etc/puppet/environments], File[foreman-proxy-settings.yml] => Service[foreman-proxy], Package[foreman-proxy] => Service[foreman-proxy], Service[puppet]

Not exactly easy to follow, or to pull much meaning out of, even if I did some slight reformatting by putting in some newlines. I then generated the png as described in the Puppet FAQs, but even scrolling back and forth on this for 20 minutes didn’t help: (note: link is to the original 14405x665px png)
dot file png

With a little research, I managed to find the NetworkX package for Python; according to their site, “NetworkX is a Python language software package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.” Among its features are the ability to read dot files using the pydot library, and the ability to find simple cycles within a graph. In about 20 minutes, I hacked together the dead-simple script below.

Given a dot file (of the type generated, for example, by Puppet), this will output all of the cycles found within the graph. I ran this script on the expanded_relationships.dot file from Puppet, which had 99 nodes (nodes on the graph, not puppet clients), and got the following output:

['File[foreman-report.rb]', 'File[puppet.conf]', 'Service[puppet]', 'File[foreman-report.rb]']
['File[foreman-report.rb]', 'Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'File[foreman-report.rb]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'Service[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'File[/etc/puppet/node.rb]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'File[foreman-proxy-settings.yml]', 'Service[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'File[foreman-proxy-settings.yml]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'User[foreman-proxy]', 'File[/etc/puppet/node.rb]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'User[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'Service[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'File[foreman-proxy-settings.yml]', 'Service[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'File[foreman-proxy-settings.yml]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'Package[puppet-server]']
['Package[puppet-server]', 'File[puppet.conf]', 'Service[puppet]', 'File[/var/lib/puppet/yaml/foreman]', 'Package[puppet-server]']
['File[/etc/puppet/node.rb]', 'File[puppet.conf]', 'Service[puppet]', 'File[/etc/puppet/node.rb]']
['File[/etc/puppet/node.rb]', 'File[puppet.conf]', 'Service[puppet]', 'User[foreman-proxy]', 'File[/etc/puppet/node.rb]']
['File[puppet.conf]', 'Service[puppet]', 'Service[foreman-proxy]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'Service[foreman-proxy]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'File[foreman-proxy-settings.yml]', 'Service[foreman-proxy]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'Package[foreman-proxy]', 'File[foreman-proxy-settings.yml]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'File[/var/lib/puppet/yaml/foreman]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'File[foreman-proxy-settings.yml]', 'Service[foreman-proxy]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'File[foreman-proxy-settings.yml]', 'File[puppet.conf]']
['File[puppet.conf]', 'Service[puppet]', 'User[foreman-proxy]', 'File[puppet.conf]']

It probably would have taken me hours to come up with that by hand, let alone realize that Service[puppet] is the common item in all of them, and hence the problem. With that little tidbit of information, I managed to track down an extraneous “require puppet” that this all originated from. I sincerely hope that this script will save someone else at least as much time as it took me to write (I know it will for me…).

You can always obtain the latest version of this script from dot_find_cycles.py (SVN), or by checking out my misc-scripts SVN repository. It’s free for use and distribution, provided that you leave my copyright/attribution and source URL notice intact, update the changelog, and send any features/fixes back to me. The script is written in Python, and depends on the python-networkx, graphviz-python, and pydot packages (all of which are available as packages in the default repos of Fedora and CentOS at least). For Puppet purposes, I’d recommend running this on expanded_relationships.dot to get the full information.

#!/usr/bin/env python
 
"""
dot_find_cycles.py - uses Pydot and NetworkX to find cycles in a dot file directed graph.
 
Very helpful for 
 
By Jason Antman <jason@jasonantman.com> 2012.
 
Free for all use, provided that you send any changes you make back to me, update the changelog, and keep this comment intact.
 
REQUIREMENTS:
Python
python-networkx - <http://networkx.lanl.gov/>
graphviz-python - <http://www.graphviz.org/>
pydot - <http://code.google.com/p/pydot/>
(all of these are available as native packages at least on CentOS)
 
USAGE:
dot_find_cycles.py /path/to/file.dot
 
The canonical source of this script can always be found from:
<http://blog.jasonantman.com/2012/03/python-script-to-find-dependency-cycles-in-graphviz-dot-files/>
 
$HeadURL: http://svn.jasonantman.com/misc-scripts/dot_find_cycles.py $
$LastChangedRevision: 33 $
 
CHANGELOG:
    Wednesday 2012-03-28 Jason Antman <jason@jasonantman.com>:
        - initial script creation
"""
 
import sys
from os import path, access, R_OK
import networkx as nx
 
def usage():
    sys.stderr.write("dot_find_cycles.py by Jason Antman <http://blog.jasonantman.com>\n")
    sys.stderr.write("  finds cycles in dot file graphs, such as those from Puppet\n\n")
    sys.stderr.write("USAGE: dot_find_cycles.py /path/to/file.dot\n")
 
def main():
 
    path = ""
    if (len(sys.argv) > 1):
        path = sys.argv[1]
    else:
        usage()
        sys.exit(1)
 
    try:
        fh = open(path)
    except IOError as e:
        sys.stderr.write("ERROR: could not read file " + path + "\n")
        usage()
        sys.exit(1)
 
    # read in the specified file, create a networkx DiGraph
    G = nx.DiGraph(nx.read_dot(path))
 
    C = nx.simple_cycles(G)
    if(len(C) < 1):
        sys.exit(0)
    for i in C:
        print i
 
# Run
if __name__ == "__main__":
    main()