Tag Archives: sysadmin

Search for a small-scale but automated RPM build system

Posted on by

This post is part of a series of older draft posts from a few months ago that I’m just getting around to publishing. Unfortunately, I have yet to find a build system that meets my requirements (see the last paragraph).

At work, we have a handful – currently a really small number – of RPM packages that we need to build and deploy internally for our CentOS server infrastructure. A number of them are just pulled down from specific third-party repositories and rebuilt to have the vendor set as us, and some are internally patched or developed software. We run websites, and on the product side, we’re a Python/Django shop (in fact, probably one of the largest Django apps out there). We don’t deploy our Django apps via RPM, so building and distributing RPMs is definitely not one of our core competencies. In fact, we really only want to do it when we’re testing/deploying a new distro, or when an upstream package is updated.

Last week I pulled a ticket to deploy node.js to one of our build hosts, and we’ve got a few things in the pipeline that also rely on it. I found the puppetlabs-nodejs module on Github that’s supposed to install it on RHEL/CentOS, but it pulls packages from http://patches.fedorapeople.org/oldnode/stable/, and the newest version of nodejs there is 0.6.18, which is quite old. I can’t find any actively maintained sources of newer nodejs packages for RHEL/CentOS (yeah, I know, that’s one down side to the distributions…). However, I did find that nodejs 0.9.5 is being built for Fedora 18/19 in the Fedora build system, is already in the Fedora 18 Testing and Fedora Rawhide repos, but is failing its EL6 builds in their system. The decision I’ve come to is to use the puppetlabs-nodejs module to install it, but try and rebuild the Fedora 18 RPMs under CentOS 5 and 6.

So that’s the background. Now, my current task: to search for an RPM build system for my current job. My core requirements, in no specific order, are:

  • Be relatively easy and quick to use for people who have a specfile or SRPM and want to be able to “ensure => present” the finished RPM on a system. i.e., require as little per-package configuration as possible.
  • Be able to handle rebuilding “all” of our RPMs when we roll out a new distro version. Doesn’t necessarily need to be automatic, but should be relatively simple.
  • Ideally, not need to be running constantly – i.e. something that will cope well with build hosts being VMs that are shut down when they’re not needed.
  • Handle automatically putting successfully built packages into a repository, ideally with some sort of (manual) promotion process from staging to stable.
  • Have minimal external (infrastructure) dependencies that we can’t satisfy with existing systems.

So, the first step was to research existing RPM build systems and how others do this. Here’s a list of what I could find online, though most of these are from distributions and software vendors/projects, not end-user companies that are only building for internal use.

  • Koji is the build system used by Fedora and RedHat. It’s about as full-featured as any can be, and I’m familiar with it from my time at Rutgers University, as it’s used to maintain their CentOS/RHEL packages. It’s based largely on Mock. However, setting up the build server is no trivial task; there are few installations outside of Fedora/RedHat, and it relies on either Kerberos or an SSL CA infrastructure to authenticate machines and clients. So, it’s designed for too large a scale and too much infrastructure for me.
  • PLD Linux has a builder script that seems to automate rpmbuild as well as fetching sources and resolving/building dependencies. I haven’t looked at the script yet, but apparently it’s in PLD’s “rpm-build-tools” package.
  • PLD Linux also has a CVS repository for something called pld-builder.new. The README and ARCHITECTURE files make it sound like a relatively simple mainly-Python system that builds SRPMS and binary packages when requested, and most importantly, seems like a simple system that uses little more than shared filesystem access for communication and coordination.
  • ALT Linux has Sisyphus, which combines repository management and web interface tools, package building and testing tools, and more.
  • The Dries RPM repository uses (or at least used… my reference is quite old) pydar2, “a distributed client/server program which allows you to build multiple spec files on multiple distribution/architecture combinations automatically.” That sounds like it could be what I need, but the last update says that it isn’t finished yet, and that was in 2005.
  • Mandriva Linux has pretty extensive information on their build system on their wiki and a build system theory page, but it seems to be largely a hodgepodge of shell scripts and cronjobs, and is likely not a candidate for use by anyone other than its designers.
  • Argeo provides the SLC framework which has a “RPM Factory” component, but I can’t seem to find much more than a wiki page, and can’t tell if it’s a build automation system or just handles mocking packages and putting them in a repo on a single host.
  • Dag Wieers’ repositories use (or used) a set of python scripts called DAR, “Dynamic Apt Repository builder”. They’re on github but are listed as “old” and haven’t been updated in at least 2 years. The features sound quite interesting, and though it’s based on the Apt repo format, it might provide some good ideas for implementing a similar system.

Update four months later: I’ve yet to find a build system that meets my requirements above. For the moment I’m only managing ~20 packages, so my “build system” is a single shell script that reads in some environment variables and runs through using mock to build them in the correct order (including pushing the finished RPMs back into the local repository that mock reads from) and then pushing the finished packages to our internal repository. Maybe when I have some spare time, I’ll consider a project to either make a slightly better (but simple) RPM build system based on Python, or get our Jenkins install to handle this for me.

Some questions from a tech interview with a big Internet company

Posted on by

A while back, I did a technical phone screen with a big online “social” company (I won’t say who, but they’re a household name, growing fast, and doing cool things; that doesn’t leave too many options). I rarely remember to write down interview questions, but I was cleaning out my desk this morning and came by a ripped-out sheet of notebook paper with a handful of the interview questions written on it. Most of them weren’t terribly difficult, or terribly unusual for competent technical interviewers, but since I happen to actually have the list written down, I though I’d share it. I don’t remember why the programming questions are all Python; likely, I was asked to choose between Python (which I’ve used, though not lately), Ruby (which I can barely muddle my way through reading on a good day), and something else I don’t know. Here are some of them…

  • What is an inode? What does it store?
  • What is a hard link?
  • What is the difference between a hard link and a soft link?
  • What is a list in Python?
  • Name some data structures that you’d use in Python. Describe them, and tell me why you would use them.
  • How would you list all the man pages containing the keyword “date”?
  • If the chmod binary had its permissions set to 000, how would you fix it?

Interesting Systems Links for September 3, 2012

Posted on by

Here is a small selection of sysadmin links that I recently found, and wanted to share:

Patch to Puppet Dashboard 1.2.10 to show arbitrary facts in the main node table

Posted on by

We use Puppet Dashboard at work to view the status of our puppet nodes. While it’s very handy, there’s one feature I really wanted: the ability to show the value of arbitrary puppet facts in the main node table on the home page. Specifically, the facts we use for environment (we have eng/dev, qa, prod, and test puppet environments), zone (physical location) and last applied configuration version. I’m not terribly experience with Ruby, but I managed to muddle my way through a working patch to do this, along with options in the settings file to enable it and configure the facts. You’ll need to restart dashboard (or your web server) to change the facts, of course. The commit is currently available on github, but it doesn’t strictly follow the puppet-dashboard contributing checklist so I may have to redo it.

Here’s a screenshot:

Dashboard screenshot after patch

And here’s that the configuration section added to settings.yml looks like:

# Enables display of arbitrary node facts in "home" page node table, between node name and latest report time
enable_home_facts: true
 
# If enable_home_facts is true, the fact names and column headings to display. Simply repeat the following two line pairs
# as needed:
#- name: 'factname'
#  heading: 'heading text'
home_facts: 
- name: 'environment'
  heading: 'Env'
- name: 'zone'
  heading: 'Zone'
- name: 'catalog_config_version'
  heading: 'Cfg Ver'

If I feel really adventurous, I’d like to implement my other big wish, some sort of pop-up list of links, based on arbitrary facts (mainly hostname and fqdn) for each node – something where I can mouse over the node name/table cell, and see links (static URLs with node name/fqdn/other facts plugged in) to things like Nagios/Icinga, our backup system, etc.

Dear Mom and Dad – or, a book about what I actually do

Posted on by

I’ve followed Tom Limoncelli’s blog for quite a while; his books The Practice of System and Network Administration and Time Management for System Administrators were infinitely helpful in the early days of my professional life, and are among the few (literally, 4 or 5) books that live on my desk. His insight and information into the soft skills of SA work – time management, hiring, working in teams, etc. – is not only excellent, but also all too rare in a largely technical field.

Anyway, Tom posted the below article to his blog about a book that recently came out, “Taming Information Technology: Lessons from Studies of System Administrators” by Eser Kandogan, Paul Maglio, Eben Haber and John Bailey. I haven’t read the book yet, and at $56, it’s going to be a while before my book budget recovers enough to justify it. But going on what I’ve read from Tom and others, I want it. Not only do I want to read it, but I want to pass it around to my parents and in-laws and everyone else who has asked what I do for a living, and I found myself at a loss for a less-than-6-hour-long explanation. So, here’s what Tom wrote on it:

Dear Mom And Dad,

Many times I’ve tried to explain to you what I do for a living. “Computer system administrator” or “sysadmin” is a career that is difficult to explain and I’m sure my attempts have left you even more confused. I have good news. Oxford University Press has just published a book by 4 scientists who video taped sysadmins doing their job, analysed what they do, and explains it to the non-computer person. They do it by telling compelling stories of sysadmins at work plus they give interesting analysis with great insight.

Why did they do this? Because businesses depend on technology more and more and that means they depend on sysadmins more and more. Yet most CEOs don’t understand what we do. The scientists made some interesting discoveries: that our jobs are high-stress, high-risk, and highly collaborative. We invent our own tools, often on the spot, to solve complex problems. We are men and women of every age group. It is a career unlike any other. These are things that most people don’t know about our profession. The book is very engaging: Some of the chapters read like the opening scene of “Indiana Jones”; others like “Gorillas in the Mist.” Kandogan, Maglio, Haber and Bailey have put together a very serious, scientific book with care and compassion.

I’m not one of the sysadmins they studied but every story they tell reminds me of real experiences I have had.

I hope you enjoy reading this book. I know I did.

Pre-order it here: http://www.amazon.com/dp/0195374126/tomontime-20

Sincerely your son,
Tom

P.S. In all seriousness, I read a preview copy of this book and highly recommend it to others. You may have seen the authors speak at Usenix LISA or LOPSA PICC conferences where they showed clips of the video tapes they made. The book conveys the same stories, plus many more, with interesting analysis. If you think that the profession of system administration would benefit from non-sysadmins better understanding what we do, I highly recommend you pre-order this book and share it. You can pre-order it here: “Taming Information Technology: Lessons from Studies of System Administrators” by Eser Kandogan, Paul Maglio, Eben Haber and John Bailey

More about the book here: http://everythingsysadmin.com/2012/07/kandogan.html

If you have any interest, I encourage you to go out and buy the book. If you know someone who’s an SA, you should buy them the book. If you can justify any sort of book budget at work, you should buy the book. And while you’re at it, if you haven’t read Tom’s other books, you should buy those too. You might be in the unfortunate position – like I am – of probably never being able to implement most of his suggestions at work, but at least you’ll be aware of them…

SysAdmin Links of The Day

Posted on by

A few links that I’ve had in my “mention in a blog post” category for a while:

Adjusting the VirtualBox F12 BIOS Boot Prompt Timeout

Posted on by

I’m working from home today, connected by VPN. I’m in the process of testing a bunch of Puppet stuff, and needed to re-image a bunch of VirtualBox VMs on my desktop at work, using PXE boot to Cobbler. I’m only connected to the desktop by SSH, and running the VMs with VBoxHeadless and connecting to them via RDP (well, VRDP). The problem with this is that if I start a VM on my console window, then switch to my RDP client and connect, by the time the VM gets keyboard focus, it’s already past the VBox “Press F12 to select boot device” prompt and booting from disk. I could modify the boot order on the VM, but then that becomes a pain when it reboots after the install.

Thanks to some of the guys on the VirtualBox IRC channel, I found out about the --bioslogodisplaytime option for VMs, which controls the length of time (in milliseconds) that the boot splash screen is shown (the default value seems to be 0). It’s included in the reference guide to VBoxManage in the modifyvm section. Setting this to a value of 10 seconds or so, as shown below, is more than enough for me to start the VM, Alt-Tab to my RDP client, connect to the VM, and hit ‘F12′ to select a one-time network boot:

VBoxManage modifyvm VMNAME --bioslogodisplaytime 10000

GNU Screen and Multiple Regions

Posted on by

Since I always seem to forget this wonderful feature of GNU screen (probably one of the pieces of software I use the most every day)…

  • To split the current region horizontally into two equal regions: C-a S
  • To switch between those regions: C-a Tab
  • To close all regions but the current one: C-a Q

This is absolutely invaluable for watching logs on multiple machines at once.

Using Templates to Track Outdated Content in a Documentation MediaWiki

Posted on by

Both my last and current jobs use MediaWiki for internal documentation. As always happens, some of this documentation will inevitably get out-of-date, or totally deprecated. As is also the case, many times when we’re looking for docs in the middle of an incident, we don’t have the time to go back and fix what’s wrong. So, I devised the following template/category system to help keep track of these problem pages.

First, create some templates that you will apply to the problem pages. I use three – one for totally deprecated pages, one for pages that need updating, and one for pages that just need cleanup. For the cleanup template, in the MediaWiki search box, enter “Template:Cleanup” and click “go”. You should be told that the page doesn’t exist, and given a link to create the page. Create it, and enter the following content:

<table class="metadata plainlinks mbox mbox-caution" style="">
<tr>
<td class="mbox-image">
[[Image:Cleanup.png]]
</td>
<td class="mbox-text" style="">'''This page needs to be cleaned up or reorganized.'''</td>
</tr>
</table>
 
<includeonly>[[Category:Pages Needing Cleanup]]</includeonly>

Now we create a category page for it, “Category:Pages Needing Cleanup”, with the content:

__HIDDENCAT__
 
This category is for pages that are mostly correct and just need minor corrections or reorganization.
 
'''To add pages to this category''', include the following at the '''TOP''' of the page:<br />
 
<code><nowiki><!-- REMOVE THIS TEMPLATE WHEN PAGE IS UPDATED-->
{{cleanup}}</nowiki></code>

and save the page.

Now there’s a few other changes we need to make. First, upload the Cleanup.png graphic, which I got from wikimedia.org here and uploaded as Cleanup.png.

If you refresh the Template:Cleanup page, you should now see the image. On a side note, “__HIDDENCAT__” on the category page prevents that category from showing up in the category list at the bottom of the pages we add to it, but this only works in MediaWiki 1.13 and up.

The last step is to add the MediaWiki mbox template and its dependencies. While I did this once before, I didn’t really remember the steps, but I found a post on Glynor’s blog that details them rather nicely:

  1. Enable the ParserFunctions extension. There are download and install instructions on the extension page, but you’ll want to enable string functions. To do this, include the extension in LocalSettings.php like:
    require_once( "$IP/extensions/ParserFunctions/ParserFunctions.php" );
$wgPFEnableStringFunctions = true;
  2. Create a new page in your wiki called “Mediawiki:Common.css”, and paste in the content from MediaWiki.org MediaWiki:Common.css.
  3. Go to Wikipedia’s Special:Export page, and enter “Template:Ambox” in the box, check off “Include templates”, and export the template (and all dependencies) to a local XML file.
  4. Go to the “Special:Import” page of your wiki, and upload the XML file you just grabbed from Wikipedia. This will import the Ambox and mbox templates, as well as their dependencies.
  5. Now, if you go back and refresh the Template:Cleanup page you created, you should see the icon and a nice message box:

cleanup message box

Finally, add the template and category pages for update and deprecated:
Template:Update

<table class="metadata plainlinks mbox mbox-caution" style="">
<tr>
<td class="mbox-image">
[[Image:Warning.png]]
</td>
<td class="mbox-text" style="">'''This page is in need of updating. Some information on it may be out of date, and should not be relied on.'''</td>
</tr>
</table>
 
<includeonly>[[Category:Pages Needing Updates]]</includeonly>

Category:Pages Needing Updates

__HIDDENCAT__
 
This category keeps track of pages that need changes or updates.
 
'''To add pages to this category''', include the following at the '''TOP''' of the page:<br />
 
<code><nowiki><!-- REMOVE THIS TEMPLATE WHEN PAGE IS UPDATED-->
{{update}}</nowiki></code>

Template:Deprecated

<table class="metadata plainlinks mbox mbox-caution" style="">
<tr>
<td class="mbox-image">
[[Image:Critical.png]]
</td>
<td class="mbox-text" style="">'''The information on this page is badly out-of-date.''' It describes a system that is no longer in production or has drastically changed, and '''needs to be updated or rewritten'''.</td>
</tr>
</table>
 
<includeonly>[[Category:Deprecated Content]]</includeonly>

Category:Deprecated Content

 __HIDDENCAT__
 
This category keeps track of pages that are '''seriously old''' or otherwise describe systems/hosts/etc. that have seriously changed from what is described in the page.
 
 
'''To add pages to this category''', include the following at the '''TOP''' of the page:<br />
 
<code><nowiki><!-- REMOVE THIS TEMPLATE WHEN PAGE IS UPDATED-->
{{deprecated}}</nowiki></code>

And the download the two images – http://upload.wikimedia.org/wikipedia/commons/9/98/Ambox_deletion.png gets uploaded as Critical.png and http://upload.wikimedia.org/wikipedia/en/f/f4/Ambox_content.png gets uploaded as Warning.png.

That’s it. To use this, just add {{cleanup}}, {{deprecated}} or {{update}} to the top of a wiki article (adding the HTML comment before it is also recommended), and it will add the page to the appropriate category and show a nice message box at the top of the page:

Cleanup:
cleanup message box
Update:
update message box
Deprecated:
deprecated message box

I also add a link to the top of the main wiki page:

Things that need to be done: [[:Category:Pages Needing Updates|Pages Needing Updates]], [[:Category:Deprecated Content|Pages with Largely Deprecated Content]], [[:Category:Pages Needing Cleanup|Pages Needing Cleanup]], [[Special:WantedPages|Links to Nonexistent Pages]]