Jason Antman’s Blog

After seeing a link to it on the SAGE mailing list, I happened to read Mark Dennehy’sblog post on
Tips for hiring new engineers. I felt the need to make a few comments on some of his findings. Perhaps someone in HR, or a recruiter, will actually read this and learn a thing or two.

The basics of professional work – getting along with others, being able to manage your time, being able to communicate clearly and well, being able to work to deadlines – it is a waste of your time and mine to put these in the job advert. You aren’t taking random people off the street here – you’re hiring trained professionals who’ve working in this field for some time. It’s nearly insulting to tell them not to apply unless they play well with others.

Unfortunately there are some places where this is required. In the public sector, at least, the job ad and description set up fixed guidelines. If you have a job description that doesn’t include “plays well with others”, and the hire doesn’t play well with others, it’s immensely difficult to fire them – the hire’s failing wasn’t listed in the job description.

Don’t have the HR people write the technical requirements. You have engineers, use them.

I have to whole-heartedly agree with what Mark said about writing technical requirements. I don’t know why HR seems to think that they can write IT job descriptions – they wouldn’t attempt it for most other professionals. IT job descriptions should be left to IT people to write – and, more importantly, people who actually understand what the person will be doing.

And don’t class all those requirements as being absolutely necessary. Have two lists – critical, mandatory skills; and skills which would be advantageous to have. Because many professional engineers will look at the mandatory skills listed and if we see some we don’t have, we won’t apply.

Also a very good point. Even when I’m not actively looking for work, I usually get a lot of phone calls and emails from recruiters. Especially if I update my resume on one of the big job sites or post a consulting ad on CraigsList, I get inundated with emails and phone calls (which I usually refer to email, unless they’re from a big player). I can’t possibly spend the hours to read and apply to all of them. The first ones that get deleted are from people who obvously didn’t even look at my resume – entry-level (operator) positions, Windows admins, anything with a primary duty that isn’t even near my skill set (i.e. they just used keyword matching and never read anything).

From there, I start looking through the descriptions and building two lists – the ones that will likely get thrown out unless there’s some amazing thing that redeems them (i.e. a company that I really want to work for) and the ones that I know I’ll follow up on. Here are some of the criteria I use:

Likely to ignore:

• I read the job description and still can’t figure out what I’d be doing.
• The buzzword-to-content ratio is horrible.
• The skills/requirements section lists every hot technology – a list that no human being can master.
• The skills/requirements section has far too disparate of a list – something that only an engineer, administrator, programmer, and hardware designer could master.

Follow-up:

• Use of the SAGE Job Descriptions.
• An actual salary range, not “varies with experience” or “competitive”.
• Some description of where I’ll work – size of the team, responsibility, work environment, etc.
• Examples of current things the team is working on, or examples of what I’d be working on.
• Skills/requirements broken down into mandatory and optional/preferred.
• A description that was obviously written by someone who understands the technology.

In closing, I have a theory for companies (especially those in the public sector that have stringent HR/hiring policies) listing jobs in the IT sector: have both HR and IT write portions of the job description. Have HR write a paragraph or two with all of their non-IT-specific stuff, and then have the IT hiring people (preferably the manager the new hire will report to) write the rest. Put the HR stuff at the bottom. Break the skills/requirements lists down into “Technical Skills” (further divided into Required and Preferred/Optional/Bonus) and a “Soft Skills” section for HR (with the usual crap like “works with a team”, “eligible for employment”, etc.).

Everyone’s up in arms about the recession, high unemployment, and low job openings. Logic would dictate that it’s an employer’s market – and, to a large extent, it is. However, that doesn’t mean that employers don’t need to worry about making the advertisement attractive and descriptive. Actually, I’d say it’s the opposite – since I know that there will likely be hundreds of other applicants, I only send a resume in for jobs that I think I have a very good chance of getting. I’m sure I’ve skipped over good positions just because the description didn’t communicate that to me – and I’m sure I’m not alone.

Ideas and Rants hiring, hr, jobs

I just came by an interesting post on Slashdot, A Public Funded “Microsoft Shop?”. The author works at a publicly-funded hospital and comments that he received an email from management stating:

Information Services is strategically a Microsoft shop and when talking to staff / customers we are to support this strategy. I no longer want to see comments promoting other Operating Systems.

Initially, my anti-Microsoft buzzer went off. But the post also stated that they were ordered to remove Firefox from any computers not specifically authorized by management. As usual, the Slashdot conversation degenerated into a proprietary vs open debate.

As I have to comment on the Microsoft issue, I have two remarks. First, software (an OS, a browser, a text editor, whatever) is a tool. A tool should be chosen base on whether it’s the right one for the job, not just because of who makes it. I’d like to see a major construction company state decree that they’ll only buy Stanley and DeWalt tools. What will their answer be when the plumbers realize that neither of those companies make a simple pipe wrench? “Use a hammer”? Secondly, as is evidenced by history, popularity is a relatively poor indicator of quality, and always ephemeral. Wigs were popular for wealthy men. The telegraph was popular, and many thought the telephone would never catch on. The fluoroscope was popular for shoe fitting. Racism was popular. Smoking was popular. BASIC was the greatest programming language ever. Decisions based solely on popularity are rarely good in the long term.

But, alas, enough of the Microsoft-bashing. What struck me more was the prohibition against Firefox, and what it means for technically-apt employees. Times are changing, and many of the people now entering the workforce are well-versed with technology. The days when employers could expect to give their new hires initial computer training are long gone. And, while many may not see it, the days when every new employee could be expected to know only a common “popular” system (Windows, MS Office, MSIE) are gone, too. Many people who work at universities, such as myself, are seeing browser stats that report less than 40% Windows, with an explosion of Mac-based users and (perhaps thanks to Android, Netbooks, and Ubuntu) a strong growth in the Linux user base.

The Rutgers University student computing labs have both IE and Firefox installed on the Windows machines (and we also have a *very* large number of Mac or dual-boot Windows/Mac clients) and a walk through a busy lab will reveal a strong majority of users on Firefox. Many cash-strapped students, even the ones I knew a few years ago, were using OpenOffice rather than pay for MS Office.

A similar trend can be seen in the new hires and young professionals who simply won’t settle for a corporate cell phone – Windows Mobile, iPhone or Android, they already have a phone and OS that they like, and consider a part of their lives.

There’s a very simple point here – for an increasing number of people, especially those now entering the workforce, technology is an inextricable part of their lives. It’s part of their sense of self, of expression, of free choice. Telling many people what browser they can and can’t use is like telling a new hire a decade or two ago how their handwriting had to look or what size note pad they could use. Asking many of my (even non-techie) friends to switch cell phone OS would be like telling them what color clothes or tie they have to wear to work. Telling the average 20-year-old that they can’t use instant messenger or facebook at work is like telling the average 40-year-old they can’t receive a phone call from their spouse or child. Most especially, with the pervasiveness of Internet access, connected devices and choice in browsers and other software, these choices are being seen as a part of life, a part of technology.

Lastly, and perhaps most importantly to many businesses, the role of IT as seen by the end-user is changing, and the role of technology in productivity is changing. For many young college-educated workers, IT is more of a procurement avenue than a support system. Many would happily install the software application of their choice (whether it is Firefox, OpenOffice, or something else) on their own, without the worry of a formal help desk. There’s also the issue of productivity – technologically proficient new hires are already used to a software environment. They’ve been able to choose their own applications, OS, browser, etc. Forcing them to switch – especially if they have been using an application for years and still do at home – will only result in lower productivity and some amount of frustration. I know that I, for one, have almost laughed when people advertising for Linux admin jobs said I’d be using the same Windows desktop environment as all of the users.

Ideas and Rants microsoft

There’s been a lot of buzz over the past few years about DRM, file sharing, “intellectual property theft”, etc. A lot of that has been the two extreme sides – the media industry and their “have it our way” attitude, and the extremists who feel that everything digital should be freely shareable by everyone. I don’t fall into either of those categories, and I don’t think the majority of people do either.

First, let’s look at a bit of history. In my early childhood (1990’s), cassettes were giving way to CDs, and VHS tapes were the norm for videos. You could go to any corner store and buy a blank cassette tape or VHS tape, and it was widely known that people recorded TV shows or copied audio or video tapes. To cope with this, a portion of the purchase price of every blank tape was distributed among media companies and artists, to compensate them for the copies being made. It seemed that everyone was happy about this – nobody was trying to ban the sale of blank tapes, and my neighborhood video rental store never made me sign a contract promising not to copy a rented tape. There seemed to be a balance between the need for profit and what consumers wanted to do.

That all changed when the world went digital – first audio CDs, then movies on DVD. It requires mention that almost all of the problems faced by the media industry (namely “piracy” and file sharing) were brought by the industry itself. I vividly remember, over a period of a mere two years or so, the transition from VHS to DVD. I remember going to the video rental store (we were late adopters, nobody in my family had a standalone DVD player) and being told that new releases were no longer coming out on VHS. We had to buy a DVD player. This was a format that was pushed on consumers by the movie industry, and was pushed hard and fast. While everyone talked of the quality benefits, it was obvious that distributors were in love with the format’s cheap and quick reproduction. I simply do not believe that the movie industry was unaware (especially given the proliferation of DVD drives in computers) that this cheap reproduction was as easily available to consumers as it was to them. If they were unaware, we must ask how their million-dollar-a-year technical teams never mentioned it. You can’t have your cake and eat it too. The movie industry chose to convert to a format that’s easily copied. The movie industry chose to convert to a format that could be easily read – and copied – on any home computer. They should be forced to accept that choice, and the effect that anyone with a computer can duplicate or share their products. If they didn’t want people to do this, they should have stuck with VHS, or gone to a higher-quality tape format.

But, I digress. The main point that I want to make is about consumer choice, and how that effects purchasing (and sharing) habits.

In my parent’s generation, and those before it, customers voiced their choice through making a purchase or not making a purchase. If they didn’t like a car salesman’s attitude, they’d buy the car from someone else. If they didn’t like the terms of a warranty, they’d buy their washing machine from Sears instead of the local store. If they didn’t like their phone company, they’d switch.

My generation, in the digital age, was faced with a different choice – buy or share. The recording and movie industries more or less made this choice for us. They wouldn’t let us buy how we wanted to, so we made the other choice.

This choice required a bit of a tangent to explain. The industry wants us to think of file sharing as stealing. When sharing digital files, they want us to think of the fact that the file is duplicated (i.e. my friend now has it, but I still have it too). This is simply a side-effect of how digital systems work. Whether right or wrong, whether antiquated or not, in most human minds the concept of stealing is inextricably linked to physical property. Walking into a library and walking out with a book that you didn’t check out is clearly stealing. However, most people wouldn’t think the same thing of photocopying some pages from the book. Most people wouldn’t think of photocopying a newspaper article and mailing it to their friend as stealing. How many people, in the day of audio cassettes, thought of it as “stealing” when they copied a tape for their friend? I’d guess that, for the vast majority of people, file sharing is much more closely associated with these actions than walking out of a record store with a CD.

My personal theory is that a large amount of file sharing (of copyrighted material) would stop if the movie industry would let people buy the way they want.

There was a time, a few years ago, when I got almost all of my music through peer-to-peer file sharing (though, unlike many, I didn’t allow uploads). I never thought much of it – I shared lots of things with my friends, why not music? Then RIAA started their PR and lawsuit campaigns. They started suing college kids for sharing music – and suing them for a lot more than even the cost of the CDs they’d “stolen” (and that’s ignoring the fact that they just “stole” the information on the CDs, so the actual cost should have been lower, less the physical media and distribution costs). So, I heard what the recording industry was telling me: we don’t like you. I stopped downloading music, and I also stopped buying it. For about 3 1/2 years, I listened to what I already had on CD, or the radio, but nothing new.

Then there was iTunes. You could buy whatever music you wanted, usually for less than $1. But you had to use their software, which didn’t run on Linux. And if you wanted to listen to it away from your computer, you had to use an iPod. And you couldn’t burn it to CD, so it wouldn’t work with the older stereo in my car.

Finally, the industry woke up. Amazon came out with their MP3 store, where I could buy individual songs or complete albums, as standard (non-DRMed) MP3 files, that I could listen to on my cell phone, any of my computers, or burn to CD and play in my car. And I’ve been hooked ever since – I get all of my music for a low price, in a standard unrestricted format. I can burn it to CD for my car, put it on my computers at home and at work, put it on my laptop, put it on my phone. Thanks to 1-click ordering and instant downloads, I probably spend more on music now than I did when I had to go to a store to buy CDs. And why? Because I have choice. Because, finally, they’ll sell music to me the way I want it – and I buy it.

I don’t know of any source of unbiased statistics, but I’d venture a guess that since various stores have begun selling DRM-free music online, the volume of peer-to-peer sharing of copyrighted music files has gone down.

But it seems that the movie industry hasn’t woken up to this, the MPAA hasn’t taken a lesson from RIAA. While options are starting to appear – NetFlix streaming and others – they still haven’t made the realization that customers will continue to choose “other” until offered the choice they want. I still can’t buy and download movies on Linux, and since I use MythTV for my home theater, it’s no use to get a NetFlix box. Until offered what they want – a download of an unencumbered, DRM-free movie file, or full DVD image, people will keep sharing movies, and will keep renting them and ripping full-resolution copies.

Finally, it’s worth mention that the secret Anti-Counterfitting Trade Agreement (ACTA) is obviously tilted in the favor of content producers, and has a number of chilling provisions for the Internet. Most importantly, it seeks to reverse previous law and hold ISPs liable for infringement by their customers. Firstly, and I say this with all my heart, this is wrong. Until publishers start successfully suing Xerox for every copy of a page of a book ever made, don’t try and hold ISPs responsible for what their customers do. But more importantly, this is braindead – we should know by now that copyright holders can’t win the cat-and-mouse game. We saw it with p2p and random ports, etc. Trying to detect transmission of infringing material is impossible. Once a new method is invented, it will be bypassed. No matter how many millions the media industry spends on trying to detect violations, there’s simply more people working on the other side, and they’re probably smarter and better motivated as well. If the media industry pushes for ISPs to use deep packet inspection (DPI) technology, the users will just turn to PKI and encryption to hide their data. If ISPs just look at traffic patterns, the users will accept slower download times and shape their traffic to look like web browsing.

If the media industry really wants to stop file sharing of their content (instead of just benefiting from lawsuits) the solution is simple – let consumers buy it the way they want.

Ideas and Rants drm, file sharing, intellectual property, IP, music, p2p, peer to peer, shopping

I read a very interesting article on Linux-Mag.com today. The gist of it is that Microsoft (as happily announced in a press release) has submitted 20,000 lines of code for inclusion into the kernel. Specifically, the code is comprised of a number of drivers that will enable Linux to run better under Microsoft Hyper-V.

Yes, that’s right, Microsoft released code under GPLv2 and is asking for it to be put in Linux. They released it under the license that they call “cancer”. And the entire purpose is, essentially, saying “we want your project to run well as a guest under our hypervisor.

The Linux Mag article did touch on some recent news, such as Microsoft’s lawsuit against TomTom (settled in late March) claiming that the Linux kernel infringes their VFAT patents and the 2004 EU antitrust case (PDF).

A number of things are immediately apparent to me:

• The only reason for this is so Linux will virtualize well under Windows/Hyper-V.
• Microsoft doesn’t seem to be making any similar effort to allow Windows to virtualize well under Xen (and it seems to me that many more people would want Windows on a reliable Linux host than the other way around).
• Microsoft reached a settlement with TomTom, but never did anything to indemnify the Linux community at large.
• This is not a Microsoft endorsement (or even recognition) of the GPL.
• Microsoft made threats about Linux violating “over 228″ of its patents in 2007.

There’s a post on Greg Kroah-Hartman’s blog (he’s the kernel maintainer who will – or will not – eventually be in charge of the inclusion of the code). It should be noted that this all started due to a guy who I really admire, Stephen Hemminger, the principal engineer at Vyatta (whose router product I absolutely love, and their mock advertisements are just as wonderful). Steve has a post on his blog giving the background.

So what do I think should be done? Include the code. But first… (I know Microsoft doing all of this at once would be a dream, but maybe one or two of them would be nice)

1. If they haven’t already done so, Microsoft should publicly recognize the GPL and all of its terms as being a legally binding license.
2. Prior to having any Microsoft code included in the Linux kernel, Microsoft publicly states that the Linux kernel, as of the time they submitted their code, does not infringe on any Microsoft intellectual property.
3. It would be nice of Microsoft would agree to some level of cooperation with the Linux community.
4. Microsoft pledges to allow, support, and actively develop for Windows as a guest under Xen and KVM.

Ideas and Rants gpl, kernel, linux, microsoft

It always amazes me to see how much “old school” web design practice is still out there. I’m talking about commercial sites (not MySpace pages) that blatantly ignore web standards about both content and user experience. This isn’t just a Linux thing, though some aspect of it certainly is. The web site of my home town, mpnj.com uses a Flash-based navigation menu that even the official, proprietary Flash player for Linux won’t support – the transparency renders as white, obscuring the text beneath the fully extended size of the menu. I emailed the developer about this on the launch day, and was told in no uncertain terms that – despite the fact that he had a fully-functional alternate version – Linux wasn’t important enough to fix the site. Ironically for a town government web page, it also doesn’t incorporate any accessibility features, which seems to be standard for most of these poor designs.

There are still countless large news sites whose Flash-based video players won’t run under Linux, and even CitiBank’s credit card site has a flash ad that plays incorrectly under Linux.

The real pain that I happened to see today was a company who uses coupons.com to allow customers to print out retail coupons. My first surprise was that to print the coupons, you have to download Windows or Mac software. I’m not quite sure how many people will do this, but it’s probably how viruses spread so quickly (people who will download anything that claims to get them half a dollar off of a roll of toilet paper, or whatever the coupons are for). So, that’s not cool – most coupons I’ve gotten were just HTML emails or PDFs. If their thinking is to control the distribution (they make some comment about a “paper-based printer, not a fax or PDF creator”), they’ve obviously forgotten about photocopy machines and scanners, let alone capturing the spool file on Mac.

More striking, however, was the shock of opening their help page. My primary monitor is a 24″ widescreen, and I generally keep a browser window occupying half the screen width and a terminal next to it. Once I opened their “help” site, it promptly resized my browser window to a tiny 640×480!

This problem, unfortunately, isn’t as rare as it should be. There are still sites that force browser size, disable right clicks (I hadn’t seen that since about 2004 until a few weeks ago… obviously someone who’s never used `wget`) or have a page that doesn’t fully work in FireFox on any platform. Even worse, my personal pet peeve (as at the time of writing this I have about 50+ tabs open in Firefox, and it’s only using a small sliver of my 2GB RAM) is sites that don’t play well with tabbed browsing – either using only JavaScript for all navigation links, or opening all links (site-wide) in the same tab/window. I don’t know how many web sites have lost my business because of this. Or the one I know of that starts a new shopping cart for every tab opened (so if I open each product I want to buy in a new tab, when I add them all to the cart, it ends up with only one).

I don’t know how there can be anyone out there who’s still not using valid XHTML with all of the accessibility features for anything new, especially a commercial site. But even more so, how can there still be people designing web sites who disregard the golden rule of web design: Don’t mess with someone’s browser. Leave things like where to open the link and how big to make the browser to the user. If they’re not technically literate, changing what “usually happens” will just confuse them. If they’re well-versed in how to use a web browser, like me, they’ll just get aggravated by having someone else change their workflow (I doubt the guys who designed those sites would like it if I told them they had to design the whole thing in Emacs). If they’re somewhere in the middle (just found Ctrl+click in Firefox), you’ll confuse them. And God forbid they’re blind and using a page reader… good luck with JavaScript or Flash navigation.

Ideas and Rants design, standards, web

To keep it short, I’m sure anyone who winds up here has already heard about the recent Microsoft lawsuit against TomTom, alleging patent infringement. Coverage has been extensive, including GrokLaw and Linux Magazine. While the mentioned patents include car navigation technology (at least the names of the patents seem amazingly vague) and FAT . Most of the news stories I’ve read say that it’s “good for Linux” and will never see the inside of a courtroom.

Maybe I’m just a pessimist, but I see the idea behind this as much worse than “good for Linux”. MS chose one company to sue. TomTom just happens to be not only a household name, but also posted a $1.2 Billion loss last year. It seems to me this is more of a FUD campaign than anything else… the best case for Microsoft is that they could strangle TomTom in a legal battle, perhaps force them to go under, and then ensure a media spin along the lines of “Know that company that made the GPS in every car? They used Linux in it, they got sued by Microsoft, and they’re no more.”

While I haven’t always been a fan of TomTom – and am still bothered by the fact that my (stolen, no longer in my possession) TomTom One ran Linux but wouldn’t give me a console or even let me see the filesystem – I’ll be watching this closely, and hoping that the powers that be will not let the angry dinosaur crush a company over a series of patents that are either horribly obvious (anyone other than Garmin having a claim to any GPS-related idea is beyond me) or just horrible (FAT?!?!?!).

On a final note – isn’t it about time that the US finally dealt with this damn software patent thing? Not only does it horribly stifle innovation (not good to do in a bad economy), and I have a hard time grasping the claim that Microsoft’s developers are so all-powerful that they’re the only people that thought of technology X, but it’s about time that the US government got the balls to look Microsoft in the eyes and say, “you’re not the only game in town anymore. Get used to it.”

Ideas and Rants lawsuit, microsoft, patents, tomtom

The external-facing web site and (internal use) mailing list for the ambulance corps is hosted by ROUThost. Not my choice, it was inherited. ROUThost, first off, appears to be a fly-by-night hosting provider that just buys a few boxes in a colo facility. I should have known to raise a stink when they say you need to fax a copy of your driver’s license to get SSH turned on, and that you have to agree – in legalese – not to mess with anyone else’s configs. Well, last night, DNS for the site went down. As in nothing, wouldn’t resolve at all. I submitted a ticket online for ROUThost’s “24×7″ support – by the way, they don’t have a phone number, only an online ticket form. After 2h 34m 40s of downtime, the issue resolved itself and I downgraded the ticket from “critical” to medium. Now, 11 hours later, it still hasn’t been replied to. And my emails to support and management – 2 hours ago – are unanswered.

Once the problem started, I knew the yearly contract with ROUThost was a bad idea – even at $35/year USD. So, given the great experience I’ve had with them as registrar for my myriad domains, I took a look at >GoDaddy’s site. They offer shared hosting at around $4/month (for shared on a Linux box) and are currently offering some deals, so I figured it would be a good idea. I know and trust GoDaddy’s support, and have had an account with them for quite some time.

The ambulance corp’s web site, hosted through ROUThost, does essentially three things; provide a minimal web presence (the whole web root is probably < 1Mb minus the photo albums), five e-mail forwarders for the officers and a GNU MailMan mailing list for internal business. Unfortunately, I couldn’t find anything in their “features” list mentioning MialMan or any other listserv, or even what MTA/MDA they run.

I put a call in to GoDaddy “Sales/Support”. The poor guy had never heard of MailMan, but asked “one of the hosting guys” and was told it would only be supported on dedicate hosting accounts. Not exactly financially feasible for a mailing list with 30 subscribers, maybe 2 messages a day, and a monthly HTTP transfer of under 20Mb. I was told their shared hosting packages don’t include any mailing list/listserv software, though they include every CMS and language known to man. Hell-bent to get away from ROUThost, I then asked if they ran an MDA that supported piping mail to a command, as can be done with .procmailrc. After a brief hold (not to sound cynical, but I’m sure the gentleman was looking up “MDA”) he came back on the line and told me they didn’t. I then switched to problem-solving mode and asked what MTA and MDA they were running. Another brief hold, and I was told “I can’t tell you that”. Speechless for a moment, I asked what that meant; “we don’t give out that information”. Just about ready to begin explaining SMTP headers, I gave up and thanked him for his time.

Ok, so Sales probably doesn’t understand SMTP headers. I’d considered trying to find mail from a GoDaddy Linux hosted box and check the headers, but I figured I couldn’t do that before the call ended. So, now I’m left with a dilemma. ROUThost is not, in my opinion, reliable, and their support is flat-out nonexistent. 11 hours is far too long to wait for a reply to a “critical” ticket when someone claims 24×7 support. However, by previous experience, GoDaddy would be my next choice – but not only do they ot support mailing lists – arguably the most used feature of our current hosted account – but they won’t even tell a customer what MTA they’re running. I’m too let down by this to telnet 25 on one of their boxes and see what happens.

So what’s left? I guess waiting until (hopefully some time within the next few weeks) I upgrade to Optimum static IP at home, and consider running it all there (and hope mains power never goes out for more than 30 minutes?)

Ideas and Rants, Projects, Reviews dns, godaddy, hosting, routhost, security through obscurity, support

On the same thread as the last post, some thoughts on my ideal network, or the hosts on that network:

• One “gold master” installation/kickstart file of a single chosen distro, with a base set of packages, including site-specific packages. (or something like this implemented in a configuration management system)
• All new installations performed over the network in an automated fashion, and from a local repository.
• Software updates are automated (either through a configuration management tool or something like the behemoth I talked about here and pulled from a local repository (perhaps one which mirrors the mainline repos, but only downloads a package the first time it’s requested?).
• Puppet or CFengine on each machine. Better than just having them is having each machine automatically added when it’s created. Even better yet would be to have Puppet or CFengine combined with something like Cobbler, so I can define a new machine in {puppet|cfengine}, list its’ MAC address, then netboot the box and come back in a few hours to have an OS installed, packages installed and the machine configured, monitored in Nagios, monitored for security and backed up.
• Tripwire or some other sort of security software, as well as centralized logging and auditing, on every box.
• A small number of additional “package groups” to add to the “gold master” via config management – something like “web server” (Apache2, PHP, MySQL, log analysis for them, etc.), “development server” (CVS, debuggers, etc.). These would also update the backup system to include appropriate directories, update Nagios configs, etc.
• A good way – if even a human making notes in a per-machine text file – of tracking the “little stuff” like that one cron script that makes everything work right, the location of that hacked-together Python script, etc. A way to easily remember the things needed to recreate a box which aren’t found in rpm -qa or any obvious overviews.
• Bacula or AMANDA setup to backup every box, perhaps with some sort of template system for server types – every machine gets /root and /etc backed up, but web servers get /srv/www and mail servers get /var/mail.
• Nagios setup to monitor everything logical on every box. Perhaps this would use a configuration management engine to handle Nagios configs, so that for example if any Proliant hardware is used, {config management program} will figure this out, install the HPASM packages, put the appropriate check scripts on the box, and update the Nagios configs. Likewise, adding Apache to a machine should cause it to be monitored in the Nagios configs.

Unfortunately, as I’m not independently wealthy, I don’t have the time to quit my job, wipe every machine I own, and start from scratch. But it sure would be nice to be able to, one day, start a server farm from scratch and be able to implement some of these cool things…

Ideas and Rants configuration, dream network, ideal network, management

I’ve been planning a lot of administrative work lately… I have a few machines that need OS upgrades, my backup system is barely functional (it needs both a new, large disk and a configuration overhaul), and I’m planning a switch to static IP service – which means not only a new router/firewall to design, configure and monitor, as well as moving some services previously run by IPcop over to a dedicated box, but also finally adding three IPsec VPNs, monitoring them and tunneling all sorts of stuff over them, and reconfiguring all of my DNS, finding any hard-coded URLs, and a slew of other projects.

So this got me thinking. While there are a number of reasons why I run such a complex network at home (mainly including maintaining my presence on the web and my email, providing temporary hosting for freelance work, the convenience of file access from anywhere, the breadth of administrative experience it gives me, and a way to test new technologies) there are some parts of it that I just don’t like dealing with. I’ve never been a really network-centric guy, and the idea of having to setup a router/firewall (I’m going with Vyatta as it seems to be the only thing that will deal with the complex configuration I want) for all this just to get 5 static IPs seems a bit much. Not to mention there’s just too much running on all those boxen (8 at home, 2 at school, plus 3 others at 2 other locations) for me to keep a handle on all of it and still be a full-time student, work 30 hours/week, and do freelance work. Something’s always bound to get ignored – sometimes backups stop for a week, sometimes Nagios goes haywire, and sometimes Cacti stops graphing for a month before I notice it.

The biggest thing I learned from running all of these systems for personal use is to start everything consistently and with a plan. My oldest box in semi-production is running SuSE 9.3, installed somewhere between April and October of 2005. It was ignored for so long (a period when it wasn’t being used for much) that I now can’t even perform updates, as the update sequence is virtually impossible to accomplish. Then again, re-purposed desktops shouldn’t be in “production” for 4 years. Anyway, perhaps the biggest lesson in trying to deal with all of this is the importance of consistency. Not just attempting to standardize on one distribution, but also making a local image that includes the standard packages, configurations, and other important stuff – like the Nagios user account and local plugins and maybe even the public SSH key of the Nagios box. Even better would be a configuration management system like Puppet or CFengine, or even manually keeping all of the distros updated to a common version.

But, I digress. The real point of this post was supposed to be a simple idea: I have all of this running at home, and I know quite a few IT people who have a similar setup at home or at work, or have considerable resources at a hosting/colo facility. So, why not start a “community datacenter” project? At home I have to do everything from backups to firewall and router administration to security. I’d be much happier just handling network/service monitoring, log analysis, and some tool and web scripting. I know a few Cisco-heads who run their “home” LANs on chassis switches, but find it such a pain to reconfigure Apache or run a monitoring app. I’m sure someone’s thought of this in the past, and probably tried it, but why don’t some guys (who can be trusted) get together, find some colo space (or anywhere with power and connectivity) and start, essentially, a co-op data center? Assuming you could find a large enough circle of trusted friends, I’m sure you could find someone willing to volunteer every service needed – from network engineering to backups, monitoring, and security – in exchange for some rack space and connectivity, or even a virtual host. I know I’d opt in any second – or even let someone throw a box in my basement in exchange for someone to help read through logs or setup a HTTPS VPN, if it weren’t for the archaic equipment I’m running.

Just a thought…

Ideas and Rants community, datacenter

For one of my wonderful classes, Internet Security, I’m doing a presentation on “patch management”. While I’m obligated to cover Windows – and, of course, will talk about MacOS – I’ll obviously be spending a good deal of time on the Unix/Linux side of things. This has gotten me thinking about one of my biggest problems with Linux (and specifically OpenSuSE, my usual default distro. Patch management is utterly awful.

Here’s the problem: I have about a dozen machines under my control. I need to keep them all up-to-date. Currently, I manually do patches and upgrades via YaST or zypper. I thought about scripting this through zypper, but that doesn’t make any sense – the packages on the machines are far from homogenous, so there’s no clear way to make one script that updates them all. I considered using Puppet or CFengine or something of that sort, but that’s too heavy-weight for me – for only a dozen machines, many of which are personal or development only, that’s a lot to keep track of by hand, and a lot of work defining which patches should be applied, and which machines shouldn’t be changed.

My other peeve is distribution upgrades. About three of my machines are still running OpenSuSE 10.0 or 10.1, both of which are unsupported, and no longer even have downloads available. Why? Becuase I’ve done major OpenSuSE upgrades before, broken a LOT of stuff, and I simply can’t risk that on machines that can’t stand extended downtime. This process *needs* to be made easier. Bottom line – it should be made no more difficult or unreliable than a kernel upgrade. IMHO, the biggest selling point for Solaris is its’ ability to do a total upgrade to a second partition, and switch-over at runtime. Why doesn’t Linux (or SuSE) have this yet?

What’s my ideal solution? A curses application that uses text-file backends (curses so I can run it over SSH even if I have a slow link or high latency, like from a SSH session on my cell phone, if need be). The app would allow me to list all of the machines I want managed. It would connect to the machines over standard SSH, and would leave an extensive audit trail of what’s done, both on the management console and on the machines (as well as running as a dedicated user). The application would maintain an inventory of all of the packages on every machine. It would check daily for new patches/updates to any of those packages, and e-mail me a daily summary of what’s new, including all dependency changes, and which machines need the update. It would also allow me to define, on a per-machine (or per-group-of-machines) basis, rules for packages that must stay at their current version – i.e. I have a bunch of PHP4 apps, so machine X needs to stay at PHP4. The e-mail summary would include any packages that aren’t going to be updated for a specific machine because of dependency/version rules, as well as warnings about any new packages that have a dependency that has a rule set. I could then run the main curses app on my admin machine and, starting from NO selections, select which updates I want to apply and whether I want to ignore or create new rules to keep something at its current version, on a per-machine or per-group basis. This curses app would generate a file (XML?) of what to do (which would also be generated or edited by hand, easily). The XML file would then be fed into a script that downloads all of the needed packages to a central (local) mirror (or, optionally, for remote machines, has them download locally on the machine), checksums them, and then installs them (running commands over SSH) on all applicable machines. It would then keep a log of all changes, both on each machine changed (in a master changelog file) and on the central administrative machine. Most importantly, the curses interface would have a simple, quick way to back out any specific update or group of updates for all machines, a group of machines, or one machine. All data needed to back out a change would be kept on each machine (say, cleaned up at the next update of that package and all of its’ dependencies) with machine-readable instructions kept in a central file, allowing local rollbacks – i.e. a machine goes down, I realize that it was because of an update to package X, and on the local machine I can check the changelog, see an entry like “Package X updated 1.0.0 to 1.0.1 on yyyy-mm-dd, Change ID 1234″ and then, to rollback, simply issue a command like “patchmgt rollback 1234″ on the effected machine.

Just some ideas, and a little rant.

Ideas and Rants linux, patch management, solaris, updates, upgrades