Jason Antman’s Blog

As has been in the news (and advertised to high hell) for the past few days here in NJ, Cablevision and ABC are involved in a feud. Cablevision is advertising everywhere – I heard at least 4 ads on the drive home from work about how ABC is making unreasonable demands, asking for $40M, etc.

Well, then I found something more interesting. I stopped at a friend’s after work and turned on the (Cablevision) cable box. Strange. It was a full-screen ad from Cablevision about the ABC situation. And it seemed to go on forever. But… huh? The cable box was tuned to channel… 1999. I changed the channel, and got regular TV back.

But when I got home and turned on my cable box – which was last tuned to channel 47 and is setup to resume on the last channel it was tuned to – it also started up to the Cablevision ad on channel 1999.

To anyone who’s read Jonathan Zittrain’s The Future of the Internet and How to Stop It, this is as clear an indication of the problems with appliancization as anything.

Don’t take this lightly. It has some serious implications. Specifically, the basics of what Cablevision did:

1. Pushed out a new software version to all of their cable boxes, very quickly, that added a new channel.
2. Changed an arbitrary setting in the configuration of their customer’s boxes (the channel that displays at power on) remotely.

The idea that Cablevision can control all of their customer’s boxes remotely, and this quickly, is a bit disturbing. If any of you have ever checked out the diagnostic mode on a Scientific Atlanta box, you’d know that it’s more or less a full computer (the “channels” are actually streaming media over IP) and the amount of control that Cablevision has is virtually limitless.

(The automatic channel change has been confirmed by a number of other people.)

Miscellaneous Geek Stuff appliance, cablevision, freedom

Just found this today on one of my all-time favorite sites, XKCD. I swear it’s about me:

Miscellaneous Geek Stuff humor, sysadmin, XKCD

In my daily reading of Tom Limoncelli’s blog, I happened by a post with a link to an article, “Geek behaviors present during conversations” by Phillip Guo. The article is quite interesting, and describes quite a few phenomena that can be noticed when conversing with a geek. One of the parts that immediately stuck out in my mind, though, was:

Using technical terms without checking for understanding

When geeks try to teach non-geeks about technical matters, they often
make heavy use of technical terminology because it’s the most concise
and accurate way to convey their thoughts. Unfortunately, many
non-geeks cannot understand such jargon and thus grow confused or
frustrated.

Non-geeks might perceive geeks as being snooty and elitist by talking to
them using cryptic ‘geek-speak’, but in fact, I suspect that their
intentions are more innocent: Geeks often don’t remember what it was
like not to have an understanding of technical concepts, especially
related to their area of expertise such as computers or electronic
gadgets. Thus, when communicating with non-geeks, they often throw
around technical terms without first tactfully checking to see whether
listeners know what these terms mean. In the geek’s mind, technical
jargon (e.g., router, dongle) and especially acronyms (e.g., DVI,
TCP/IP, SQL) sound as natural and colloquial as regular English
words. “How could anyone NOT know what these words mean?”

While I’m definitely guilty of throwing jargon into explanations when I’m at least half-conscious that the other party doesn’t understand it, I’d say that it’s usually (for me and many who I know) for a different reason. Or, more accurately, a combination of reasons that I’ll call “the expectation of effort” and “the Google effect”. Firstly, among most geeks (and definitely those of us in the open source world) there’s an expectation that people are interested, are curious, and will research something that they don’t know about when they hear it. The second part is what I call “the Google effect”; most of us geeks do most of our communication (especially about technical things) at a computer, whether via email, IM, forums, or chat. Therefore, there’s usually an expectation that if someone doesn’t understand a term, the answer is as easy as Ctrl+T and typing the term into Google.

I also must take exception to one other item in the article:

Favoring complexity and detail over simplicity in descriptions

When a geek is describing his latest pet project, he often tries to make
his descriptions sound complicated and detailed, in order to convey
the sophistication of his technical prowess.

I’ve described plenty of my projects to plenty of people, and yes, my descriptions are generally very detailed. But it’s not out of any desire to convey sophistication or prowess – my description may be the same whether I’m telling it to someone who can’t operate a computer or someone of much greater skill than my own. It’s simply the burning need to be accurate and complete. Put as simply and generally as possible, if I’m describing a project, I don’t want whoever I’m talking to to ask questions when I’m done describing it. It’s like documentation – if I were writing a man page, whoever reads it shouldn’t be left with major questions to ask when they finish.

Miscellaneous Geek Stuff geek, interaction, jargon, social

While I’ve read and really appreciate Tom Limoncelli’s Time Management for System Administrators, the current state of my life (mainly that it’s split between work, personal projects, a freelance client, administering the systems of a the ambulance corps and real people, and that my “work day” is whenever I’m awake) has prevented me from really implementing most of the advice. However, I do try to be as productive as I can.

Without getting into details, a few weeks ago, $WORK suffered a major electrical failure that required everything in the data center to be powered down. This happened around 10:30 AM, and the majority of groups simply powered down their machines and left, planning to return around 2 AM (the estimated power restoration time). After getting our machines down and stopping for pizza, I remembered how much of a pain it was to work in the racks bringing everything down. While my group only has two racks, we’ve had a lot of changeover lately, and the cabling had gotten quote messy. Noting this, I mentioned it to my two higher-ups, remembering that we had a stock of assorted length patch cables. We were able to make an “emergency” run to our cable vendor and pick up a box of 1- 2- and 3-foot power cables.

While everyone else was home or in their offices dodging the pieces of falling sky (everything was down including VoIP and mail), we were the only group getting real productive work done in the data center. The power failure, rather than a catastrophic event, was a great opportunity – the only time we could pull every cable in a production rack and re-do all power and patches.

So, here’s my SA tip for the day – everyone has some big projects that they’d like to do, require downtime, but aren’t critical enough to schedule something. So, keep a list of these and have the parts on hand. Whether it’s a “just in case” hardware swap-out, re-patching, or anything else, eventually you (depending on the environment that you work in) might have one of those times when the solution to the problem is out of your hands and there’s nothing else to do. Use this time productively.

Miscellaneous Geek Stuff sysadmin, time management

So it’s been a very busy day. I was up until 5 AM or so working on implementing Puppet at home. I’m building two new boxes – a storage (centralized home directory)/syslog (to MySQL) server and a second web server (possibly also to handle Nagios) – and I decided that they’ll be totally built by Puppet. The only thing I had to give up on was setting up the NFS share for my home directory on the new storage box and installing and testing rsyslog on it.

This afternoon around 7, I started on my weekend projects for the ambulance corps – setting up Nagios to receive SNMP traps from the APC UPS and moving over to the new Vyatta-based router (from m0n0wall). I’d attempted the router before, but had to rollback – I’m using an old BlueSocket controller for hardware – it’s just a nice black 1U enclosure with a stock Intel motherboard, 20GB HDD, 512MB RAM and three 10/100 NICs. The first time, I was unable to get link on either of the two NICs I was using, so I decided to rollback.

Nagios SNMP Traps

I found a good starting point for Nagios SNMP traps on the OpsView blog. I setup `snmptrapd` on the Nagios server and hacked together a little Python script to just write all of the traps to a file. After some testing with `snmptrap` on my laptop, I did a test by pulling the power plug of the UPS, waiting about 30 seconds, and then plugging it back in. Sure enough, the little old AP9605 PowerNet SNMP card generated two SNMP traps – one for power loss and one for power regained – both of which showed up in the test file

The next step will be deciding how to get the traps into Nagios – specifically whether I want to go with something heavy-weight, like SNMPtt that can handle other devices, or whether I want to code a simple script myself just to deal with the APC cards.

Router

The main reason why I wanted to make the switch from m0n0 to Vyatta was to ease the setup and maintenance of an IPsec tunnel from the ambulance HQ to my house, so I could push backups (relatively small) over the WAN to my infrastructure (or, rather, have Bacula pull the backups). Another big bonus was finally having a way of configuring and checking things through SSH without having to port-forward a web GUI. Another bonus of having a real Linux system under the router is the ability to make custom Nagios check scripts and easily execute them. Something I hadn’t thought of – but became obvious during the switchover – is the ability to run full-fledged `tcpdump` on the router itself.

After building the new config myself, and confirming that the system ran in isolation, I moved it over to production. The first issue was a bit of a thinko on my part – the interfaces on the BSC are actually arranged on the back of the box like eth0—–eth2—–eth1, so I originally had the LAN uplink in the wrong interface. After correcting that and waiting for the network to stabilize, I noticed a total external connectivity failure. After some troubleshooting – thanks to tcpdump on the router – it occurred to me that the (ancient) cable modem needs to be rebooted when the router MAC changes.

I honestly don’t remember the other problems that I ran into, but eventually I ended up getting almost-full functionality – and then a total network outage. A tcpdump on my laptop showed some really really weird BOOTP traffic with addresses of 255.255.255.255. After doing some troubleshooting and monitoring port counters on the switch, I narrowed it down to coming from a single Windows box and the wireless access point. After shutting off both ports, things seemed to stabilize. I also had some “martian address” issues with one of the boxes, but decided to roll the box and that solved it.

Over the next day or so, I’ll be reconfiguring Nagios both at home and at the ambulance corps to cope with the changes and add in the requisite monitoring, and keep an eye on things. Assuming all goes well, I’ll power down the old router on Sunday.

On the home front, I’ve moved over from my old storage machine to the old one – essentially just the NFS mount, and moved over a tarball of everything else. I also added a 1000Base-SX card to the new box, though it appears that I’m out of fiber patch cords. The old storage box was brought down for the first time in about 3 years (aside from brief outages for hardware upgrades or array rebuilds). Assuming I got everything off of it, it will be relegated to the spares pile.

I’m going to make a serious effort to post on a daily basis, if only for my own future reference. I should have the demo of RackMan out soon, and I’m also about to start on integrating it with Nathan Hubbard’s MachDB as well as a PHP script I wrote to pull port names and MACs from Cisco switches and associate them with NICs in machines. Hopefully I’ll also have some interesting Puppet stuff out soon.

Miscellaneous Geek Stuff apc, bluesocket, Nagios, snmp, vyatta

I know I’ve been letting my blog die off a bit lately, mainly due to the giant amount of work I’ve been doing. I plan on updating a lot more over the next few weeks.

I happened to get a comment on one of my posts today, so I decided to take a peek at webalizer and see how things are doing. Apparently, I’m getting 300+ hits/day, including a quite nice array of search queries. Unfortunately, due to the way I redid my logging infrastructure, I don’t have any historical data (and it seems like the webalizer history is still broken, as I’m only seeing data for this month). But cool!

Miscellaneous Geek Stuff

This is by no means rare, but I happened to be checking through my spam folder before deleting the contents (I used to just do this once a week or so, but apparently since jasonantman.com is up and running correctly and listening on port 25, the volume has increased a bit) and came by this email:

Subject:
Dear $TO_EMAIL 17.3.2009 84% 0FF
From: Best Price <$TO_EMAIL>
Date: Tue, 17 Mar 2009 16:45:54 -0400 (EDT)
To: <$TO_EMAIL>

Wow. I know that spammers aren’t exactly world-class programmers, but this seems like something that anyone who’s been in CS101 would realize is wrong….

On the positive side, maybe I should just add a SpamAssassin rule to trigger for anything with “$TO_EMAIL” in the headers. Not that I’d need it in this case, as the message had a score of 21.7 and my threshold is 5.0….

Miscellaneous Geek Stuff lazy, spam

It seems like it’s taken forever. I don’t know how many times I’ve struggled with web apps that I wrote for the ambulance corps to try and get printed output correct. I’ve done everything from generating PDFs, doing screen captures of a browser and printing as a PDF, to things as crazy as trying to print schedules directly from the server, where they’re setup as postscript.

I just happened to be going through my news in Google Reader and found some interesting W3C draft standards for CSS Print Profiles and CSS3 Paged Media Module. Finally, after waiting for literally years, there will (from the way it looks) be a way for me to specify, in CSS, the page size – and more importantly, orientation – and have something print correctly.

Perhaps the end of me having to manually print ambulance corps rosters, and then set the print settings back to landscape and 73% scaling for the schedule, are finally going to be over!

C’mon, Mozilla, I’m counting on a quick implementation once they’re finalized.

Miscellaneous Geek Stuff css, printing, standards, w3c

Shawn Powers wrote an interesting article on LJ, Where Do YOU Send NetBook Users For Help?, which speaks about the myriad different distributions used on NetBooks (like the eeePC and its’ brethren), many of which are significantly different from the usual mainstream distros.

My comment:

From a vendor’s point of view, it probably seems a good idea. However, there are two major problems that I see in it. Firstly, while there aren’t a gigantic number of people that can give aid with Linux, there are quite a few – especially in one of the hottest areas (that I’ve seen) for the eeePC, colleges and universities. Unless the vendor is prepared to offer high-quality OS and application tech support, they should do all they can to make use of whatever Linux support resources already exist. Ubuntu is becoming increasingly common, so the best move (in my mind) would be to re-brand Ubuntu, but keep the functionality the same, therefore making use of (arguably) a relatively large experience base, by Linux standards. If you’re a hardware company, and don’t focus on providing (software) technical support, the smartest thing to do is to try and maximize the amount of third-party software support that’s available.

Secondly, and perhaps more importantly, we’ve finally started to reach a time when Linux is becoming standardized. Most popular distros follow (more-or-less) the LSB specification, and Gnome and KDE have become the accepted standards for pretty much all non-geek graphical environments. It’s taken years to reach this point, and the introduction of mass-market netbooks, many with their own customized distros, is not helping, nor is it smart on the manufacturers’ part.

Perhaps the biggest problem is that the manufacturers themselves just see Linux as an OS that doesn’t cost them anything, without seeing the big picture. If they truly understood Free software, they’d be working to improve on existing technology, while retaining the existing code as a base, rather than struggling to differentiate their offering. If I was to start selling Linux-based netbooks, I think the solution is quite simple, just by following the trends in the Linux world: sell a “business” version that runs RedHat or CentOS, maybe SuSE, and a “consumer” version that has Ubuntu (which, as far as I’ve seen, is the most popular with the non-linux crowd). Rather than trying to build my own distro and differentiate it from everything else, I’d build on what little Linux experience the general public has, by using the most common distros. If the marketing guys want differentiation so badly, that’s the wonder of Free software – just add vendor-specific logos and a custom theme.

On a personal note, I’ve had an eeePC 701G Surf since December 2007. The *first* thing I did when I got it was create a 4GB SDHC card with a full install of my favorite distro, and set it up to boot from the SDHC. I haven’t booted Xandros more than twice. Even the “expert” mode, which is somewhat like a normal desktop environment, is severely lacking in common tools, administrative tools, and security (the user separation is abysmal). I would’ve been much more happy to see it ship with a good install of Ubuntu, even locked down with a “simple” desktop for the default user. Things like sed, awk, grep, and an SSH server should be instaled on every Linux system. Also, just a theory – set each system to have a unique, randomly generated root password, and print it on a label on the bottom of the machine.

Miscellaneous Geek Stuff

I don’t know if I’m the only one experiencing this, but this is the third time in two days that I’ve gotten a 503 when trying to view my bookmarks on del.icio.us. This is posing a problem, as I use the service to keep my bookmarks in sync between five computers.

As of the time of this post, I’m able to get to the site again, but I was unable to get there – or update my bookmarks on my main workstation at work – for about ten minutes.

Miscellaneous Geek Stuff bookmarks, del.icio.us