If only Dante were here to help document this….
I get back from Hawaii, throw down my last two posts in a jet-lag induced afternoon binge, do a little sleeping in, work from home on Friday, enjoy the weekend, roll back to the office on Monday the 16th to find… two urgent voice mails from adminstrative folks in the CS department alerting me that my blog site has been compromised. OK, what does that mean exactly?
Well, first of all it means we’re unbelievably stupid (by “we” I mean first and foremost me, and secondarily everyone in my lab that had anything to do with helping to set up the blog site that you see before you). Well, okay, so we’re not so much stupid as naive to yet another in a ridiculously long list of Internet abuses available for heapage upon innocent netizens like us. Excuse us for acting like this is a safe neighborhood.
What happened was we got up close and personal with splog. Here’s the approximate play-by-play.
The first big problem was that we had installed the multi-user version of WordPress. Why did we do that? I teach a class called Computers and Society, and I have students deliver their thoughts and reactions as short posts on actual blogs in the actual blogosphere. It’s an interesting experience for students to submit their homework to the world where the instructor and TA are two of a potentially larger number of random readers (including the entire class). Strangely it tends to generate higher quality work.
We’ve tried different approaches in the past, but this Fall I was determined that we should host blogs on our server for any students that didn’t already have one, and that we would make the process for them to set up a blog very easy (courtesy of multi-user WordPress). What we failed to grasp was that this was very much like going into a really bad neighborhood, leaving your front door wide open, the keys in the ignition of your car, and a sign on the front lawn reading, “FOOD IN THE FRIDGE!”
What happened next was that someone received an email from a stranger in the blogosphere suggesting that maybe our site had been compromised. Painfully obvious after the fact what had happened. Splog bots had stumbled on my blog, saw that WordPress was powering our world, checked to see if there was an easy way for a stranger to just launch a blog here. Sure enough. Paycheck loans? No problem, we’ll host that blog. Viagra? We’ll take two. Other… um… stuff… Sure why not?
By the time we became aware that things had become extremely dumb, we were hosting more than 300 blogs on our server on every topic imagineable (or unimaginable, as the case may be). We thought there were protections on automatic creation of blogs, like admin approvals, or at least email notifications, but nobody had seen any notification. The system hadn’t alerted us and we hadn’t noticed.
The next steps were pretty obvious. Take down the server to be sure it wasn’t compromised. Then bring it up except for Apache while we figure out just how big the problem was on the blog front and explore possible solutions. Ultimately we migrated the blog over to a single-user version of WordPress, gutted all the skanky content living on our server, and brought Apache (and the blog) back up.
The next realization was really shocking. In an attempt to determine the broader damage, I did a search for our server domain and found 84,000 Google hits (and for the most part they weren’t pretty)! Ordinarily Google power rankings are a desirable thing. But when the good names of your server, your department, and your university are being dragged through 84,000 mud puddles, it’s a really really bad thing. (As an interesting side note, MSN only had around 400 hits for the same domain, and Yahoo! had only 18! Not sure what that implies about the relative effectiveness of the search engines.)
We made a request of Google that the contents of our server domain be purged from their cache, which they quickly responded to. Apparently it’s a common enough request that they have an automated system for doing it. That dropped the directly hosted garbage from Google’s cache, but it didn’t do anything for every other splog or spammed blog on the planet that pointed to our server with a promise of replica watches or worse. After a week, those links have begun to weed out, and we’re now down to around 69,000 Google hits for our domain. Er, our former domain.
Because of our sense of the damage to the reputation of our server’s name (not to mention the unfortunate association of numerous inappropriate topics to our university domain name) we changed the server name to sequoia (after the new software engineering lab name). That doesn’t help the fact that there are still thousands of sites indirectly associating BYU with all this garbage, but we didn’t want live searches for my blog to turn up the garbage still in the cache.
We’ve since been fighting redirection to try and get traffic to the right places when people look for this blog. Some of it works right, some still doesn’t. You may see sequoia.cs.byu.edu in the URL, or you may see okoboji still. We’ll sort it out eventually.
Meanwhile, my next move is probably to grab an entirely new domain (yet to be determined) and move the blog there for permanent safe keeping with a well-secured single-user version of WordPress, and a fresh reputation.
As President Bush said… “Fool me once… Shame on… shame on you. Fool me… can’t get fooled again.”
