Reg HPC man relives 0-day rootkit GROUNDHOG DAY
HPC blog This is a difficult article to write, and I’ve put it off for way too long. But it’s time to bite the bullet and make an embarrassing admission to the Register audience. I’ve been hacked and hacked hard.
Admitting this publicly to Reg readers is like chumming shark-infested waters with my own blood. Or like telling people that I have a lice infection: sure, some will say that being infested with lice doesn’t have anything to do with your personal hygiene, but who believes that?
When it comes to computing hygiene, I think I run a fairly tight ship. Every system on our little network is up to date when it comes to operating systems (Windows 7) and applications – including updates and patches.
Every system also has a full, and up-to-date, security suite which includes antivirus, anti-malware, anti-spyware, and anti-everything else. Our network is protected by our ISP’s firewall and security infrastructure plus our own name-brand hardware firewall.
I’m also pretty careful about what I download and install on any of these systems. I only download apps that I’ve checked out beforehand, from locations I trust.
Even with all that…
I got hacked. I first noticed it just after Christmas. My main business system suddenly started playing random audio. It sounded like snippets from radio or TV broadcasts, including short news blurbs, commercials, and bits of music.
I figured the problem was something associated with my browsers. I typically have a couple up at any given time, all with multiple open tabs. Sometimes embedded videos, commercials, etc, will play when the sites automatically refresh. Restarting the browsers seemed to do the trick… at first.
But it came back. Even after locking down the media options for webpages. Then it started happening even when I didn’t have a browser open – or any applications at all. Oh-oh.
I did my routine checks: full system scans (no problems noted), looking at the processes and services that were running (nothing unusual), and paring back programs that started on boot to see if that made a difference.
I also took a hard look at any programs I had installed prior to the start of the problem, uninstalling them and deleting all references to them just in case. But I just wasn’t able to find the offending program or process.
Then it was on to Google to see if anyone else reported this problem and how they fixed it. There were a few examples, and all signs seemed to point to some flavour of rootkit, which made my blood run cold.
My Problem metastasises
My first move was to check my other systems to see if they exhibited the same behaviour. I found that my laptop had the same problem as my main business system, which could mean that I’d picked it up on the road.
I immediately stopped using email and stopped using these systems to access our shared NAS. I didn’t want this thing to spread any further. And I certainly didn’t want to become a carrier by spreading it to clients or business contact via emails, documents, or any other befouled form of communication from my sick machines.
Attempted cures fall short
I tried most, if not all, of the suggested procedures that worked for other users. I downloaded the recommended rootkit detectors and put them through their paces. None of them found my rootkit or detected the problem I was dealing with.
(I’m not naming names here. I don’t want to slime them just because they couldn’t find my particular rootkit. I have reason to believe that my infection was somewhat unique at the time, or at least rare, and you can’t expect any tool to be 100 per cent accurate all the time. But rest assured that I tried the most prevalent and best-reviewed tools out there.)
Bringing in the big guns
All of the above steps took time, and I was falling farther and farther behind on work while trying to fix it. I finally threw in the towel and brought in the professionals. All of the major security firms have subscription or one-time services that almost guarantee a fix for your system. They use a remote agent to take control of your PC and apply hardcore tech antibiotics.
I cut a deal with one of the largest and most respected security firms, signing up for around $100 bucks to start up a multi-system subscription. Since I could cancel the subscription after the first month, I figured it was a good deal if the virus had spread to other systems in our little infrastructure.
They got to work the next day. I booted my system and stood by while they loaded tools, ran scans, etc. I could watch what they were doing and noticed that it all looked pretty familiar – full system scan, looking at processes, running rootkit detectors, etc. I finally got bored and wandered off to work on my goal of doubling my body weight in less than six months.
After they logged off, I fired up the system and was greeted 10 minutes later with exactly the same random audio.
The next day was the same routine, but with second-line support – smarter guys, I guess. I booted up, they played around for a few hours, and then shut it down. I reboot, find the problem is still there, and call them back. The second-line tech confessed that he couldn’t find a virus or anything out of the ordinary on my box.
It was time to bring in the A Team – third-line technicians. Guys who have Jedi-like powers over all things virus-related. They’d beat the hell out of the virus, and for no extra charge, find the guy who wrote it and kick his ass for me. But that would probably have to happen tomorrow or the next day, because they were busy.
After some bureaucratic snafus, I finally got to talk to third-line tech guy. His voice had that world-weariness that comes from having seen too much darkness in too short a time. We talked about Trojans, rootkits, data hostage schemes, and other disasters. He was definitely the right guy to tackle my problem.
He did say one thing that was very disquieting: “Dan, I do have to let you know something. I’ve seen these symptoms once before, and I wasn’t able to fix or save the system. So be prepared for that, just in case.”
Huh? Wasn’t able to save the system? What the hell? How well could this thing be hiding itself? And how did it get in?
Turns out that there are things that even a Jedi can’t do, and these include identifying and fixing some rootkits. Even figuring out how the rootkit got in can be beyond their powers because, according to the Jedi, rootkits can be piggybacked onto normal software updates and the like. Once inside, they burrow deeply into the morass of files, often disguising themselves as something innocuous.
After delivering that warning, he got to work and promised to call me when he had any news. After a few hours, I finally went to check on the system and found an open Notepad session on my desktop. It read:
“As I mentioned, this is a brand new infection that not even our most advanced tools will clean. I would recommend using a previous image as we discussed to put the pc back a few days. Please give us a call if there is anything further we can assist with.”
And that was that.
Rebuilding my tattered life
The next steps were straightforward, albeit hugely tedious. The first was to call Big Security Company, cancel my security subscription, and get a refund for my initial payment. Then began the process of going to my backups, putting an image on a clean hard drive, then testing to see if this new image had the same virus as the old one.
The virus’s erratic behaviour – sometimes starting right up with the inane audio, other times waiting for as long as an hour or two before tormenting me – made a long job longer. Couple that with not being able to pin down exactly when the virus first emerged, and you end up with a long and tedious job.
Fortunately, I have firm backup procedures in place. Every key system is backed up incrementally daily, with a clean image saved weekly. All of these backups are stored for 60 days just in case of, well… this.
It took what seemed like forever to find an image that didn’t have the virus on it. I ended up going back several weeks, which made quite a bit of work and raw materials (video, notes, etc) disappear. Those files had to be brought in individually and tested, just in case one of them was the virus carrier. That process accounted for another long period of time… sigh.
Targeted or unlucky? Plus lessons learned
I kept wondering how it happened and how I could prevent it in the future. Since this was a new virus, according to Big Security Company, why were my well-protected computers the first to get hit?
Did the Trilateral Commission finally decide to even the score with me? Or the Yakusa? Russian mobsters? Or someone who was just looking to mess with me? It could have been all of the above, or maybe it was just my time to be a zero-day guy. I do have some slight grounds to suspect that I was targeted and if I find out more, I’ll let you know.
What did this experience teach me? To be more fearful than ever. I still don’t have any idea what I was infected with, how I picked it up, or how to prevent it in the future. According to third-line tech Jedi at Big Security Company, this is the world we live in today. Our cyber safety is under constant attack, and the bad guys have the first-mover advantage.
The most valuable lesson? It pays to back up, and an investment in fast and solid NAS boxes (shout out “Thank you, Synology!”) is worth every penny. My terabytes of backups saved the day and got me back in business. Without them, I’d be looking at clean installs of everything and then a file-by-file inspection and test of all of my stored data. Yikes.
I don’t want to think about how long this process would have taken if I were trying to do this number of restores via the cloud. I have a fairly fast pipe into the home office; it typically tests out at 20 Mbit per second. But when you’re talking about full-sized images of around 150GB, it would take anywhere from 15 to 18 hours to complete a single download. My local NAS was able to copy these images over in half an hour or so.
Now that I’m back, it’s time to start dealing with the backlog. I have plenty to tell including some interesting and compelling experiences at the SC13 supercomputing conference, stories on my trip to the second annual South Africa Student Competition, info on the upcoming ISC’14 Cluster Challenge, and the usual HPC industry happenings. Stay tuned.