It happened. It has taken me a couple of weeks to get around to writing about the incident but finally I have a few minutes to put things down for posterity. In a way I am kind of glad it happened because I think I had gotten quite a bit lackadaisical when it comes to protecting myself online. But, let’s not put the cart before the horse.
On August 15th I noticed something was wrong with my blog. There was a bunch of what looked like jumbled code at the top of the page. My initial assumption was that I had updated a template and it was somehow messed up. I tweeted the problem as soon as I noticed it.
I didn’t really get a response for my plea for help and it didn’t take me long to just decide to restore the entire site from backup. The restore went well and the site came back to its normal self. Everything seemed fine and I continued on with the day.
It wasn’t long though until I noticed a second problem. I was getting fake anti-virus messages on my PC. I had seen this before and wasn’t too concerned. I proceeded to clean it up and assumed all of my problems had been solved. What I failed to understand, however, is that the two incidents were directly related and signified a bigger issue. In fact, I kind of joked about the whole thing.
Two days later I had someone report that they had seen girl on girl porn pictures on my Facebook account. As of this writing I still can not confirm this report but I took the obvious step of changing my password at the time. Only one person ever told me they saw anything strange but I think it is worth reporting regardless of whether or not anything was wrong.
By the next day, however, I had grown suspicious of things. I have been working with desktops long enough to know that once a PC is compromised the only real way to clean up the problem is to wipe it clean and start over. I guess things were slow at work on he 18th so I decided it was a good time to format and reinstall my work PC. It was upgraded from Windows Vista to Windows 7 and I felt it would be good to get a fresh install of Windows 7 on the machine anyway. So at some point on the morning of the 18th I commenced the re-installation of my desktop.
Several days passed and I slowly restored my PC to complete working order. Then, on the 22nd of August, the other boot fell. I checked the office website just to test a notebook’s wireless access and what do I find but the same seemingly jumbled code at the top of the site. It was at this point when I realized something really bad had gone wrong and I went looking for answers.
I called the ISP that hosts the site and between the two of of we figured out what had happened. What follows is a timeline / itemized explanation of what we think happened. I may not explain everything correctly so if anyone out there wants to make corrections or fill in some details please leave a comment.
1. Somewhere around August 14th I hit a webpage that was compromised with and used some kind of exploit to drop a piece of malware onto my work PC. What is interesting here is that my virus scanner, which was up to date, totally missed it. A site which I will mention later reported that most scanners did not pick up this virus until well after the outbreak.
2. The malware executed and first read several files that contained all of my ftp sites and passwords. I only had two sites listed which were this blog and the corporate site which I administer. I now know that popular FTP tool, FileZilla, stores passwords in plain text by default. Any person or program with access to the files can completely compromise a website using this information. This is a HUGE SECURITY HOLE AND SHOULD NOT BE TAKEN LIGHTLY!
3. Once the site information was passed to somewhere in eastern Europe (according to the ISP’s research) all index files as well as many other files on the two sites were downloaded and injected with malicious code. The interesting thing here is that the bad guys screwed up initially and that is why the code showed up at the top of the page when accessed by a browser. Obviously this was not intended and they later re-injected correct, malicious code into the pages after discovering their error. They didn’t delete the bad code, however, and it is the bad code that alerted me (and thousands of others) to the problem because it rendered in the browser.
4. Without trying to get into specifics, mostly because I don’t really understand the technical details, the malicious code dropped the fake anti-virus onto any user who accessed the site and wasn’t protected. So, when I accessed my site and noticed the problem I was infected with the fake AV. The other possibility here is that the original malware both stole the FTP information and installed the fakeAV virus. I am not sure about that one but I find it likely.
A very detailed overview of the attack can be found at the Armorize blog.
You will get a message on the site that tells you that the site will post snippets of malicious code so don’t let that frighten you. The site even took the time to create a YouTube video of the infection process which is very enlightening.
Once we figured out the rough series of events, we tried to restore the backup of the corporate site but the ISP’s local backups didn’t go back far enough and their tape backups had unknowingly failed. Bad on them and it is something they have to look at. The site was ultimately cleaned by using a program that could search all the files on the site and clean the code out manually. We seem to have everything clean at this point.
WHAT DID WE LEARN FROM ALL OF THIS?
1. Filezilla and other common FTP programs do a very, VERY poor job of protecting your passwords. I can not say this loudly enough so pay attention:
IF YOU ARE USING FILEZILLA YOU NEED TO PROTECT YOURSELF NOW!
Filezilla stores passwords several places and they are completely unencrypted and can be read by anyone or anything that can access the files. Do a Google search on this topic and you will find plenty of information. Here is a site to get you started.
If you still want to use Filezilla, I HIGHLY recommend using “kiosk mode”. I have set this up and tested it and it seems to work as advertised. It won’t store passwords so you will have to have another way to keep up with them but using Filezilla’s password store feature is just too dangerous.
Seriously, this is something you should fix immediately.
2. No matter how good you think your virus scan software is, it can’t keep up with everything.
When the outbreak was first reported by Armorize.com, only 5 of 43 tested programs picked it up. I still don’t know where exactly I picked up the exploit that originally dropped the malicious software on my PC. I consider myself a safe internet user, ESPECIALLY at work, but some site, somewhere got me. I honestly don’t know what I could have done to protect myself. It seems I just got unlucky. With that said, I have some suspicions and I will be changing my browsing habits accordingly. The best advice I can give here is to keep ALL of your software up to date and practice safe browsing. Nothing can protect you completely but you have to be vigilant.
3. Never assume anything when you notice a problem.
My first mistake was to assume the problem on my blog was isolated and caused by a template upgrade. I had no reason to believe this was true and as soon as I fixed the problem I put it out of my mind which was wrong. I should have taken more time to try and figure out what happened. By then it was already too late but it would have saved the several day delay in noticing the problem on my work site and I would have avoided the problems getting it restored.
The end result of this incident is that I have a renewed focus on security both at work and at home. In fact, I came home the night on the 22nd and did a thorough search of my daughter’s computer and found some problems I had to deal with there as well.
I realize that I have to put more effort into understanding these issues. It certainly was a learning experience and was quite enlightening but it isn’t exactly something I have time for on a daily basis. Nevertheless, it is my job to try and deal with desktop security so I have to get better at defending against these attacks. It’s no small task that’s for sure.
Oh, and if you saw the junk code on my blog that day you better check your own systems out thoroughly. Just FYI.