Issues with your account? Bug us in the Discord!
Firstones.com - Status Update
Random Chaos
Actually Carefully-selected Order in disguise
in Zocalo v2.0
This is our first week where we have been completely hack-free in the past two months that I have been triaging the site. Unfortunately this has resulted in us essentially not having a site: all that is left are the forums.
The result? I am going to slowly go through the other parts of the site and try to bring them back up. This includes:
1. Our Wiki - even upgrading the wiki will not solve all the issues with it - all uploaded images have to be cleaned up - they are heavily infected with exploit scripts.
2. The Sierra Babylon 5 site archive - needs cleaning, but it is in far better shape than the wiki.
3. Our hosted mod and archive sites - these need both cleaning and code auditing before they can be put back up. Even one hole or malicious script could reinfect the entire site. Some sites may not return.
Some things we believe about these attacks:
1. These were targeted at us. The attacker was too persistent every time we believed we had cleaned the site, sometimes going to pretty extreme measures to get back in. After talking with Sanfam, we believe the original attack was probably the IFH forums last summer, over six months ago. This then propagated to our other sites, with the main follow-up attack vectors coming through our Wiki and forums; most of these subsequent attacks began to be noticed around three months ago. My guess is it was targeted against IFH rather than Firstones as a whole. We were just collateral damage.
2. The person was skilled. After putting up vBulletin 5, still a Beta product and a complete rewrite, the attacker spent the time and effort to analyze the vB code sufficiently to utilize its internal redirect scripts to help mask the attack vectors.
3. The person was using pretty powerful code obfuscation techniques that have only recently been seen. Samples of this obfuscation were forwarded on to SANS Internet Storm Center for analysis, and they had not seen the method used before. During the course of the attacks, the obfuscation evolved into a more structured method, but did not change in technique, indicating that the attacker perfected easy generation of the obfuscation between when we first saw it around two months ago and when the attacks peeked on our site occurred around 2-3 weeks ago.
I hope to start going through the down parts of the site within the next week, though auditing every component of the site will probably take several months to complete. In consultation with Sanfam, some hosted sites may not return, though I do expect most will.
The result? I am going to slowly go through the other parts of the site and try to bring them back up. This includes:
1. Our Wiki - even upgrading the wiki will not solve all the issues with it - all uploaded images have to be cleaned up - they are heavily infected with exploit scripts.
2. The Sierra Babylon 5 site archive - needs cleaning, but it is in far better shape than the wiki.
3. Our hosted mod and archive sites - these need both cleaning and code auditing before they can be put back up. Even one hole or malicious script could reinfect the entire site. Some sites may not return.
Some things we believe about these attacks:
1. These were targeted at us. The attacker was too persistent every time we believed we had cleaned the site, sometimes going to pretty extreme measures to get back in. After talking with Sanfam, we believe the original attack was probably the IFH forums last summer, over six months ago. This then propagated to our other sites, with the main follow-up attack vectors coming through our Wiki and forums; most of these subsequent attacks began to be noticed around three months ago. My guess is it was targeted against IFH rather than Firstones as a whole. We were just collateral damage.
2. The person was skilled. After putting up vBulletin 5, still a Beta product and a complete rewrite, the attacker spent the time and effort to analyze the vB code sufficiently to utilize its internal redirect scripts to help mask the attack vectors.
3. The person was using pretty powerful code obfuscation techniques that have only recently been seen. Samples of this obfuscation were forwarded on to SANS Internet Storm Center for analysis, and they had not seen the method used before. During the course of the attacks, the obfuscation evolved into a more structured method, but did not change in technique, indicating that the attacker perfected easy generation of the obfuscation between when we first saw it around two months ago and when the attacks peeked on our site occurred around 2-3 weeks ago.
I hope to start going through the down parts of the site within the next week, though auditing every component of the site will probably take several months to complete. In consultation with Sanfam, some hosted sites may not return, though I do expect most will.
Comments
If you don't mind RC I'm going to quote your report above in private message with Bob over at F3D and see what kind of coincidences or stuff might drop out of the discussion.
F3D got really hammered from both China and Russia IP's in recent times.
Certainly forward it to F3D. I don't know the source of our attacks. All the IPs I logged were either open proxies or infected intermediary systems. If you want to drop by IRC (freenode.net channel #firstones), I can show you some of the attack scripts and how they work. I don't want to post them on here.
Psi-killer:
It was very bad. For about 2 weeks right around Christmas, I was spending at least one hour every day cleaning the site, with some days reaching 4-6 hours. I think the peak day was over 12 hours, though some of that was reversing his obfuscation scripts. I've got it down to a science now, though - I can clean the forums in about 15 minutes. It would be faster, but the damn tarball I upload has several thousand files in it that take a while to inflate :) - though that is one of the reasons I haven't restored other parts of the site yet - I do not have a clean tarball I can just upload of them.
As for directed against breaking vB code, no. The IFH forums hit was not against vB - IFH was using I believe PHPBB, which is very vulnerable to attacks, even though we told them a couple years ago to get rid of that for more secure forum software. They apparently didn't listen and kept using PHPBB, and Sanfam never followed up to make sure they had changed.
To try and answer your question about where the attack originated, the bots were all coming from eastern and central Europe. That is geographically significant, since it indicates that the internet infection method contains either linguistic or cultural reasons not to extend beyond that region. However, the IPs used appear to be part of a much, much larger botnet which was not targeted toward us. Possibly someone buying time on another person's botnet, otherwise I would think other IPs would have been utilized.
Also of interest is that it does not appear the attacker mined our email address for spam email, which would be very unusual for a botnet operator. I say this because I use a different email address on Firstones than anywhere else, and I have gotten no spam sent to that address. It is unusual since one of the core ways of expanding a botnet is to send targetted unsolicited emails that look legitimate from either a website or another user on that website. A botnet operator is not going to miss that opportunity, lending credence to the idea that the attacker may have been renting usage of several infected computers on another person's botnet.
???
[ATTACH=CONFIG]n683[/ATTACH]
[ATTACH=CONFIG]n684[/ATTACH]
Sorry, just got here. I hate to be responsible for the hack, if that's how this happened. Believe me, I feel bad... :(
However, let me clarify something.
After the "old hack", back in 2006, when Firstones banned free forum engines we switched to commercial IPB engine and have been using it ever since! We also installed all security updates as soon as they were out.
So, while I may agree to take the blame for great many things, I can't take it for using old, non-secure engine.
Do you have any more info on how it was infiltrated?
Sorry, just got here. I hate to be responsible for the hack, if that's how this happened. Believe me, I feel bad... :(
However, let me clarify something.
After the "old hack", back in 2006, when Firstones banned free forum engines we switched to commercial IPB engine and have been using it ever since! We also installed all security updates as soon as they were out.
So, while I may agree to take the blame for great many things, I can't take it for using old, non-secure engine.
Do you have any more info on how it was infiltrated?
Sorry, just got here. I hate to be responsible for the hack, if that's how this happened. Believe me, I feel bad... :(
However, let me clarify something.
After the "old hack", back in 2006, when Firstones banned free forum engines we switched to commercial IPB engine and have been using it ever since! We also installed all security updates as soon as they were out.
So, while I may agree to take the blame for great many things, I can't take it for using old, non-secure engine.
Do you have any more info on how it was infiltrated?
We do not have any more information. It appears that the ifh.firstones.com still had an old forum engine on it, even though you moved over to a new URL and forum, with the ifh.firstones.com address as a redirect. There was an uploaded file in that forum tree that was used as a re-attack point and possibly an original attack point. We know that location and our Wiki were two of the locations where the attacker was able to upload files to our website, which he then executed. We do not know the order, or enough of the history of when attacks occurred. Sanfam indicated to me that there was an attack last summer (I recall that he said it was via the IFH site), but I was not involved in cleaning up that one - my guess is it was never fully cleaned up and this attack is residual. Unfortunately we do not have access to any historical information from the server that would allow us better analysis of what occurred, and when each part occurred.
What bothers me most is somehow the attacker hit us with a privilege escalation attack - he hit using the webserver user, but was able to modify files that were owned by other accounts. My biggest concern is that he did stuff to the server we use as a host that will allow him back in, and Dreamhost is unfortunately not very responsive to reports of site attacks (last time I reported something, I got stuck in an automated queue that did nothing useful...).
I do hope to start cleaning up sites soon and bringing mods and other parts of our website back online. My time is unfortunately stretched thin these days so triage was the best I could accomplish in such a short time over the past few weeks.
As for your applying security updates - unfortunately updates are usually weeks or even months behind vulnerabilities :( - makes it really hard to keep ahead of hackers no matter how diligent you are :(
I get Error saving content messages simply replying to a thread, no attachments. Takes forever too.
*sigh*
EDIT: When editing a message it does not populate the window with what should already be in here either. I had to copy paste.
May be time for a fresh start?