Site was down, why?

The site was down for 30 minutes earlier tonight. There was a database error, it's been fixed. There might be a little more trouble while we dig around and see if there was something specific that caused the problem.

Update Tuesday, February 2nd, 1pm EST:
The site was knocked out last night and this morning by a large scale attack that attempted to flood our users with spam using the Private Message system. We have systems in place to protect our users from this kind of abuse (such as the 50 post requirement to PM), so the PMs were denied and we do not believe any were successfully sent. However, the sheer volume of queries to the database caused the server to seize-up, shutting down the site. The attack also corrupted the database, and it took some time to restore a backup that wasn't corrupted.

Until we can implement a fix to prevent this from reoccurring, the PM system is currently disabled. I'm sorry for the inconvenience, but at the moment we have to choose between a site that is accessible but with the PM system disabled, or a site that is completely down.

Update Wednesday, February 3rd, 12:40am EST:
Database was fubar'd again for about 3 hours. I'm trying to get an explanation from the server management company. I will not be surprised if we experience more downtime before the issue is completely fixed.

Update Wednesday, February 3rd, Noon EST:
Things have been back to normal and stable for the past 11 hours. I've reactivated the private message system and will monitor it for problems. Thank you for your patience.

 

What originally appeared to be just a simple database error was actually a large-scale spam attack, which ate up system resources. The constant barrage eventually corrupted the database, putting the site down hard. We finally got a restore from backup just a little while ago. We have a dynamic database backup system, so we were able to restore from around 10:30pm EST last night, which means there should only be a minimal number of posts lost.

The Private Messaging system is temporarily disabled while we put some additional defenses in there.

 

We're still getting hammered, doing our best to keep things going. I just want to reassure everyone that the site isn't being hacked, no data is stolen, they're just sending a huge amount of queries at our server which is causing it to seize up.

 

We might need to restore from a 7:30pm database from last night. If we do, it'll happen in the next half hour, and this thread will be lost. I'll try to keep everyone updated.

 

Unfortunately, it does look like we'll need to do that earlier restore point. Sorry, we're going to lose all posts that were made after 7:45pm EST last night, including this thread in the next few minutes.