The Great TMN Crash of 2014

[Kevin]

First, allow me to apologize to each and every one of you for what happened over the weekend. I view any downtime as unacceptable and I certain view what happened this weekend as a complete breakdown in how our site should run. For those that are interested I'm starting this thread to explain and discuss what happened. Here's a breakdown of what went down, starting on Friday night.

Friday:

At 7:45PM I was notified by a member that the site was down. After initial troubleshooting I contacted our hosting provider and learned they were having database server issues and hoped to have them fixed quickly. I wasn't given an estimate for when we would be back online but was assured there was nothing we could do. Given that our primary database and any backups are managed by our host moving the site to another server elsewhere wasn't an option.

Saturday:

At 11:30AM I contacted the host for an update. They said they were still looking at it and no completion time was available.

At 2:30PM I contacted the host again. This time I was told that our site wasn't affected by the issue and our database had too many connections because the software was poorly written. When I told them that the database code hadn't changed in 6 months they had no answer.

Between 3:00 and 7:30 I contacted the host numerous times with escalating levels of frustration. I learned they did unannounced maintenance on our server and it corrupted a number of databases, of which ours was one. They had tried to repair our database and it was too corrupted to be saved. I told them to restore from a backup with they tried only to find out that the backup was corrupted as well.

At 8:30PM I requested a supervisor to discuss the situation with. At this point we were faced with not having any database backups newer than the end of last year since all of the ones from this year were on their servers and were all apparently corrupted. After a heated discussion they allowed me to have the raw database files for our site. I began repairing each file manually. The corrupted files were the table that stored posts with 306,000 records and the private messages table with 26,000 records. Over the course of last night and today I have been able to successfully recover the tables.

At 3:45 the database had been repaired, checked and installed on a server that we control. The site was brought back up and is functioning.

The site is currently going to be a little sluggish based on the fact that the install was hurried in order to get the site running.

Moving forward we have been left with no option but to move our site from a VPS environment to a dedicated server of our own. That way we can control backups, site and database settings and all aspects of the server. Until we find a new home the site can run on its current setup without any issues.

There will be abbreviated downtime when we move the site again. That downtime will be 15-20 minutes and will be one of our normal downtimes that happen late at night.

If you see anything not functioning on the website let us know. So far everything looks good. Hopefully it will stay that way.

Again, on behalf of the leadership team I apologize for the downtime. Moving forward we will take steps to make sure it doesn't happen again.

[Aaron]

Great work. We appreciate that effort you guys put in here.

[Lee]

Great work. We appreciate that effort you guys put in here.

Yeah, what Aaron said. Thanks, Kevin - that's a lot of hassle to go through for no compensation other than the satisfaction of maintaining the best Mountaineer fan site on the interwebz. I really appreciate all the work you and the rest of the leadership team do to keep this place open for us.

[eerfrombirth]

Yeah thanks Kev, I figured it was bad for it to be down that long.

[sc-wvu4ever]

Kevin, thank you and all of the leadership for what you do. I really appreciate the site, my favorite place on the world wide interwebz

[WVUgirl2003]

I totally thought I just got kicked out!
THANKS!

[WVUgirl2003]

or that the new owner of TOS that must not be named hacked this site or something!

[Ryan]

I totally thought I just got kicked out!
THANKS!

I've often considered banning you for all of your offensive posts and behavior.

The sexual harassment of Aphetto was really over the line, but we let it slide.

[Aaron]

I'm ashamed but i thought i got banned at first lol

[Tito]

I'm ashamed but i thought i got banned at first lol

See how good a warning feels instead of an arrest? Pay it forward...

[WVUgirl2003]

I totally thought I just got kicked out!
THANKS!

I've often considered banning you for all of your offensive posts and behavior.

The sexual harassment of Aphetto was really over the line, but we let it slide.

He still doesn't come around very much anymore. I feel so ashamed.

[Kevin]

I appreciate the kind words. It truly is a pleasure keeping the site running for everyone. Given this week is an off week I'm planning on moving the site to a dedicated server sometime over the weekend. I'll keep everyone updated as to when that goes down. We'll also be making provisions for a hot backup server so if one goes down the site will automatically flip to the other. That will be down the road.

[jdakanwv]

Sometimes these things happen with this new fangled technology thing and the inter webs.

Thanks for the time you put in to bring the site back up for us.

[Ryan]

I totally thought I just got kicked out!
THANKS!

I've often considered banning you for all of your offensive posts and behavior.

The sexual harassment of Aphetto was really over the line, but we let it slide.

He still doesn't come around very much anymore. I feel so ashamed.

He's a pansy. Most of us would have taken a little sexual harassment in stride.

[jdakanwv]

If i got sexually harassed it would be the most action ive had in months.