The Great TMN Crash of 2014

Rules and Regs for TMN
Kevin
Administrator
Administrator
Posts: 15705
Joined: Sun Jan 18, 2009 5:47 am
Status: We got a German here who wants to die for his country! Oblige him!

The Great TMN Crash of 2014

Post by Kevin »

First, allow me to apologize to each and every one of you for what happened over the weekend. I view any downtime as unacceptable and I certain view what happened this weekend as a complete breakdown in how our site should run. For those that are interested I'm starting this thread to explain and discuss what happened. Here's a breakdown of what went down, starting on Friday night.

Friday:

At 7:45PM I was notified by a member that the site was down. After initial troubleshooting I contacted our hosting provider and learned they were having database server issues and hoped to have them fixed quickly. I wasn't given an estimate for when we would be back online but was assured there was nothing we could do. Given that our primary database and any backups are managed by our host moving the site to another server elsewhere wasn't an option.

Saturday:

At 11:30AM I contacted the host for an update. They said they were still looking at it and no completion time was available.

At 2:30PM I contacted the host again. This time I was told that our site wasn't affected by the issue and our database had too many connections because the software was poorly written. When I told them that the database code hadn't changed in 6 months they had no answer.

Between 3:00 and 7:30 I contacted the host numerous times with escalating levels of frustration. I learned they did unannounced maintenance on our server and it corrupted a number of databases, of which ours was one. They had tried to repair our database and it was too corrupted to be saved. I told them to restore from a backup with they tried only to find out that the backup was corrupted as well.

At 8:30PM I requested a supervisor to discuss the situation with. At this point we were faced with not having any database backups newer than the end of last year since all of the ones from this year were on their servers and were all apparently corrupted. After a heated discussion they allowed me to have the raw database files for our site. I began repairing each file manually. The corrupted files were the table that stored posts with 306,000 records and the private messages table with 26,000 records. Over the course of last night and today I have been able to successfully recover the tables.

At 3:45 the database had been repaired, checked and installed on a server that we control. The site was brought back up and is functioning.

The site is currently going to be a little sluggish based on the fact that the install was hurried in order to get the site running.

Moving forward we have been left with no option but to move our site from a VPS environment to a dedicated server of our own. That way we can control backups, site and database settings and all aspects of the server. Until we find a new home the site can run on its current setup without any issues.

There will be abbreviated downtime when we move the site again. That downtime will be 15-20 minutes and will be one of our normal downtimes that happen late at night.

If you see anything not functioning on the website let us know. So far everything looks good. Hopefully it will stay that way.

Again, on behalf of the leadership team I apologize for the downtime. Moving forward we will take steps to make sure it doesn't happen again.
We will be cruel to Pitt.  And through our cruelty they will know who we are.  And they will find the evidence of our cruelty in the disemboweled, dismembered and disfigured bodies of their brothers we leave behind us.  And Pitt won't be able to help themselves but imagine the cruelty their brothers endured at our hands and our boot heels and the edge of our knives.  And Pitt will be sickened by us.  And Pitt will talk about us.  And Pitt will fear us.
Aaron
Hall of Fame
Hall of Fame
Posts: 28029
Joined: Sat Jan 31, 2009 11:07 pm
Location: Summersville

Re: The Great TMN Crash of 2014

Post by Aaron »

Great work. We appreciate that effort you guys put in here.
WARNING the above post does not reflect my views or the views of TMN in any way.
Lee
Hall of Fame
Hall of Fame
Posts: 32154
Joined: Sat Jan 31, 2009 8:52 am
Status: Tried to retire. Failed. Trying again.
Location: Cheat Lake, WV
Contact:

Re: The Great TMN Crash of 2014

Post by Lee »

Aaron wrote:Great work. We appreciate that effort you guys put in here.
Yeah, what Aaron said. Thanks, Kevin - that's a lot of hassle to go through for no compensation other than the satisfaction of maintaining the best Mountaineer fan site on the interwebz. I really appreciate all the work you and the rest of the leadership team do to keep this place open for us.
The Old Grad formerly known as wvu1980ae.
User avatar
eerfrombirth
Hall of Fame
Hall of Fame
Posts: 20503
Joined: Fri Jan 30, 2009 5:34 pm
Location: Danville, WV

Re: The Great TMN Crash of 2014

Post by eerfrombirth »

Yeah thanks Kev, I figured it was bad for it to be down that long.
The seeds we sow today determine the kind of fruit we'll reap tomorrow.
User avatar
sc-wvu4ever
MVP
MVP
Posts: 7858
Joined: Tue Jan 20, 2009 9:19 pm
Location: South Charleston
Contact:

Re: The Great TMN Crash of 2014

Post by sc-wvu4ever »

Kevin, thank you and all of the leadership for what you do. I really appreciate the site, my favorite place on the world wide interwebz
From Weirton to Welch and Martinsburg to Matawan the hills resound with sounds of Gold and Blue Football.
User avatar
WVUgirl2003
MVP
MVP
Posts: 7528
Joined: Tue Jan 20, 2009 9:20 pm
Status: I will go in this way and find my own way out.
Location: Hilo, Hawaii

Re: The Great TMN Crash of 2014

Post by WVUgirl2003 »

I totally thought I just got kicked out! :)
THANKS!
WVU Mountaineers: 1922 Football National Champs & 1942 Basketball National Champs. Just as legitimate- never proclaimed; until NOW!
User avatar
WVUgirl2003
MVP
MVP
Posts: 7528
Joined: Tue Jan 20, 2009 9:20 pm
Status: I will go in this way and find my own way out.
Location: Hilo, Hawaii

Re: The Great TMN Crash of 2014

Post by WVUgirl2003 »

or that the new owner of TOS that must not be named hacked this site or something!
WVU Mountaineers: 1922 Football National Champs & 1942 Basketball National Champs. Just as legitimate- never proclaimed; until NOW!
User avatar
Ryan
Administrator
Administrator
Posts: 55171
Joined: Mon Jan 19, 2009 1:53 pm
Location: Pittsburgh, PA formerly Mangtang

Re: The Great TMN Crash of 2014

Post by Ryan »

WVUgirl2003 wrote:I totally thought I just got kicked out! :)
THANKS!
I've often considered banning you for all of your offensive posts and behavior.

The sexual harassment of Aphetto was really over the line, but we let it slide.
Aaron
Hall of Fame
Hall of Fame
Posts: 28029
Joined: Sat Jan 31, 2009 11:07 pm
Location: Summersville

Re: The Great TMN Crash of 2014

Post by Aaron »

I'm ashamed but i thought i got banned at first lol
WARNING the above post does not reflect my views or the views of TMN in any way.
User avatar
Tito
Hall of Fame
Hall of Fame
Posts: 28151
Joined: Wed Jan 21, 2009 9:54 am
Status: One post over the line sweet Jesus.
Location: The Senior from Warwood with 4 seconds on his stopped clock.

Re: The Great TMN Crash of 2014

Post by Tito »

Aaron wrote:I'm ashamed but i thought i got banned at first lol
See how good a warning feels instead of an arrest? Pay it forward...Image
___________________________________________________________________________________

DISCLAIMER-WARNING:
Please be advised that the above post is for amusement only. Any emotional involvement is done so at your own risk. Thanks for your interest. Image
___________________________________________________________________________________

True, Father Time has long and cruelly sodomized me with the splintered haft of his great scythe, but I shall out-live the God-damned lot of you!
User avatar
WVUgirl2003
MVP
MVP
Posts: 7528
Joined: Tue Jan 20, 2009 9:20 pm
Status: I will go in this way and find my own way out.
Location: Hilo, Hawaii

Re: The Great TMN Crash of 2014

Post by WVUgirl2003 »

Ryan wrote:
WVUgirl2003 wrote:I totally thought I just got kicked out! :)
THANKS!
I've often considered banning you for all of your offensive posts and behavior.

The sexual harassment of Aphetto was really over the line, but we let it slide.
He still doesn't come around very much anymore. I feel so ashamed.
WVU Mountaineers: 1922 Football National Champs & 1942 Basketball National Champs. Just as legitimate- never proclaimed; until NOW!
Kevin
Administrator
Administrator
Posts: 15705
Joined: Sun Jan 18, 2009 5:47 am
Status: We got a German here who wants to die for his country! Oblige him!

Re: The Great TMN Crash of 2014

Post by Kevin »

I appreciate the kind words. It truly is a pleasure keeping the site running for everyone. Given this week is an off week I'm planning on moving the site to a dedicated server sometime over the weekend. I'll keep everyone updated as to when that goes down. We'll also be making provisions for a hot backup server so if one goes down the site will automatically flip to the other. That will be down the road.
We will be cruel to Pitt.  And through our cruelty they will know who we are.  And they will find the evidence of our cruelty in the disemboweled, dismembered and disfigured bodies of their brothers we leave behind us.  And Pitt won't be able to help themselves but imagine the cruelty their brothers endured at our hands and our boot heels and the edge of our knives.  And Pitt will be sickened by us.  And Pitt will talk about us.  And Pitt will fear us.
jdakanwv
Junior Varsity
Junior Varsity
Posts: 1037
Joined: Tue Jan 12, 2010 9:16 am
Status: Apathy is a bitch and the death of men.
Location: Morgantown, WV (Originally a Wetzel County Boy)

Re: The Great TMN Crash of 2014

Post by jdakanwv »

Sometimes these things happen with this new fangled technology thing and the inter webs.

Thanks for the time you put in to bring the site back up for us.
User avatar
Ryan
Administrator
Administrator
Posts: 55171
Joined: Mon Jan 19, 2009 1:53 pm
Location: Pittsburgh, PA formerly Mangtang

Re: The Great TMN Crash of 2014

Post by Ryan »

WVUgirl2003 wrote:
Ryan wrote:
WVUgirl2003 wrote:I totally thought I just got kicked out! :)
THANKS!
I've often considered banning you for all of your offensive posts and behavior.

The sexual harassment of Aphetto was really over the line, but we let it slide.
He still doesn't come around very much anymore. I feel so ashamed.
He's a pansy. Most of us would have taken a little sexual harassment in stride.
jdakanwv
Junior Varsity
Junior Varsity
Posts: 1037
Joined: Tue Jan 12, 2010 9:16 am
Status: Apathy is a bitch and the death of men.
Location: Morgantown, WV (Originally a Wetzel County Boy)

Re: The Great TMN Crash of 2014

Post by jdakanwv »

If i got sexually harassed it would be the most action ive had in months.
Post Reply