The Great TMN Crash of 2014

Rules and Regs for TMN

The Great TMN Crash of 2014

Postby Kevin » Sun Sep 21, 2014 4:28 pm

First, allow me to apologize to each and every one of you for what happened over the weekend. I view any downtime as unacceptable and I certain view what happened this weekend as a complete breakdown in how our site should run. For those that are interested I'm starting this thread to explain and discuss what happened. Here's a breakdown of what went down, starting on Friday night.

Friday:

At 7:45PM I was notified by a member that the site was down. After initial troubleshooting I contacted our hosting provider and learned they were having database server issues and hoped to have them fixed quickly. I wasn't given an estimate for when we would be back online but was assured there was nothing we could do. Given that our primary database and any backups are managed by our host moving the site to another server elsewhere wasn't an option.

Saturday:

At 11:30AM I contacted the host for an update. They said they were still looking at it and no completion time was available.

At 2:30PM I contacted the host again. This time I was told that our site wasn't affected by the issue and our database had too many connections because the software was poorly written. When I told them that the database code hadn't changed in 6 months they had no answer.

Between 3:00 and 7:30 I contacted the host numerous times with escalating levels of frustration. I learned they did unannounced maintenance on our server and it corrupted a number of databases, of which ours was one. They had tried to repair our database and it was too corrupted to be saved. I told them to restore from a backup with they tried only to find out that the backup was corrupted as well.

At 8:30PM I requested a supervisor to discuss the situation with. At this point we were faced with not having any database backups newer than the end of last year since all of the ones from this year were on their servers and were all apparently corrupted. After a heated discussion they allowed me to have the raw database files for our site. I began repairing each file manually. The corrupted files were the table that stored posts with 306,000 records and the private messages table with 26,000 records. Over the course of last night and today I have been able to successfully recover the tables.

At 3:45 the database had been repaired, checked and installed on a server that we control. The site was brought back up and is functioning.

The site is currently going to be a little sluggish based on the fact that the install was hurried in order to get the site running.

Moving forward we have been left with no option but to move our site from a VPS environment to a dedicated server of our own. That way we can control backups, site and database settings and all aspects of the server. Until we find a new home the site can run on its current setup without any issues.

There will be abbreviated downtime when we move the site again. That downtime will be 15-20 minutes and will be one of our normal downtimes that happen late at night.

If you see anything not functioning on the website let us know. So far everything looks good. Hopefully it will stay that way.

Again, on behalf of the leadership team I apologize for the downtime. Moving forward we will take steps to make sure it doesn't happen again.
We will be cruel to Pitt.  And through our cruelty they will know who we are.  And they will find the evidence of our cruelty in the disemboweled, dismembered and disfigured bodies of their brothers we leave behind us.  And Pitt won't be able to help themselves but imagine the cruelty their brothers endured at our hands and our boot heels and the edge of our knives.  And Pitt will be sickened by us.  And Pitt will talk about us.  And Pitt will fear us.
User avatar
Kevin
Administrator
Administrator
 
Posts: 16071
Joined: Sun Jan 18, 2009 6:47 am
Status: We got a German here who wants to die for his country! Oblige him!

Re: The Great TMN Crash of 2014

Postby Aaron » Sun Sep 21, 2014 4:48 pm

Great work. We appreciate that effort you guys put in here.
WARNING the above post does not reflect my views or the views of TMN in any way.
User avatar
Aaron
Hall of Fame
Hall of Fame
 
Posts: 28551
Joined: Sun Feb 01, 2009 12:07 am
Location: Summersville

Re: The Great TMN Crash of 2014

Postby Lee » Sun Sep 21, 2014 5:07 pm

Aaron wrote:Great work. We appreciate that effort you guys put in here.

Yeah, what Aaron said. Thanks, Kevin - that's a lot of hassle to go through for no compensation other than the satisfaction of maintaining the best Mountaineer fan site on the interwebz. I really appreciate all the work you and the rest of the leadership team do to keep this place open for us.
The Old Grad formerly known as wvu1980ae.
Lee
Hall of Fame
Hall of Fame
 
Posts: 26943
Joined: Sat Jan 31, 2009 9:52 am
Location: Cheat Neck, WV
Status: De-stressed

Re: The Great TMN Crash of 2014

Postby eerfrombirth » Sun Sep 21, 2014 6:09 pm

Yeah thanks Kev, I figured it was bad for it to be down that long.
The seeds we sow today determine the kind of fruit we'll reap tomorrow.
User avatar
eerfrombirth
Hall of Fame
Hall of Fame
 
Posts: 18228
Joined: Fri Jan 30, 2009 6:34 pm
Location: Danville, WV

Re: The Great TMN Crash of 2014

Postby sc-wvu4ever » Sun Sep 21, 2014 6:17 pm

Kevin, thank you and all of the leadership for what you do. I really appreciate the site, my favorite place on the world wide interwebz
From Weirton to Welch and Martinsburg to Matawan the hills resound with sounds of Gold and Blue Football.
User avatar
sc-wvu4ever
MVP
MVP
 
Posts: 6861
Joined: Tue Jan 20, 2009 10:19 pm
Location: South Charleston

Re: The Great TMN Crash of 2014

Postby WVUgirl2003 » Sun Sep 21, 2014 9:15 pm

I totally thought I just got kicked out! :)
THANKS!
WVU Mountaineers: 1922 Football National Champs & 1942 Basketball National Champs. Just as legitimate- never proclaimed; until NOW!
User avatar
WVUgirl2003
MVP
MVP
 
Posts: 7426
Joined: Tue Jan 20, 2009 10:20 pm
Location: Hilo, Hawaii
Status: I will go in this way and find my own way out.

Re: The Great TMN Crash of 2014

Postby WVUgirl2003 » Sun Sep 21, 2014 9:16 pm

or that the new owner of TOS that must not be named hacked this site or something!
WVU Mountaineers: 1922 Football National Champs & 1942 Basketball National Champs. Just as legitimate- never proclaimed; until NOW!
User avatar
WVUgirl2003
MVP
MVP
 
Posts: 7426
Joined: Tue Jan 20, 2009 10:20 pm
Location: Hilo, Hawaii
Status: I will go in this way and find my own way out.

Re: The Great TMN Crash of 2014

Postby Ryan » Mon Sep 22, 2014 10:14 am

WVUgirl2003 wrote:I totally thought I just got kicked out! :)
THANKS!


I've often considered banning you for all of your offensive posts and behavior.

The sexual harassment of Aphetto was really over the line, but we let it slide.
User avatar
Ryan
Administrator
Administrator
 
Posts: 45538
Joined: Mon Jan 19, 2009 2:53 pm
Location: Pittsburgh, PA formerly Mangtang

Re: The Great TMN Crash of 2014

Postby Aaron » Mon Sep 22, 2014 11:37 am

I'm ashamed but i thought i got banned at first lol
WARNING the above post does not reflect my views or the views of TMN in any way.
User avatar
Aaron
Hall of Fame
Hall of Fame
 
Posts: 28551
Joined: Sun Feb 01, 2009 12:07 am
Location: Summersville

Re: The Great TMN Crash of 2014

Postby Tito » Mon Sep 22, 2014 3:59 pm

Aaron wrote:I'm ashamed but i thought i got banned at first lol


See how good a warning feels instead of an arrest? Pay it forward...Image
________________________________________________________________________________________________________________________________
DISCLAIMER-WARNING: Please be advised that the above post is for amusement only. Any emotional involvement is done so at your own risk. Thanks for your interest.
________________________________________________________________________________________________________________________________

True, Father Time has long and cruelly sodomized me with the splintered haft of his great scythe, but I shall out-live the God-damned lot of you!
User avatar
Tito
Hall of Fame
Hall of Fame
 
Posts: 25927
Joined: Wed Jan 21, 2009 10:54 am
Status: One post over the line sweet Jesus.

Re: The Great TMN Crash of 2014

Postby WVUgirl2003 » Mon Sep 22, 2014 7:16 pm

Ryan wrote:
WVUgirl2003 wrote:I totally thought I just got kicked out! :)
THANKS!


I've often considered banning you for all of your offensive posts and behavior.

The sexual harassment of Aphetto was really over the line, but we let it slide.


He still doesn't come around very much anymore. I feel so ashamed.
WVU Mountaineers: 1922 Football National Champs & 1942 Basketball National Champs. Just as legitimate- never proclaimed; until NOW!
User avatar
WVUgirl2003
MVP
MVP
 
Posts: 7426
Joined: Tue Jan 20, 2009 10:20 pm
Location: Hilo, Hawaii
Status: I will go in this way and find my own way out.

Re: The Great TMN Crash of 2014

Postby Kevin » Mon Sep 22, 2014 9:20 pm

I appreciate the kind words. It truly is a pleasure keeping the site running for everyone. Given this week is an off week I'm planning on moving the site to a dedicated server sometime over the weekend. I'll keep everyone updated as to when that goes down. We'll also be making provisions for a hot backup server so if one goes down the site will automatically flip to the other. That will be down the road.
We will be cruel to Pitt.  And through our cruelty they will know who we are.  And they will find the evidence of our cruelty in the disemboweled, dismembered and disfigured bodies of their brothers we leave behind us.  And Pitt won't be able to help themselves but imagine the cruelty their brothers endured at our hands and our boot heels and the edge of our knives.  And Pitt will be sickened by us.  And Pitt will talk about us.  And Pitt will fear us.
User avatar
Kevin
Administrator
Administrator
 
Posts: 16071
Joined: Sun Jan 18, 2009 6:47 am
Status: We got a German here who wants to die for his country! Oblige him!

Re: The Great TMN Crash of 2014

Postby jdakanwv » Mon Sep 22, 2014 11:00 pm

Sometimes these things happen with this new fangled technology thing and the inter webs.

Thanks for the time you put in to bring the site back up for us.
jdakanwv
Junior Varsity
Junior Varsity
 
Posts: 1056
Joined: Tue Jan 12, 2010 10:16 am
Location: Morgantown, WV (Originally a Wetzel County Boy)
Status: Apathy is a bitch and the death of men.

Re: The Great TMN Crash of 2014

Postby Ryan » Wed Sep 24, 2014 8:24 am

WVUgirl2003 wrote:
Ryan wrote:
WVUgirl2003 wrote:I totally thought I just got kicked out! :)
THANKS!


I've often considered banning you for all of your offensive posts and behavior.

The sexual harassment of Aphetto was really over the line, but we let it slide.


He still doesn't come around very much anymore. I feel so ashamed.


He's a pansy. Most of us would have taken a little sexual harassment in stride.
User avatar
Ryan
Administrator
Administrator
 
Posts: 45538
Joined: Mon Jan 19, 2009 2:53 pm
Location: Pittsburgh, PA formerly Mangtang

Re: The Great TMN Crash of 2014

Postby jdakanwv » Wed Sep 24, 2014 10:22 am

If i got sexually harassed it would be the most action ive had in months.
jdakanwv
Junior Varsity
Junior Varsity
 
Posts: 1056
Joined: Tue Jan 12, 2010 10:16 am
Location: Morgantown, WV (Originally a Wetzel County Boy)
Status: Apathy is a bitch and the death of men.

Next

Return to themountaineernation.com Rules and Information



Who is online

Users browsing this forum: No registered users