Server trouble?

kimmy · January 21, 2009

The forum appears to be having a really hard time the past couple of days.

Slow response, pages often load only partially and occasionally not at all.

And last night I was trying to post a message and found the forum went completely to sleep when I hit the "submit" button and became not-responsive for several minutes afterward; this happened a few times. (A database issue perhaps?)

Did Greg forget to feed the hamsters again?

-k

guyser · January 21, 2009

Well, now I know I am not the only one. The pages load slow, sometimes the font changes, other times it stops loading halfway through.

I suspected maybe the Obama shtuff was clogging the net the same as cells were down in washington, but today is the same , so it aint that.

DogOnPorch · January 21, 2009

ditto...logs me off constantly, as well.

capricorn · January 21, 2009

I've been having problems also and thought it was a problem with my internet provider. Yesterday I received messages about a driver error when I tried to use the back button but I can't remember the exact words of the error message.

Mr.Canada · January 22, 2009

Well, now I know I am not the only one. The pages load slow, sometimes the font changes, other times it stops loading halfway through.
I suspected maybe the Obama shtuff was clogging the net the same as cells were down in washington, but today is the same , so it aint that.

Same here, it's only on this site though. Everything else loads fast for me.

Greg · January 22, 2009

The forum appears to be having a really hard time the past couple of days.
Did Greg forget to feed the hamsters again?

Thankfully the hamster's are still spinning the wheels!

However, we are experiencing some issues relating to CPU load on the server. Over the past few months, we've seen traffic spike drastically, and this is putting increasing strain on our database server and server CPU. However, a new server with upgraded hardware is going to be provisioned tomorrow, and we should be back up and running smoothly by early next week.

In the mean time, you may experience some temporary delays regarding page loading.

Thank you for your understanding.

AngusThermopyle · January 22, 2009

However, a new server with upgraded hardware is going to be provisioned tomorrow,

Ooooh nice!

What you gettin, come on, tell us. CPU speed, type and configuration, drive configurations, RAM capacity and speed. Who is providing it, come on, give us details. This is exciting.

Mr.Canada · January 22, 2009

Ooooh nice!
What you gettin, come on, tell us. CPU speed, type and configuration, drive configurations, RAM capacity and speed. Who is providing it, come on, give us details. This is exciting.

Seconded. I'd like some specs to the new rig as well. This is like porn to some of us.

Greg · January 22, 2009

Seconded. I'd like some specs to the new rig as well. This is like porn to some of us.

Sure,

There server will have the following:

CentOS

3.0 GHz Quad-Core "Clovertown" 64bit Xeon Processor

4 GB of DDR2 ECC 667MHz RAM

2 x 146 GB 10K RPM SCSI Hard drives

Raid 1

From what I understand, we'll be able to upgrade to 8 GB of memory if needed and should we ever need more power, we could slap another Quad-Core on the motherboard.

This should be plenty of power for the next 12 months.

AngusThermopyle · January 22, 2009

Very cool, I figured you'd be using a Xeon/Clovertown Quad but wasn't sure if you'd use a multi CPU board, always wanted to build one myself but those boards are horendously expensive. I am a bit surprised by the RAM and Drives though, RAM is cheap right now unless its ECC, but of course you'll be running ECC for the Xeon so maybe its not so cheap. The RAID 1 configuration of the drives is expected but the overall size is surprising, they're not that large, but then again they're only used for text so maybe not. You don't happen to know anything about the SCSI controller do you? Is it an onboard solution or a seperate card?

Thanks for indulging us junkies.

M.Dancer · January 22, 2009

I have a Wii.......it's plugged into my Cathode TeeVee....

Greg · January 22, 2009

Very cool, I figured you'd be using a Xeon/Clovertown Quad but wasn't sure if you'd use a multi CPU board, always wanted to build one myself but those boards are horendously expensive.

Right now we're only running a single 3.0 Ghz XEON chip, and we're seeing loads that are too high for my liking. We've attempted to tweak Apache and MySQL to reduce the load, but then we started to see performance issues.

We could technically drop another XEON chip into the existing motherboard, but the cost of that would be nearly the same as the Quad. So naturally, we're going with the Quad.

I am a bit surprised by the RAM and Drives though, RAM is cheap right now unless its ECC, but of course you'll be running ECC for the Xeon so maybe its not so cheap.

It's true you can always use more RAM, but in this case, 4 GB of RAM should be sufficient. The data center we work with doesn't provide anything other than ECC RAM with this particular motherboard - it's all enterprise level hardware.

Right now we're only running two 80 GB SATA drives, so the new 10K HotSwap SAS/SCSI drives will really help with read times. Also, we're not running Raid 1 right now, but rather just using the second drive as a backup drive (no mirroring).

The RAID 1 configuration of the drives is expected but the overall size is surprising, they're not that large, but then again they're only used for text so maybe not.

Ya, other than the cartoons, our overall disk usage is quite low. The databases are large, but no longer than 2 GBs.

You don't happen to know anything about the SCSI controller do you? Is it an onboard solution or a seperate card?

Not sure, but I can find out.

capricorn · January 23, 2009

I have a Wii.......it's plugged into my Cathode TeeVee....

I have a wireless router with a soft blinking blue LED light. State of the heart gadget. :lol:

Melanie_ · January 23, 2009

I have a wireless router with a soft blinking blue LED light.

Don't tell kimmy! She'll come over with her wire cutters...

M.Dancer · January 23, 2009

I have a wireless router with a soft blinking blue LED light. State of the heart gadget.

I won the Highlands Open last night playing Tiger Woods 09...my Wii remote has a steady blue led..

capricorn · January 23, 2009

The hardware improvements seem to have kicked in. I find the pages of the forum are loading much more quickly.

GostHacked · February 4, 2009

Greg, ever thought of those new solid state hard drives?? For faster access to those items that are accessed most often?? Also I am very surprised that you are a single core server.

We have a few IBM NetVistas in our server room, and our main production and back up servers runs 4 dual core Xeons each. Then again, this beast runs our entire warehouse. Then there is the two fast racks fiber connecting to the drives aprox 40 140GB 15K drives.

Not to put your E-penis down or anythying Greg .. I'd be proud of it tooo.. you own it .. I am just the babysitter for the servers here lol

AngusThermopyle · February 5, 2009

Greg, ever thought of those new solid state hard drives?? For faster access to those items that are accessed most often?? Also I am very surprised that you are a single core server.
We have a few IBM NetVistas in our server room, and our main production and back up servers runs 4 dual core Xeons each. Then again, this beast runs our entire warehouse. Then there is the two fast racks fiber connecting to the drives aprox 40 140GB 15K drives.

Thats all very nice but I think cost might be a major consideration for MLW. Its also very possible that traffic and load is no where even close to what you have to contend with so such a setup would be a little overkill to say the least.

Greg · February 5, 2009

Greg, ever thought of those new solid state hard drives?? For faster access to those items that are accessed most often??

Nope, I've never heard for using solid state drives in a enterprise server environment - however, that doesn't mean it isn't done! In our particular situation, we lease out server from a large datacentre in BC, which means we're limited by the hardware they provide for the servers. However, we do have the option of purchasing our own server and then colocating it in the same datacentre - which would mean we could build and use any hardware - but that significantly increases the cost of operating a server.

Also I am very surprised that you are a single core server.

Right now we're operating on a single core, but we've moving to a quad core system.

We have a few IBM NetVistas in our server room, and our main production and back up servers runs 4 dual core Xeons each. Then again, this beast runs our entire warehouse. Then there is the two fast racks fiber connecting to the drives aprox 40 140GB 15K drives.

Ha, that's certainly more than we'd need!

Not to put your E-penis down or anythying Greg .. I'd be proud of it tooo.. you own it .. I am just the babysitter for the servers here lol

As I said, we don't actually own the hardware, we just lease it. Owning and collocating is just too pricey at this point.

We considered buying our own server and putting it in the datacentre here at the University of Lethbridge, which was an option when we first started in 2000. However, after much consideration, I decided that the University IT department has better things to worry about than a box in the corner; particularly a box that isn't part of the universities core network.

If, after moving the site to a new server, we continue to see increased loads, we'll probably have to move to a two server configuration, with one server handling the http side of things, while the other handling the database. Our MySQL database is huge - 400,000 posts plus from the forums + everything else on MLW (drupal, our CMS can be hard on a database, particularly with alot of modules loaded). So I maybe going through all of this again in a few months.

Greg · February 5, 2009

Thats all very nice but I think cost might be a major consideration for MLW. Its also very possible that traffic and load is no where even close to what you have to contend with so such a setup would be a little overkill to say the least.

Right now we transmit around 60 GB a month of traffic - not much. We're primary a text based website.

However, CPU load is a funny thing. You can have a site that receives 100k + unique visitors a day and it will have a negligible impact on CPU load, while another can only have a few thousand a day and it can bring a server to its knees. It really depends on how the site is build. If the server is serving static pages - which apache (the webserver) prefers - then a normal server can usually withstand an impressive amount of traffic. However, if the site is build with PHP/ASP/etc via a MySQL/MSQL database, then things can get a bit more tricky. Things will get even worse if the site is designed and coded poorly, which means the database and apache have to work even harder to serve each page.

Drupal, the content management system that drives MLW, is an enterprise CMS and it can do amazing things. However, it can be a resource hog if not handled properly. Shortly after we migrate to the new server, I'm going to be upgrading Drupal, and removing some of those modules that could be causing load problems.

However, considering that we are attracting between 60,000 to 80,000 unique visitors a month to MLW (and seeing 10-20% growth each month), I'm not overly worried about dealing with these issues!

GostHacked · February 11, 2009

Right now we transmit around 60 GB a month of traffic - not much. We're primary a text based website.

60 is a lot for just text, imagine avatars and junk like that

Shakeyhands · February 11, 2009

Greg, ever thought of those new solid state hard drives?? For faster access to those items that are accessed most often?? Also I am very surprised that you are a single core server.
We have a few IBM NetVistas in our server room, and our main production and back up servers runs 4 dual core Xeons each. Then again, this beast runs our entire warehouse. Then there is the two fast racks fiber connecting to the drives aprox 40 140GB 15K drives.

Not to put your E-penis down or anythying Greg .. I'd be proud of it tooo.. you own it .. I am just the babysitter for the servers here lol

You have desktops in your server room? Boy... I need to come in and have a chat with you! Get rid of the ( I assume) System x's and get a BladeCenter... way better solution for back up and growth. 2 cents.

Shakeyhands · February 11, 2009

We considered buying our own server and putting it in the datacentre here at the University of Lethbridge, which was an option when we first started in 2000. However, after much consideration, I decided that the University IT department has better things to worry about than a box in the corner; particularly a box that isn't part of the universities core network.

IBM System i.... It's like the Ronco 'Set it and forget it" platform. Once set up you could literally sit in your corner and forget about it.

http://www-03.ibm.com/systems/ca/en/servers/i/index.html

Jeez, msg me next time.

GostHacked · February 13, 2009

You have desktops in your server room? Boy... I need to come in and have a chat with you! Get rid of the ( I assume) System x's and get a BladeCenter... way better solution for back up and growth. 2 cents.

I said the wrong machine ... only now do I realize this .. yes they are X series servers, fast racks, but no blades .. that is slated later this year when we go live with new production software (that's gonna be fun!!!) . I got into this role last year, and have been giving the network teams some headaches, because this place has not been on the radar for some time. Not to mention another fiber connection from the post to the server room will help with communication to T.O.

I been doing my best to get things up to date .. my new boss is great in this area of gettin 'er dun.

It might be old hardware, but it still runs like a charm. Don't need new shit if the old shit still works.

GostHacked · February 24, 2009

Greg and CO.

Do you have an issue with double posts? I have seen more than usual in the last couple weeks. It seemed to have just happened to me in a thread, a duplicate of my post was created. Just a heads up.