WordPress MU

hello,

i host few blogs on a single server ( Intel Xeon @ 2.80GHz with hyper-threading + 2GB ram + 36GB hdd SCSI ) that have around 32000 hits / day and a peak of around 50000 hits / day and between 250 and 1500-2000 comments / day spread around 2 of the bussiest blogs.

From time to time when there is something important announced and users refresh the pages like crazy and comment alot the server can't handle it and the load average gets very high and sites are not usable anymore.
Last time it happened yesterday and i tried to fix it somehow but nothing i did worked (changed some apache/mysql configs) so im asking for your help maybe there is something i can do so it can handle more hits without adding an extra server.
From what i could saw using top the mysql didn't seem to be the problem (it used ~2% memory and 10-20% max cpu) but few apache processes that were eating each between 20% and 40% cpu, but maybe im wrong.

Server hosts 2 normal Wordpress blogs and 2 WordpressMu (16 blogs one the busy one and 65 the less busy one) with just Akismet plugin on all and ENABLE_CACHE set in wp-config.php

Below are my server configs, if i forgot anything that could be relevant please tell me.

My Apache/2.0.58 configs are:

KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 3
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 512
MaxClients 512
MaxRequestsPerChild 4000

Mysql 5.0.26 configs are:

skip-locking
key_buffer = 256M
max_allowed_packet = 1M
table_cache = 512
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_type = 1
query_cache_size = 50M
query_cache_limit = 5M
max_connections = 500
log-slow-queries = /var/log/mysql/slowquery
long_query_time = 4
thread_concurrency = 8

My PHP 5.1.6 configs:

max_execution_time = 30
max_input_time = 60
memory_limit = 256M
magic_quotes_gpc = On
mysql.allow_persistent = On

i also have APC with the following config:

apc.enabled="1"
apc.shm_segments="1"
apc.shm_size="30"
apc.optimization="0"
apc.num_files_hint="1024"
apc.ttl="7200"
apc.user_ttl="7200"
apc.gc_ttl="3600"
apc.cache_by_default="1"
apc.file_update_protection="2"
apc.enable_cli="0"
apc.max_file_size="1M"
apc.stat="1"
apc.write_lock="1"

Thank you for your time :)

Search for a thread about "calling all large hosts" or something like that.

We discuss quite a bit on mysql.conf.

I'd like to see a little more mem in mysql, but it doesn't appear to be eating up a lot of resources yet either.

I've had this same issue myself at times. It isn't often, but something weird goes down once every so often, and sends my load average up from about 1, to 20+. Only remedy is to get in and kill the apache processes that are spiking out.

Couple things to consider:

Make use of APC's filter attribute in httpd.conf, and make sure kses.php is excluded.

Look up the thread about modifying WP-Cache for Mu. It's about a 4 page thread, and a working solution starts with a post by Quenting on page 2.

I have a summary on page 3, I might have missed a part in it though, and then we discuss making sure it is enabled only for certain blogs.

That "should" help.

With comments, which will/should force WP-cache to nuke the cache and create a new one, I'd recommend AJAX comments, as it will lessen the load considerably with a bunch of comments coming in.

WP-Cache will still nuke the cache, but the upside to that is that the page will not be refreshed when a comment is submitted, due to the AJAX. So, that takes a load off a bit too.

you should disable kses in apc or you may face random segmentation faults at some stage.
You might want to check apc.php to see how much of those 30 megs are used. I generally give it 64 megs per MU install on the same box.
Make table_cache in mysql way higher. Like 10000.
Set keepalive in apache to 1 or 2.
Up min and max spare servers (i have something like 20-40 on most boxes).
How many online users do you have at peak ?
If as you say very few blogs receive most traffic, enable WP_CACHE on those blogs. The boost will be tremendous. See the thread as mentionned by luna. I ended up disabling ENABLE_CACHE myself, since it wasn't providing much, and since my mysql boxes are underloaded compared to apache boxes.
Ultimately at some stage you'll need to throw in another server...

Whatever happened to your nginx set up br41n?

What about the idea of moving all your static files to subdomain running nginx or something like that?

There seems to be a massive overhead created with the admin system. Consider what you have on the new post page:

../wp-admin/wp-admin.css?version=wordpress-mu-1.2.3-2.2.1
../wp-includes/js/fat.js?ver=1.0-RC1_3660
../wp-includes/js/dbx.js?ver=2.05
../wp-admin/dbx-admin-key.js?ver=20070417&pagenow=post.php
../wp-includes/js/prototype.js?ver=1.5.0-0
../wp-includes/js/wp-ajax.js?ver=20070306
../wp-includes/js/list-manipulation.js?ver=20070306
../wp-admin/custom-fields.js?ver=3733
../wp-admin/cat.js?ver=20070417
../wp-includes/js/tinymce/tiny_mce_gzip.php?ver=20070326
../wp-includes/js/tinymce/tiny_mce_config.php?ver=20070225
../wp-includes/js/jquery/jquery.js?ver=1.1.2
../wp-includes/js/scriptaculous/wp-scriptaculous.js?ver=1.7.0
../wp-includes/js/scriptaculous/builder.js?ver=1.7.0
../wp-includes/js/scriptaculous/effects.js?ver=1.7.0
../wp-includes/js/scriptaculous/dragdrop.js?ver=1.7.0
../wp-includes/js/scriptaculous/slider.js?ver=1.7.0
../wp-includes/js/scriptaculous/controls.js?ver=1.7.0
../wp-admin/admin-ajax.php

Turn off the rich text editor and get rid of half of that. :)

The good news is that those should only be loading once and not every time.

I think turning off the rich text editor means less users.

The good news is that those should only be loading once and not every time.

Ultimate it all adds up. What would be cool is setting the image uploader on another seperate subdomain as well.

Or maybe some plugins for photobucket etc...

There is a series of interesting articles about how real performance gains are found by working on the front end not the back end. As my site is on an Intranet I don't have to worry about these things.

You can read more at:

http://developer.yahoo.net/blog/archives/2007/03/high_performanc.html

Richard

Um, isn't that like a given?
Of course, ESPN.com and FoxSports.com could take some lessons from that one!

Well, I guess it's a given if you read Zeldman and Meyers, and that sort of thing. :)

Um, isn't that like a given?

Not to me!

When we talk about increasing performance; I think making queries faster, caching responses, streamlining code, some have even suggested hard coding instead of bloated plugins :) All back end stuff.

As far as front end goes all I usually do is make sure my images aren't too big and rely on defaults for the rest.

Richard

I think turning off the rich text editor means less users.

Actually I disagree with that. One of the most made suggestions over in wp.com land. That and "logout, clear your browser's file and cookie cache, and try again."

I agree that the "rich editor" tends to cause more problems than it's worth.

The basic editor has enough buttons to play with, and can do enough that it comes out fine.

As a note, is it still necessary to modify tinymce so that it can be cached?

if i turned off rich editor, I'd be left with 2% of my users. This is really not good advice to turn it off if your audience isn't at least a little geeky. My users don't understand, for the most, the concept of tags and you'd want them to use this crappy buttons ? You'd want them to remember the actual code to insert a video or something ? C'mon, not everyone has a clue about HTML.

Depends on what you let users do, really.

As with anything, target audience dictates addition/removal of anything.

What's the difference between an html tag and a bb code tag in a forum?

"Most" forums don't have big bloated editors. Heck, this forum doesn't have them at all. Granted, the main WP forums do, but it sure isn't tinymce.

So if a button inserts

[b] or <b>

what's the diff?

Sometimes, less is more.

I think it really depends on your target audience with regards to TinyMCE.

well, my users wouldn't get bbcodes either. Sometimes they ask me what "URL" means :-p. Indeed, it really depends on your audience, but i think it'd be a mistake to assume some things you find trivial are going to be found trivial by a general audience.