WordPress MU

Well the aim of this thread is to see if there are large wpmu hosts (in terms of number of blogs hosted) visiting here regularly, and maybe do so sharing of specific tweaks we may have applied to have wpmu behave better / done specifically due to large numbers of visitors. Probably to be considered a "large" host in wpmu terms would mean at least 1000 blogs hosted.

I run a french language wpmu blog hosting service, got past 12000 blogs hosted today. It is about 1 month and a half old, growing quickly (300-400 new blogs daily, not all of them becoming active though).
It runs with other sites (a few of them quite popular) off a dual xeon 3.2 box with 2 gigs of ram and scsi raid disks.
I'm having it updated to 4 gigs due to swap problems, that seem to be related to the growth of the blog hosting site.

I'd be particularly interested myself in discussing database optimizations / changes others may have made on the db side.
- Has any of you made changes in mysql configuration that lead to significant improvements of performance?
- How do you manage a database of multiple thousands tables?
- Has anyone implemented yet a multiple database instances installation?
- In such case, how did you do it, and how did you migrate from the single db to multiple db version?
- Talking about migration, has anyone tried yet moving versions of wpmu, what is your feedback on how easy / complex that was?

It'd be nice if we could get this thread into a tip-sharing discussion for handling large wpmu sites.

wordpress.com - hosting 230,000+ blogs :)

I've had to raise the number of open files mysql can have, but the error I was getting was ONLY during backups using mysqldump, normal operation wasn't affected.

I raised the number by editing /etc/my.cnf and setting open-files-limit=8192 and restarted mysql.

As for performance, I don't think the biggest problem is the database, it's the overhead of compiling and serving the same php files over and over. I gained a 10x speed boost (and much lower load averages) by installing eAccelerator.

Lastly, it should be noted that the Wordpress object cache is no longer enabled by default. It caches database responses to disk bypassing db access whenever possible.

To enable it, edit wp-settings.php and add the following line immediately before the line that says wp_cache_init():

define('ENABLE_CACHE',true);

thanks guys for your feedback.

donncha - impressive number for wp.com. Are these blogs all served from the same database or multiple? How many servers, and what kind of configuration is used? Is the software close to the wpmu release, or has it been hacked a lot just for wp.com needs?

itdmanager - thanks for the tip, although i had already the same open files as yours. I'm looking at the database side of things because currently I have more problems with ram getting full than CPU getting overloaded. Have you experienced any other problems during backups? How long do they last generally? I'm a bit concerned about the number of files in the database (10000 users = 300000 files in the mysql folder). What's your settings for table_cache, thread_cache, query_cache?
Any drawbacks to using wordpress' database cache?

It's documented elsewhere but there are multiple database servers, and between them the user database is split between 4096 MySQL databases. Slave servers are used to distribute reads.
There are multiple web servers too, all running Litespeed for performance reasons.
Open files and security limits for files open are set to 64k or more for each box and each box is tuned as much as we can.
We use wp_cache as well for the most popular blogs on the site which helps a lot.

Don't use innodb for the db. It doesn't let go of memory when tables are opened. When you have many tables this causes a couple of problems..

thanks, excellent info.

>> between them the user database is split between 4096 MySQL databases

may I ask what was needed to achieve the splitting? in wp-config there seems to be support for multiple DB included, but this is not documented, and it's not very clear to me:
1) if this infrastructure is functionnal, supported, needs to be tweaked
2) what needs to be done to set up the multiple databases layout
3) what would be the steps to migrate from the default mono-database layout to the multiple DBs one.

>> It's documented elsewhere

where? googled for it but didn't find anything.

>> Open files and security limits for files open are set to 64k or more

are you talking about mysql table_cache, open_files_limit, or linux file descriptors? I'm not sure i get what you reference with these two parameters.

Are you for hire for a simple mission such as migrating the mono-DB version to a multiple-DB one? quentin at unblog dot fr if interested ;-).

We have a highly customized version of wp-db.php on wordpress.com where requests are sent to different databases based on an md5 hash of the blog ID.

Matt has talked about it on his blog on occasion -> http://photomatt.net/

I'm refering to both mysql and linux limits. We did the usual tuning anyone does on a large db.

Unfortunately I can't work on a competing site, but the multi-db stuff will make it into the open sooner or later. We want to get a stable release out there first before making big additions to the codebase!

So far, I haven't had any other problems with db backups, and they only take a few seconds for the dump to run. I'm sure there are better ways to do it because the dump write-locks the db, but again only for a second or two. I haven't had to change any other settings from defaults. As for wp-cache, it works perfectly and dramatically reduces db load.

Also, I'm still using stock mysql rpm on rh fc2 which seems hard set to use 256mb of ram, never more, never less. Maybe thats why I don't have ram issues.

How did you guys get so many people to go to your site? My main problem is getting traffic :(

itd: how much blogs are you handling out of 256mb ram ???

donncha: well, without helping directly then (although i don't really have a competing site since it's french), could you hint on something:
You said you had a custom db script, which i can imagine how it can be coded. However, what I'm wondering is the following:
- do you figure the DB to connect to from the URL the user is connected to?
- is the db connection script the only one that needs modification for things to work properly?
- how do you handle sitewide stuff, like searches from the admin panel?
- do you have the main wpmu tables replicated to each 4096 databases?

Thanks!

quenting: The mysql process alone uses a constant 256MB of ram. The server has 2GB.

Farkel: Try showing some leg :-)

I have another question on scaling:
OK so the users have their own tables, which means there should be not too many issues dispatching them on multiple servers/DBs.
What about disk space? Currently all user upload space is gathered in wp-content/blogs.dir, but that folder/disk may not be extensible enough to handle all uploaded content.
Has anyone got ideas on how to scale disk as well as databases? Do you need multiple web servers too, and dispatch requests on them also based on some sort of blogid's hash ?

You'll need multiple web servers, an nfs server or two, something to rsync nfs servers if you have more than 1, you should hash the blog_id because Linux can only store 32,000 directories in one directory. Once you have more than 32,000 blogs you'll run into trouble.
Disk space is cheap though in comparision to other elements!

One of the things I do with a new box is juggle the numbers in the my.cnf file for MySQL. Here's a quick thread on the subject over at ev1. Mine vary from box to box depending on what's on it. (ie How many vBulliten installs on each one. Never seen a worse way to leek memory...)

quenting take a look at Perlbal which is a reverse-proxy to on your web frontend server(s). That can reverse proxy to multiple webservers which serve up the content for the blogs. You can also run a couple instances of memcached instances instead of caching data to disk to reduce the load on your MySQL servers. Look at using mogilefs for storing user content.

Splitting users up into over database clusters with a global wpmu database is suggested. Look at the various presentations from LiveJournal on how they scaled their infrastructure and learn from their mistakes which they made in the past. Start using their tools like perlbal, memcached, etc.

bsdguru, I thought Livejournal used pound.

thanks bsdguru.
I started things up using simply apache's mod_proxy for now. When perfs become a problem on the proxy server, I'll look into something more efficient, but for now mod_rewrite is convenient.

Guys, I found another reverse proxy that's worth looking at.

Varnish was written by Poul-Henning Kamp for the most popular web site in Norway. Prior to Varnish, the website (Norways largest newspaper VG) used 12 squid servers. Now they are using 1 server running Varnish (i.e. Varnish is 10-12 times as fast as squid running on the same hardware).
Varnish has a BSD license.

thanks for the heads up. Do you know if it allows for routing depending on the URL like mod_rewrite does ?
Have you implemented a reverse proxy too ? If yes maybe we could share tips.

quenting, yes it can route based on the URL. Varnish is using a config language called VCL. See the second example.
I haven't implemented a reverse proxy yet, I'm still planning the platform configuration. So far I've ended up with this config (on paper):

• Zend Platform for caching and debuging
• Your DB config
• My file structure
• Varnish as reverse proxy

Btw: the Zend Platform is free for application development, I really recomend using it for debuging.

cool, looking forward to you pushing things forward so we can discuss some issues.
Typically, one thing i'd like to do is be able to serve images from a lightweight http server. Currently this isn't easy due to the need to retrieve blog id to access the right upload folder, which means php and mysql for each served image (inneficient !).

Donncha, how are you running Litespeed? I thought wpmu is only compatible with apache.

Litespeed? It support mod_rewrite fairly well and PHP of course so it works.

Apache is the recommended platform simply because it's the easiest to get working but other httpd servers will work too.

Litespeed's pricing is a little confusing, so I probably won't use it? Does layeredtech offer Litespeed?

I don't see it on their website, server setup or forums. Maybe a custom setup though.

You could also check lighttpd and nginx (which im currently using)they are both fast httpd servers and reverse proxy and more.

br41n, are they Apache compatible or Apache alternatives?

As I read lighthttp, it was apache with a cache but I'm not familiar with it.

Must be half decent though since it's running wp.com.

Um, wp.com, .org, etc are all on Litespeed.

I'd use it, if I could afford it.