The MU forums have moved to WordPress.org

Simple 'how-to' required on mega scaling WordPressMu (more than 3200 blogs!) (14 posts)

  1. ninjaboy
    Member
    Posted 13 years ago #

    I was wondering if there was a simple (complete!) guide somewhere lurking on the net on how to scale a WordPress Mu site above the theoretical limit of 32000 blogs created? I understand that the main issue is the nestled uploads directories for each individual blog (yes, each will need uploads).

    We are expecting to have many, many more than this after only a few months of site go-live and I want to put the scalability in-place from day 1 so I don't have to reverse engineer anything in the future.

    The site I am working on is initially going to run off 2 dedicated servers - server 1 was going to be used to run the Mu site, server 2 is being used for some extra 'non-WP' functionality for the site, but could be used for file storage and to scale some of the Mu site across to).

    I have been using WP for over 4 years, but this is my first proper Mu project. Although I have been checking out various pages on the net about this there doesn't seem to be any clear, definitive up-to-date answers for this. More a bunch of gems of knowledge spread around various sources.

    If I can't find a solution I may have to go to the darkside (another platform) which is something I REALLY don't want to do as I love WP so much!!

    Please - I could really do with some solid advice on this!

  2. ninjaboy
    Member
    Posted 13 years ago #

    I have had a look round on this subject and was hoping to find a good guide on this, but am gathering information slowly but surely.

    Am I right in thinking Hyper DB will only deal with the database scaling - which leads me back to being a bit puzzled on a clear answer about how to get around this 32,000 limit for file directories (well, preferably distributing the uploads and individual blogs between servers from the start really).

    Any advice really would be appreciated!

  3. Farms
    Member
    Posted 13 years ago #

    At Edublogs.org we use Multi DB.

    It comes with pretty comprehensive installation instructions and tons of help and support via the WPMU DEV forums.

  4. ninjaboy
    Member
    Posted 13 years ago #

    Yes, I have been checking out your site - very nice!

    Can I ask - how does Multi DB cope with the uploads directory issues for multiple blogs (ie, does it distribute those across multiple servers too?) I have to say I'm pretty much sold on Multi DB as a solution, looks very neat and tidy!

    Are there any other scaling issues that I need to be concerned about - I'm trying to set this site up with capacity for the future and have 2 dedicated servers at my disposal that will be purely dedicated to this single site (with potential to scale to more servers if required).

    One final point - is it really best practice with Mu to setup the sub-domain URL structure rather than sub-directory... will it make my life easier in the long-run or doesn't it matter?

    Cheers for the advice - and I hope that some other helpful person can respond to this topic. I'm not a WordPress noob (far from it!) but this is my first major Mu project (apart from samller testbed installs) so I would appreciate anyone's opinion on this please!

  5. ninjaboy
    Member
    Posted 13 years ago #

    Considering that WordPress Mu is shortly going to be rolled into the core code of WordPress, and a big trending topic is 'enterprise' usage I'm suprised that there is not more info around on this subject... what's the big secret?

    I have spent my time and done quite a bit of research around the subject. I've seen the MD5 trick for directories (ok, it works... but shesh!) and a couple of other suggestions.

    Am I missing something here - I'd appreciate your opinions!

  6. tdjcbe
    Member
    Posted 13 years ago #

    I don't have the code handy buy modifying the md5 hash to instead use the first three characters of the hashed blog_id works well for us. Going (I believe) eight directory layers is just too much and a waste of overhead for us. One layer for us is fine when you remember that first layer is still 4k directories with it being broken up underneath that by blogid.

    We have a single install set on two servers. To be honest, it's only running about 40k blogs with about 25% being active. It's more of an ego thing than a requirement though. The second server holds the databases and the uploads while the first server holds the install and the locally cached files.

  7. andrea_r
    Moderator
    Posted 13 years ago #

    Have you read all of Barry's posts about the wordpress.com setup?

  8. tdjcbe
    Member
    Posted 13 years ago #

  9. andrea_r
    Moderator
    Posted 13 years ago #

    Thanks, couldn't remember, was in an hurry before the power went off (again).

  10. ninjaboy
    Member
    Posted 13 years ago #

    Cheers guys, I do appreciate your posts and a nice alternative to the MD5 directory trick there!

    I will take a good look over Barrys Blog, looks like there is some good stuff in there. I'm still a-little puzzled that there is not a clear, all-in-one description for more 'enterprise' usage of Mu. I'm used to WordPress (normal) where pretty much all the info I need is easy to lay my hands on (or I already know it... racked up over 4 years, I love WP!).

    In my very humble opinion 'Out the box scalability' should surely be a core feature of Mu, configured through wp-config or similar. I sadly don't have the skills or resources to contribute to the project, but if I did this would be top of the list! I'm not complaining - WP rocks and Mu is an amazing achievement - I encourage and promote its use wherever I can, but this surely fairly common Mu task is trickier than expected!

  11. andrea_r
    Moderator
    Posted 13 years ago #

    To be fair, though - the scaling has far more to do with server tech in general, not MU.

  12. ninjaboy
    Member
    Posted 13 years ago #

    Yes, that is a fiar comment - I agree there and understand that many users of Mu probably would never get past 10,000 blogs - but on this project I'm potentially looking at 100,000's of blogs. From day 1 go-live it's being run off 2 dedicated servers with potential for more if required as the site grows (a luxury for me - I don't usually have this kind of capacity!)

    I'm really being drawn to http://premium.wpmudev.org/project/multi-db - does anyone have any experience of this?

  13. tdjcbe
    Member
    Posted 13 years ago #

    the scaling has far more to do with server tech in general

    Ego too. I hear tell my soccer mom client who runs the site likes to brag about being a server tech as well.

    We run the multidb code on all of our installs right from the start, even if there's no plans to expand. While there's a script in there to move the tables around, it's just easier.

    I'm still a-little puzzled that there is not a clear, all-in-one description for more 'enterprise' usage of Mu.

    You also have to remember that at that level, the larger installs are commercial and you run into trade secrets and folks being business competitors. Even Automattic and wp.com doesn't give away all of their secrets and code.

  14. ninjaboy
    Member
    Posted 13 years ago #

    You also have to remember that at that level, the larger installs are commercial and you run into trade secrets and folks being business competitors. Even Automattic and wp.com doesn't give away all of their secrets and code.

    True! Bahhh, looks like I've got a-lot of R&D in-front of me!

    Luckily I'm writing every plugin that is going to be used on the site from scratch (not too many hopefully, and at the moment there doesn't seem to be any user admin required for plugin code which should save me some trouble) so I'm feeling fairly confident that I wont get problems there at-least!

About this Topic

  • Started 13 years ago by ninjaboy
  • Latest reply from ninjaboy