The MU forums have moved to WordPress.org

WordPress MU Forums » How-To and Troubleshooting

Upgrading a very large site (22 posts)

  1. Farms2
    Member
    Posted 17 years ago #

    Well, first the good news... you can safely upgrade from a January version of WPMU to RC4 without too much pain - I know I've done it / am doing it.

    The not so good news is that the bigger your site the more of a pain it is.

    It seems like the speed of the Site Admin auto upgrade script caller is directly proportional to teh size of your site - for example I've had uniblogs.org (1 Gig SQL) upgrade running for a couple of hours now an it's upgraded 1730 sites... whereas learnerblogs.org (on the same server and with the same config) but with a 4 Gig database - is only on 400.

    But I thought I could be clever and direct users to maintenance screen while giving the upgrade all the cpu grunt it needed by putting this at the start of my .htaccess to allow only me to access the site and send other people off to a maintenance page:

    RewriteCond %{REMOTE_ADDR} !220.234.169.04
    RewriteCond %{REQUEST_URI} !^/maintenance.html
    RewriteRule ^(.*)$ http://incsub.org/learnerblogs/maintenance.html [r=301,L]

    And it worked!

    In that it allowed only my IP to access it... when I went to run the upgrade script it ran like a beauty (dead fast too) but without, um, upgrading anything.

    So I'm back where I started - blogs giving errors over a 24 hr+ period while the script plugs through them :(

    And that's just learnerblogs... edublogs is going to be a completely different kettle of fish.

    Any suggestions / ideas?

    Cheers, James

  2. mrball
    Member
    Posted 17 years ago #

    An upgrade plugin that calls the upgrade script the first time a blog is visited and the version number increases.

  3. Farms2
    Member
    Posted 17 years ago #

    Oooooo that'd absolutely rock. I'd pay for that.

  4. Farms2
    Member
    Posted 17 years ago #

    learnerblogs still going BTW, been at it for 20 hours now and is at 7135...

  5. andrewbillits
    Member
    Posted 17 years ago #

    At the very least I think the upgrade script should give you the option to upgrade say 100(or more) blogs at a time. That way you could upgrade over the period of a few days and during the hours in which the site has less traffic.

  6. Farms2
    Member
    Posted 17 years ago #

    Yeh, I thought of that... shou;dn't be that hard to hack I imagine.

    Dunno if it would make it any quicker though... and a big part of my problem is that once the files have been updated (from my Jan build) to RC4 then all blogs go 'nothing found' until their upgrade scripts are run... so it has to all happen at once.

  7. mrball
    Member
    Posted 17 years ago #

    farms: i'd give it away... if i ever do get round to writing one

  8. Farms2
    Member
    Posted 17 years ago #

    Well I'd have to buy you a beer at the very least :)

    I'm wondering if the problems I've been experiencing might be an SQL config issue.

    Correct me if I'm wrong but each upgrade script goes and does a heck of a lot of things to that database, does that mean it completely clogs the allowed accesses?

    Basic question: could a tweak to SQL .cnf assist in speeding up the upgrade of a large site?

    What would the settings have to be like?

  9. mrball
    Member
    Posted 17 years ago #

    farms: had a quick look at your first post again- the reason why your upgrade script

    "ran like a beauty (dead fast too) but without, um, upgrading anything."

    is because what the script basically does is request the upgrade script from each of your blogs via normal http- which you have redirected with your .htaccess to a maintenance page.

    about the SQL config, it is unlikely any tweaks will significantly speed up the upgrade. the bottleneck is the method of upgrading (i.e., calling each blogs upgrade script via normal http) which is not only slow, but also, as you have demonstrated, unreliable (it didnt even tell you it was being redirected to a maintenance page... what if database connections were full on that call? etc etc)

  10. Farms2
    Member
    Posted 17 years ago #

    Thanks mrball.

    It's been a bot of a nightmare actually - everything ground to a halt on 7500 upgraded and I ran out of weekend time / sleep to fix it... got it running again last night though and bu this morning it had almost made 12k... only 3 to go (fingers crossed!).

    That's a 4 day upgrade for 15,000 blogs...

    Edublogs has over 25K blogs on it though :( It's almost like I'd have to accept that the site would be trashed for up to a week to upgrade and that I wouldn't be able to communicate to users that we were upgrading anyway...

    ... and naturally the problem gets worse every day.

    Bugger.

  11. mrball
    Member
    Posted 17 years ago #

    ok farms, an ad hoc script just for you, just dump it in mu-plugins and upgrade the site per normal. You can remove it once your site has finished upgrading:


    <?php

    // Adhoc WPMU upgrade script. Put in mu-plugins, and run site upgrade per normal. http://jason.lah.cc

    $mu_upgrade_version='784'; //change to current version, will trigger upgrade as long as it is different

    if(get_option('mu_upgrade_version')!=$mu_upgrade_version) {

    $http_fopen = ini_get("allow_url_fopen");
    if(!$http_fopen) require_once('../../wp-includes/class-snoopy.php');
    if($http_fopen) {
    var_dump($fp = fopen( get_option('siteurl') . "/wp-admin/upgrade.php?step=1", "r" ));
    if( $fp ) {
    while( feof( $fp ) == false ) {
    fgets($fp, 4096);
    }
    fclose( $fp );
    }
    } else {
    $client = new Snoopy();
    @$client->fetch(get_option('siteurl') . "/wp-admin/upgrade.php?step=1");
    }

    update_option('mu_upgrade_version',$mu_upgrade_version);
    }

    ?>

  12. mrball
    Member
    Posted 17 years ago #

    all usual disclaimers apply, no guarantees here! ;)

  13. Farms2
    Member
    Posted 17 years ago #

    Like wow mrball -yerran absolute legend - I'll test it on some smaller installs tonight.

    Thanx big time.

  14. Farms2
    Member
    Posted 17 years ago #

    Well that was pretty cool... worked a treat for when a site was visited (i.e. it updated it) so that's pretty damn ace :)

    One bug - it freezes the Site Admin upgrade script

    One feature request - If it could show something that tells the user that tehir blog is updating while it's doing it (rather than just a white screen for 30 seconds or more) then that'd be ace!

    But no stress... I'm just darn happy with it!

  15. umair
    Member
    Posted 17 years ago #

    guys i have a question !!

    for a large site having large number of blogs, is it better or necessary or what do you say it to divide the database or split the database on multiple servers. i mean to say that whether we should do it ???

  16. Farms2
    Member
    Posted 17 years ago #

    If you can it probably makes very good sense... it's not easy to manage one great multi-Gig SQL database.

  17. umair
    Member
    Posted 17 years ago #

    can anyone pls give me a link where i can find some tutorial on how to split database and how to manage a splitted database ??

  18. andrea_r
    Moderator
    Posted 17 years ago #

    There isn't.

  19. andrewbillits
    Member
    Posted 17 years ago #

    The good news is that since WPMU uses the wpdb class for literally everything, it's super easy to make a multiple db site and have everything upgrade to a newer version very easily.

  20. Farms2
    Member
    Posted 17 years ago #

    Any chance of a 'for dummies' guide... I'm kinda fond of them :)

  21. andrewbillits
    Member
    Posted 17 years ago #

    Welp, it kinda goes like this:

    1) decide how you wish to divide up the blogs. Whether it be by hashing the blog id or another method

    2) Create your own wpdb class that will interact with your database(s).

    That's about the best I can do without knowing how you'd like to divide up the blogs.

  22. andrewbillits
    Member
    Posted 17 years ago #

    Oh, btw. Unless you're on a dedicated machine (or maybe a VPS), there really isn't much of a point to splitting up the db. You really need more control of MySQL than what you get on a shared account to have *any* fault tolerance.

    This snippet of info isn't as much for farms(he's on a dedicated machine) as it is for other people starting to toy with multiple dbs. ;)

    Also, I just finished testing/toying with a multiple db setup similar to wp.com's hash-based system. I set it up on three boxes(one web, two db) running CentOS 4 and i'm here to tell ya, it's not the easiest thing on the planet.

About this Topic

  • Started 17 years ago by Farms2
  • Latest reply from andrewbillits