WordPress MU

Yesterday I checked in a new script at
NEW LINK:
http://trac.mu.wordpress.org/browser/trunk/wp-content/mu-plugins/sync-taxonomy.php?rev=1141
OLD LINK:
http://trac.mu.wordpress.org/browser/trunk/wp-content/mu-plugins/sync-taxonomy.php

The first thing you'll notice looking at it is that the whole thing is commented out. There's a good reason for that. I didn't want anyone blindly svnupping and enabling a major new bit of beta code. If you're going to try this, do it on a development server first please. I need feedback that it's working but I don't want anyone to screw up their database!

So what does it do?
It creates the new taxonomy tables and then synchronises operations on the old categories tables.

How do I run it?
1. First of all you'll have to remove the comments around the code. They are on lines 23 and 424 of that file.
2. If you want to upgrade every blog on your site, open the main page of your site and append "?upgradetaxonomy=1" to the end of the url. Your url should now look like http://example.com/blog/?upgradetaxonomy=1
3. That will generate a secret key in your site_options table called upgradetaxonomysecret. You'll have to examine that table in phpmyadmin or otherwise grab that value from the db to do a full site upgrade.
4. Once you have that key, open your main blog again like above but this time replace 1 with your secret key. It should be a long 32 character string of numbers and letters.
5. The URLs of blogs from your site will be printed on the page, 10 at a time, until the script runs out of blogs to work on. The blogs that are displayed should be upgraded and syncing until you upgrade your site to the new WPMU release.

How do I test it?
Once your blog is synchronising, do some admin work on the categories. Create a category, delete another, edit yet another one. Open your new taxonomy tables and verify the changes are carried over from the old categories tables.

What if I want to revert?
I also have code in the script to handle synchronising the other way from taxonomy to categories. This code is completely untested and may be removed before the final release.

If all that sounds like too much work, you could wait until the new release in the next week and upgrade your blogs through the normal wpmu-upgrade-site process and when your bloggers visit their backend the upgrade script will be called too in the background. Unfortunately on large and busy sites this is likely to cause load problems as many tables are created at the same time.

This release won't include a sitewide tags page but the site_categories table will be updated as per usual.

It creates the new taxonomy tables and then synchronises operations on the old categories tables.

Does this mean this mean that the old categories tables will be synced with the new tables or that the new tables will be synced with the categories tables and then the categories tabled removed?

The reason I ask is that several plugins we use rely on the categories table and I need to know whether or not I need to code new plugins prior to upgrading.

Also, are there any plans to add a site-side tag feature in the near future or is it safe to go ahead and start coding it myself?

Thanks,
Andrew

Before upgrading to the WP 2.3 codebase it synchronises the new taxonomy tables with the old categories table. After upgrading to 2.3 there's code in there to do the reverse, so that the old categories tables are kept in synch. That should only be a temporary situation until you know all your blogs are working ok, and you've updated your plugins.

Those plugins will have to be rewritten I'm afraid. If they go through the regular WP API then they won't need to be changed, but if they access the tables directly then the SQL has to be changed, and boy is it just a little more complicated now!

There's already the sitecategories table that makes sure the categories used on each blog are globally unique, and the taxonomy code will use that table too, but there isn't going to be a global tags page there. If you code it, attach it to a ticket and I'll hopefully get it into a minor release soon after the dust has settled on taxonomy.

That should only be a temporary situation until you know all your blogs are working ok, and you've updated your plugins.

That's good news. It won't take long to code new plugins but I didn't want to have to hold off on upgrading.

and boy is it just a little more complicated now!

No kidding, I looked at the code yesterday and nearly fell out. I'll definetely be using the API this time around.

Thanks,
Andrew

Oops, forgot something. Will the the sitecatagories table be synched with tags or catagories? Probably a dumb question but I just wanted to be sure.

Thanks,
Andrew

It'll be synched with both because the term_taxonomy_id which is a unique ID for a tag id+tag type is only unique local to each blog.

A tag called "test" and a category called "test" will both have the same cat_ID in sitecategories. It's only when they're used in a blog's taxonomy tables are they differenciated by the term_taxonomy_id.

Ryan has a great write up on what the new taxonomy tables look like. Definitely worth looking at.
http://boren.nu/archives/2007/08/26/wordpress-23-taxonomy-schema/

If it's still confusing, install WP 2.3 and examine the taxonomy tables. It should be reasonably clear after seeing a worked example.

Thanks for clarifying that. I just looked at the tables and I see what you mean.

One last question, will the sync script work with 1.2.4 or just 1.2.5a? Obviousely this isn't for us but we have a few clients who like to skip upgrades.

Thanks,
Andrew

1.2.5a was a security update with no db changes AFAIR so it should be fine to jump from 1.2.4 up!

Is the following anything to worry about:

[04-Oct-2007 19:58:09] WordPress database error: [Duplicate key name 'option_name']
ALTER TABLE wp_3_options ADD KEY option_name (option_name)

[04-Oct-2007 19:58:29] WordPress database error: [Duplicate key name 'option_name']
ALTER TABLE wp_2_options ADD KEY option_name (option_name)

[04-Oct-2007 19:59:55] WordPress database error: [Duplicate key name 'option_name']
ALTER TABLE wp_1_options ADD KEY option_name (option_name)

steveatty - where did that appear?

in my php error log when I went into a blog and created a new category

That's strange. That's after the backfill ran is it? What is your blog's db_version? Check your blog_options table for that.

Looks like some upgrade function tried to run.

They all claim to be 5200

It happens after I've run the back fill. All the other tables exist (terms, term_relationships, term_taxonomy) and they do contain data which does seem to make sense.

Hi,
I have a small site still running version 1. Can it be upgraded to the new version in one step?

Thanks!

In-between upgrades should automatically be handled by the upgrader, but YMMV.

Selad - Version 1? Try version 1.2.5a first and go to the site-upgrader in "WPMU Admin"

Steve - problem there. The update_option() at the end of the backfill should have set the db_version to 6124. I'll look into it. In the meantime, you could create a small plugin to do that update on db_version. Even with a db_version of 5200 none of the upgrade scripts should have run. It's strange.

i got this error:

Warning: require_once(../../wp-includes/class-snoopy.php) [function.require-once]: failed to open stream: No such file or directory in
/home_dir/wp-content/mu-plugins/sync-taxonomy.php on line 395

Fatal error: require_once() [function.require]: Failed opening required '../../wp-includes/class-snoopy.php' (include_path='.:/usr/local/php5/lib/php:/usr/local/lib/php') in /home_dir/wp-content/mu-plugins/sync-taxonomy.php on line 395

it works fine on a per blog basis, the whole WPMU update is the one that fails with this error. It seems it's because it can't fetch a file internally due to host security restrictions?

Didn't work for me. Here's what I did:

1. Uncommented the plugin
2. Ran the script w/o key
3. Ran the script w/ key. This showed the blogs updating, but nothing was synced.

I would create new categories and try again, but I must be doing something wrong.

Can anyone confirm this works? And maybe offer up some suggestions.

Some missing info: The script goes in the wp-content/mu-plugins/ directory and was actually already in there in my case. I downloaded that version, uncommented it and reuploaded it.

Having appended ?upgradetaxonomy=1 as instructed, there was no confirmation, but no errors,

The value of the key is not put in the site_options table, which doesn't actually exist, but in the wp_sitemeta table.

Having retrieved this and run
blog/?upgradetaxonomy=xxxxxxxxxxxxxxx
The script seems to cycle through the blogs, but doesn't actually make any changes to the tables.

Also, running
blog/subblog/?taxonomysync=xxxxxxxxxxxx
Doesn't seem to do anything.

So maybe the problem which is stopping the normal upgrade script from working is also stopping this from working.

"If all that sounds like too much work, you could wait until the new release in the next week and upgrade your blogs through the normal wpmu-upgrade-site process and when your bloggers visit their backend the upgrade script will be called too in the background."

Doesn't this mean that as 1.3 has now been released this script is redundant and replaced by the regular site upgrade function?

That's what I thought, but in my case, something seems to be preventing the upgrade from working. I'm currently investigating copying the three new "term" tables from a working blog to a non-working blog, and this seems to be a solution, if somewhat long-winded.
It also raises the question, which tables can safely be deleted as they are no longer needed?

Never delete anything if it can be avoided ;)

I installed wpmu 1.3 and 3 tables for each blog are created. Now do I have to uncomment those two lines in mu-plugins/sync-taxonomy.php?

From what I gather, if you upgraded and got the three term_ tables for each blog, then you're good to go. It worked!

My problem has been that the upgrade didn't do this for any of my pre-existing blogs, and running the sync-taxonomy.php script post-upgrade also has no effect. Blogs created since the upgrade are fine though.

Added modlook as the link in the first post is now incorrect. Can a staff member please correct it?

Link

I have tried and tried to upgrade a WPMU site from 1.25a to 1.3 without success using the "typical" upgrade instructions and these special taxonomy upgrade instructions. No love in either case.

I am not sure where to start, but here goes.

When I try to upgrade, the problem I run into is I lose all of my categories. I made another post about this and was subsequently referred to this one. I've tried following the steps outlined here, although if I'm not mistaken, I should not have to follow these because the latest download package automates this process (???) The last time I tried to upgrade was on 7 December 2007, before the holidays hit. I rolled back and put this off til after the first of the year, buying myself some time with the users of the system.

But they are of course wanting the latest and greatest, and I want to deliver. So, I set up a 1.25a install on my test box yesterday and populated it with some test posts and categories. I downloaded the latest 1.3 package and went through the "typical" upgrade steps, didn't do a thing with taxonomy, and GLORY BE the categories imported properly and the site works great!

So, I figured I'd give it a go on the live site tonight in hopes that it would work. And I'm right back at square one with the categories not working. I tried running the taxonomy script and nothing appears to happen, and I cannot find a secret key in the options table. When I go to the categories table via phpMyAdmin , the categories are all listed as they should be.

On the site itself, all posts are now "uncategorized" and going to Site Admin - Manage - Categories shows that there are no categories, not even "uncategorized" which is usually there by default. But they are still in the database.

So, what am I doing wrong? Also, perhaps I'm just overlooking or can't read well enough between the lines, but I'd really like to know definitively - do I, or do I *not* need to manually run this taxonomy script? I did not need to run it on my test site.

Help! Thank you so much. :)

Have you tried taking a copy of the live site, including db, and then try the upgrade locally to see what happens?

I've done it for several sites, and it's worked fine. I will note that even though the upgrade script is run, as usual with upgrading, it doesn't necessarily update every blog. However, once the admin area of a blog is visited, it's upgraded flawlessly.

Safest bet is to mirror your live site locally, including the db, then make a copy of the db so you have an initial one to make additional copies of for multiple tests, and see how it goes.

Hi Luna, thank you for the quick reply. :)

My live site is run on a server that I have full access to; this is something we are running on a corporate intranet. My test server is just another SLES box under my desk that I use as a sandbox.

I tried to mirror the live site on the sandbox, including the DB and the problem I ran into was all of the links are hard coding back to the live server.

I should mention that back in September when we first started this project, the user who manages the site content had us set things up as subdomains initially, but then upper management decided that they wanted to have a subdirectory structure. Unfortunately the user had already populated the DB with a bunch of data and we were trying to avoid a complete start-over from scratch, so we tried converting. That was *PAINFUL.* The initial conversion we attempted failed miserably, and we ended up pulling an all-nighter to manually sift through the database and change all subdomain references to the new subdirectory structure. It's been working great since then for the most part, but now I'm questioning whether or not anything from that effort is sabotaging this upgrade attempt.

Any thoughts?

That could be the case.

However, through the miracle of the hosts file you can map a domain to your test box when you need to work on it.

Realistically, even developing on one domain and then moving shouldn't be too big of a deal if you set it up right, and won't be moving the database over.

But, now we have your predicament.

I'm actually working with a company now that had previously hired someone to work on their site. Funny thing about it, they never upgraded. Well, let me rephrase that. They selectively upgraded certain files, but never fully because they hacked everything to bits. I'm all for editing the core when it's prudent, but this person hacked all kinds of needless things.

This particular client of mine actually wanted some new visual stuff first, but now that's wrapping up to where there's some new "pretty" stuff for the management to see, where we can now get into digging into getting the site back on track with updates and such.

When I first took it over, their site was literally pulling about 1000 queries per page. Without optimizing, I've cut it almost in half. When I get into the optimization phase though, it'll be loading 50 or less. Talk about a huge difference in load times and processing power right there.

Anyway, I don't think what you're trying to do is unreasonable, but there has to be a reason why a local test would be OK, then the remote fails.

So, the obvious is what's different live than local?

Software? Paths? Config? Memory? The domain is a given, since you mentioned it, but what else? PHP? MySQL?

Here's my suggestion, without typing a book (further).

First, dump your live db and import it locally.

Second, copy over all the live files to the local box.

Next, edit your hosts file so that you map example.com to the IP address of your local box. When you need to go to the actual live site, you can re-edit your hosts file and put a # at the beginning of that line (restarting your browser if need be).

Now, slight modification to wp-config aside, you should have a local copy of your live site presented to you, but running off your local box.

Next, make a copy of the database and name it so that you know it's a copy of the original that's untouched. (This comes in handy for later, if you need to make a new copy of that to use to play with.)

OK, now you're running. Try the upgrade. As of the 1.3 release, it should be as simple as replacing files (as usual) and then running the upgrade script in the back.

Individual sites may/may not be upgraded, but once their backend is hit they'll be upgraded instantly. If you don't have too many sites, just check your site admin->blogs list, and open each backend from the link (in a new tab/window, whatever you prefer).

Now, this depends on the version you have running currently too. You may have to go back through trac, and do some upgrading before you can upgrade, if that makes sense.

Like if you were on 1.2.3, you may need to upgrade (with the script, not necessarily replacing all the files) before you can move over the 1.3 files and then run that upgrade script.

Off the top of my head, I'm not sure how far back 1.3 will upgrade a database. If you're running really old code though, you're probably going to have to run the upgrade files from earlier versions though, before doing 1.3.

Initially I started out just using a different domain. Like .net vice .com, and mapping the .net permanently to my local server. It works OK for playing with themes and such, but once your database starts to grow, it's more hassle than it's worth to try and find/replace a domain name. So, that's where I'm going with mapping the live domain to the local box when you need to work on it.

Anywho... that should give you a good start on it.

Luna - thank you SO MUCH. :) You really helped out on this. Here's what I ended up doing:

1) Copied WPMU folder from live server to test server to /srv/www/htdocs

2) Exported DB from live server and copied to test server

3) Changed HOSTS file to trick it into thinking test server was live server

4) Created new user and new DB on test server and imported DB SQL file

5) Ran Upgrade under Site Admin

It worked! Categories were as they should be.

So, I then re-exported the newly upgraded DB from the test server and copied the WPMU folder and brought them over to the live server. I overwrote the folder, and I ended up dropping both the DB and the User and recreating them, then importing.

Initially, it seemed a total success, but then I found that some scripts were not executing. Thankfully, my boyfriend came in with an assist (he's a Linux guy) and noted the memory errors in the Apache log. I could have SWORN I already changed this ages ago, but I still had the default 8MB memory_limit set in the PHP.INI file. D'OH!!!! We upped that and the scripts execute properly now!

In retrospect, I think this was probably the cause of the upgrade not working properly on the live server all along, because that is the only real difference in configuration between the test and live servers I found. It just goes to show, always double check your work. :)

This is a fantastic way to leave work and start my weekend, feeling accomplished and having solved a problem. I am very grateful to have had your assistance in making this happen, Luna. If we ever happen to be in the same pub, I'm buying the first round. :)

Thanks again, very much.

Trase