The MU forums have moved to WordPress.org

Can WPMU support ~50,000 active blogs ? or more (19 posts)

  1. hitrick
    Member
    Posted 15 years ago #

    Hi
    Can WPMU as-is support around 50000 active blogs ?
    I saw some articles talking about performance problem – some of it due to the DB design .
    50,000 blogs – result something like 500,000 tables in mySQL …

    Does anyone have some experience with WPMU in an enterprise environment?
    Ps: I can only guess that sites like friendster.com made some major changes in WPMU so it's serving so many users per day

    Thank You
    Sincerely
    Hit Rick

  2. VentureMaker
    Member
    Posted 15 years ago #

    Well, having 50000 active blogs raises not only WPMU-related questions :)

    First of all, bandwidth. Second - storage in case you give a considerable amount of disk space for each blog.
    Third - server resources, and MySQL is not of the 1-st place. Apache + PHP in CGI mode require a good CPU and decent amount of RAM.

    What pertains MySQL - if set it on a separate server and configure mysql to run in multi-process mode then you'll have no problems.
    Of course, Multi-DB plugin for WPMU will also help.

  3. hitrick
    Member
    Posted 15 years ago #

    Thanks
    Let say that hardware ,bandwidth , storage aren't the problem now for the discussion .

    I'm asking more from DBA point of view .
    let's assume that I have in my club\community 500K members ,
    I want to open automaticly blog for each member
    (I know that something like 10% of them - thats 50K members will be active in the blog for the beginning ) .
    so I have to plan DB with something like 5 milion tables !! (10 tables for each blog) -
    this is a nightmare for every DBA , and a very wrong DB design as I know it "by the book" and from experience as DBA and software engineer .
    I hope You will show me the light here :-)

    Thanks

  4. andrea_r
    Moderator
    Posted 15 years ago #

    So you have multiple databases.

    WP.com has millions of blogs.

    Edublogs.org has hundreds of thousands of blogs.

    They use multiple databases.

    (it's more like 8 tables per blog. In our DB experiences, gong back quite a ways, it six of one, half-a-dozen of the other. You still have to store the same amount of info.)

  5. VentureMaker
    Member
    Posted 15 years ago #

    Is there a difference how large your DB is?
    The difference is how efficiently it is used.
    If you use multi-db plugin for WPMU you'll get X of smaller DBs, if you configure MySQL to run in multi-process mode and give it enough RAM and a decent CPU - there'll be no problems.

    BTW, the latest HSphere has some load balancing capabilities for web and mysql built in, so you might consider using it.

  6. lunabyte
    Member
    Posted 15 years ago #

    Honestly, the tables per blog vs. single table debate has been beat to death over the past few years here.

    Granted, that was before MySQL was pretty sound with things like sharding.

    While I will agree that in a best case scenario that singular tables (InnoDB, with OurDelta fixes) would be great, that's not something MU does for various reasons that we decided long ago.

    Unfortunately, I doubt we'll see a change in this policy.

    Pretty much what Andrea said, for something that large the most common solution is breaking the tables up into multiple databases. Typically done by doing an MD5 hash on the blog id in the db class, by stripping the blog id out of the table name. Then taking the first 1, 2 or 3 characters of that hash to designate which db to connect to. (Making for 16, 256, or 4096 databases.)

    By default MU has 8 tables for each blog, as mentioned. With plugins that can get up to 15 or 20, or more depending on the plugin. Yep, it's a lot.

    One easy note would be to not "auto create" a blog for a user. If they want one, let them sign up for it. You could pre-populate their info for them, but let them make the choice. I'd bet that you'll find that your 10% conversion rate is a bit high. However, that's a good thing. That means (usually) that someone who has a blog intends to use it, and that you're not wasting space or resources on inactive folks who didn't really want one to begin with.

  7. hitrick
    Member
    Posted 15 years ago #

    http://codex.wordpress.org/WPMU_Database_Description

    quotation : "Every new blog your users create entails creating a new set of these 10 or so tables. So if you have 1,000 users you will have 10,000 tables. These become of the format wp_<n>_tablename. "

    they talking about 1:10 (users / Tables)

    do You know for the example that WP.com didn't change the DB structure ?
    We talking here about using dynamic tsql on few DB's or even on more then one mySQL instances .

    10x

  8. hitrick
    Member
    Posted 15 years ago #

    My last replt was for andrea_r

    lunabyte , the reasone that I'm understand it's 8-10 tables per user in the MU are historical and for supporting past plug-ins and add-on

    from the point of performance and maintenance for the Enteprise environment it's can't be good .

    I'm sure that in every test 1 milion small tables or 1 big table with 1 milion records the last will win and in big time (You have indexes and other tools to optimize big tables , and what about backup\restore , dinamic vs static tsql , using transaction in vertical update , etc )

  9. lunabyte
    Member
    Posted 15 years ago #

    You're really not serving any point by trying to argue with someone who agreed with you.

    Point being, it's in the core, has been in the core since inception, and with all the MU sites out there will most likely continue to be so. Anyone can see that. Changing after this long would create a nightmare for upgrades, so it is what it is.

    Yes, the codex page says 10. It's out of date/incorrect, and it's currently 8 default tables.

    As for the WP.com db structure, in a nutshell they're running multiple databases (of the 4096 variety). Granted, they have a highly customized set-up that has evolved quite extensively based on their environment, but like any other end user of this software they're by no means required to publish their changes.

  10. andrea_r
    Moderator
    Posted 15 years ago #

    Yep, the codex page is outdated. Hard to keep up with on a voluntary basis. (you know the codex is written by the community, right?)

    "the reason that I'm understand it's 8-10 tables per user in the MU are historical and for supporting past plug-ins and add-on"

    No, it's that way because it's that way in single WP.

    Everyone in this thread who has replied has actually worked on enterprise-level setups. So we're not just guessing or going on theory. ;)

  11. hitrick
    Member
    Posted 15 years ago #

    lunabyte - Your are 100% right - didn't ment to argue :-)

    Sorry for the misunderstanding

  12. hitrick
    Member
    Posted 15 years ago #

    Thank You for the patience explaning such basics thing to a newbie like me in this world of mySQL and WP

    let me please explain to You mey background -

    I came from MsSQL2005 and oracle - the concept behind such DB design as in WPMU is very wrong from the point of view of every DataBase expert , I really can't imagine someone making such design . when I wrote that I guess it's due to historical reasone - I ment as You wrote the original version of WP .

    but still I really have a problem to imagain my DB with 5,000,000 tables and more ..... using dynamic tsql querying , maintaining few Databases and even few mySQL instances on one or more machines etc.
    that instead of heaving 10-12 tables in one DB that every table contain ~ 1 milion records +-

    maybe the solution with the 5,000,000 will work somehow . but I see this as logical as driving my car backward in reverse gear every day to work (I will still will be going from point A to point B )

    maybe You can advice on mySQL professional with some experience with WPMU that can give me (for a fee )some consultation on this issue ?

    Thank You again

  13. hitrick
    Member
    Posted 15 years ago #

    anyone ?

  14. goko
    Member
    Posted 15 years ago #

    Start modifying wordpress code. As I see it, it can be done fairy easily to just have the default ~8 tables for gazillion blogs by adding blog_id to these tables.

  15. lunabyte
    Member
    Posted 15 years ago #

    That would create a nightmare for upgrading, now wouldn't it?

    Not to mention you need more than a blog_id field in the tables.
    Some tables have one, but it's not used and is always 0.

    "maybe the solution with the 5,000,000 will work somehow"

    Works just fine for wordpress.com, and that's with millions of blogs.

    I'm not saying it's perfect, but it works. You may end up with 1000 databases spread across 4 servers, but it still works.

    So what exactly are you not getting answered here?

    As heavy of a modification as would be required, I'd recommend against that as upgrading would be a nightmare, and you'll need to keep on top of upgrades.

    While there are many routed to go from point A to point B, there doesn't have to be a "one way only or I won't go" route.

  16. gpo1
    Member
    Posted 15 years ago #

    Well from this quote " Very exciting news that Friendster, a popular social networking service with over 90 million global members, has relaunched their blog offering and selected WordPress MU. These member blogs, which are used by millions of Friendster members"
    they must be using separate DB servers etc!

  17. goko
    Member
    Posted 15 years ago #

    Of course upgrading would be a nightmare but so what, we are talking 500k+ blogs here and it's not like upgrades are once every month.

    Why not just come up with (close to) perfect solution in stead of "but it works".

    Right now, MU db design is hardly any different than regular WP cloned multiple times. It has centralized admin and that's pretty much it.

    At the end it all comes down to CPU cycles and it's up to everyone themselves to calculate price/performance.

  18. lunabyte
    Member
    Posted 15 years ago #

    While I don't disagree, the migration for existing installations to something of this nature would be a disaster waiting to happen.

    Considering the scope of changes which would be required, as in almost every query in the core that deals with the blog itself, there wouldn't be a feasible way to do so.

    If you want and MU style site in a single table fashion, there is an alternative. Granted that last I looked a few months ago it was only synced with the WP 2.3.x, but hey, who cares about all the holes and new features since 2.3 right?

    My point being with the reference, intentionally left unnamed, is that upkeep on something like that isn't easy and takes a lot of time.

    MU is what it is. You either work with the tools given, and do your thing, make your own, or skip it all together. Such is life.

    And no, he was estimating 50k blogs, which from experience is a pretty lofty goal and real world values may be less. They may not be, and thinking ahead is good, but in the end, multiple databases spread out isn't much different than sharding. The end result becomes quite similar. The main difference is that the pieces are smaller, and easier to move around.

    Take the low use blogs, group them all into a single database since they're rarely accessed. Spread the resulting savings around to the blogs that are used.

    Really, this debate is one that extends beyond MU as well.

    There's more than one way to do things. MU does it one way, which may not be everyone's preference, but it's not like it's a broken, world ending crisis either, and something completely unmanageable that has no hope for more than a handful of small sites.

    I'll again point back to wordpress.com. Millions of blogs, and they use MU. Granted they've customized it, but who hasn't? Point being, it works. It might not be someones preferred way, but that doesn't mean it's wrong and the world has to come to an end.

    To add to that, it's nothing a good DBA can't handle. Sure, it may go against their training but it is something that is quite capable and manageable. If they're any good, they'll be able to work with it.

  19. hitrick
    Member
    Posted 15 years ago #

    goko - I agree with You , the problem that most of the widgets\add-on will not work if I will change the Core that way . what will be left from the original app ...?

    If I will have to modify every hot fix and widget - I will loose most of the advantage of WP-MU . I prefer to build my own app with custom design DB and GUI
    That way It will be much wiser and easy for me (and even maybe cheaper and for sure more secure.
    due to that I'm checking me option to take WPMU as is from the DB side

    lunabyte/ lunabyte - I guess You not much objective . I know asking You about WPMU it's like asking MAC user about OS X and if it really better then Vista :-)
    just kidding Guys ... but still please try to see it from my point of view . I don't really want to manage 4 SQL servers and 1000 DataBases only because it's working ...
    My goal is to have good , stable and easy to manage and maintain Env.

    I asking this Questions here so I will have all the facts and Ideas for deciding for the best solution for me .

    Please accept my sincere apology if my post sounds like I wanna to criticize WPMU - I'm not

About this Topic