WordPress MU

Alright -- this is a big one, and really driving me crazy.

My ENTIRE WPMU server is crashing, and the error logs show nothing -- no errors, no system errors. Just a locked up system that has to be hard reset.

I know I'm not offering much, but I dont have much to go on, either.

We moved the entire WPMU instance onto a new identical box, and had the same issues.

We are running on a Max OSX server with apache, but WPMU has otherwise been fine for almost a year.

I'm desperate and would love some suggestions, please help.

Is anyone doing a top on the server?
Does it happen at a predictable time? Like for instance, will it run fine for a day, then slow down and eventually crash?
Have you tried eliminating plugins?
Did you add anythign new lately?

Oddball out there situation: in some cases, users put their own rss feed link in the RSS widget. This'll cause issues like this.

Andrea,

re: top - We cant run Top at the time of the crash -- the server becomes totally unresponsive and we have to reset the box.

re: timing - The timing is unpredictable. It had been doing this every couple/few weeks, and it wasn't a big deal, but now it's doing it every 1-2 days. Sometimes its less than 24 hours, sometimes its closer to 48.

I cant detect slowness before it crashes, but in one rare occurrence, I did see database connection errors.. about 30minutes/1hr later, the entire server was frozen. Could this mean the MySQL server is croaking first?

re: plugins - I have begin removing major plugins (like caching), and those which I dont need. I also went and upgraded every administration plugin I used. It hasnt helped so far.

re: anything new - The only new things that we have done are standard updates to WPMU, but there are chances I might be missing somethnig.

re: rss - any way to check for this? does this just create a major internal redirect problem? I dont have widgets enabled on the vast majority of my sites. There are only a few 'true' blogs that can use widgets like this.

Yeah, I know you can't top when it's frozen. :) I'm suggesting to just monitor it when you think it might be going down.

" in one rare occurrence, I did see database connection errors.. about 30minutes/1hr later, the entire server was frozen. Could this mean the MySQL server is croaking first?"

Yep. What do you use to stop spam? How much is it catching? I've got a vps I'm looking at that blips predictably, and it's basically spam hits on the server (mail and comments).

re: rss - you'd have to go look at each blog and see. If it's only a few, that's good.

But I'm gonna say probably the most likely reason is spammers and depending on what you're using, you want to stop them before MU processes their comment.

The spammers issue is a tough one because we only allow comments from those who authenticate against our LDAP server (we use an LDAP authentication plugin, and only allow registered users to leave comments). We also disable trackbacks/pings. So, we don't actually have a very robust use of comments on our sites.

I did find one blog who had enabled the RSS widget, and while they didn't have any data filled in for it's use, it was throwing me an error. I removed the widget from their active list... not sure why they had it up.

It may still be worth it to track down how many spammers are *attempting* to leave comments though. Even if they can;t get through, if each attempt hits the db, a hundred failing at once is still a hundred hits on the db at once. Which may bring it down.

(I'm pulling that number out of thin air, but you get the drift.)

Otherwise, you'll have to disable as many plugins and mu-plugins as you can and wait while watching the server.

It seems odd for it to freeze up solid like that though. Usually when unix servers get totally maxed out you can usually get to a command prompt even if its like treacle.

There is nothing out of the ordinary in the apache access/error logs or the php error log?

Have you applied any system patches recently?

andrea: I will check the access logs to see if we have a high percentage of login attempts, but from what we saw in the logs previously, the activity seemed low key, with few hits to anything outside of normal content.

You're right about the plugins, though. It's a process I've already started. The next batch of removals will start making my life painful, though, as I rely on them.

Here's a list of the plugins I use:

AHP Sitewide Recent Posts for WordPress MU 0.6.1
cets_blog_defaults.php 1.2.4
cets_simple_dashboard.php 1.3.2
domain_mapping.php 0.4.3 (donncha)
More Privacy Options 2.9.1
Toggle Admin Menus 2.5.2
LDAP Authentication Plug-in 2.8.2
Listem .1
WPMU Plugin Manager 1.4.1
WordPress Mu Google Analytics (by Rafik)
Viper's Video Quicktags 6.2.6

steveatty: i'll have to check with the server admin about system patches. the access logs and errors logs (of the server, php) have been clean. An occasional 404, but nothing serious. no real errors or concerns from the system.

The system was up to date with updates as of a week ago.

Today we experienced another crash (I had disabled AHP Sitewide Recent posts, so that isnt causing this).

Right now I have the plugin 'Listem' disabled, and we've enabled logging for the MySQL server, a long with minute dumps of TOP.

Well.. I hope I am not posting this prematurely, but the crashes have stopped for almost 3 weeks now.

The only thing I did this time was to remove an extra file, a blogs.php that was in the /wp-admin/ folder. I believe it was left over from an old update.

...I am sure that as I type this, the server is going to crash :)

I don't understand how that could have caused these crashes, however.

Well there isn't a blogs.php file in my includes folder so that could well be it.

wp-admin/blogs.php was in 2.7.1, but appears to have been removed in 2.8, there's a link to that page from the user profile in 2.7, fwiw.

Shows how fuzzy my brain was last night - I wasn't even looking in the right folder!

Keep it up Steve, and I'm gonna start worrying about you too. :)

I've been worrying about myself for a long time Andrea ;-)

..well we were crash-free for 2 months. I hadn't TOUCHED the install, and the crashes started up back up again.

At this point, my theory is some kind of attack, perhaps an exploit.

We enabled a more rigorous firewall (didnt work, so it's coming in on standard ports).

We had TOP dump all the processes every minute to see how performance was leading up to the crash, and within 30 seconds of a crash happening, TOP does not show anything out of the normal (normal loads).

It's like a button is getting pressed, with no build-up.