Nonexistent blogs redirect to wp-signup.php (i.e. if "blog10" doesn't exist, blog10.example.com redirects to a signup page encouraging the user to create a blog with that name).
I'd prefer it redirects to the homepage or the theme's 404, but not sure where to start. It's not in .htaccess, and I can't find anything in wp-signup.php that makes sense to edit. Thanks in advance.
It's been a while, but I *think* you may be able to define this in the wp-config file.
Got it!
So, for anybody who finds this later, here's how I did it: You have to edit wpmu-settings.php (in your root directory) around line 146. Find the "if" statement for when !is_object( $current_blog )
and just comment out everything inside the statement and redirect! I redirected to my main blog's 404 page.
It might be easier for you to use the correct method, as Andrea suggested, which would be to define NOBLOGREDIRECT in your wp-config.php file.
Doing it the way you describe above will break on your next upgrade unless you remember to do it again - defining it in wp-config won't. ;)