bigdoug - from what I can see/know of mu, files in blogs.dir aren't directly accessed through URLs, they are always controlled via an existing RewriteRule which is created in the .htaccess file during installation.
You'd want to change this line:
RewriteRule ^(.*/)?files/(.*) wp-content/blogs.php?file=$2 [L]
and probably add in some RewriteCond (conditions) to check the host name (assuming you're using subdomain, rather than path hosting), then when certain conditions (hostnames) are matched, you use a different RewriteRule something like:
RewriteRule ^(.*/)?files/(.*) http://lowly-image-server.com/wp-content/blogs.php?file=$2 [L]
Of course you'd also need to make sure that everything was replicated over to this server as well so that images users upload are immediately available.
To make this really powerful, you might want to look at the "RewriteMap" directive for Apache, and consider setting up a mapping of subdomain -> blog_id so that you can use that to find the right directory on your media server, then you could avoid DB access altogether on it perhaps.
HTH
Beau