Here's my 2 cents
666 did not work for me 777 did.
There are two reasons why we hit this error. One is the permissions, and the other is the ownership. While uploading files, images etc., wordpress tries creating directories which are not owned by the owner for the rest of the directories. For the sake of this discussion let me call him 'own1' and the group "grp1". In my case wordrpess tries creating directories owned by "apache:apache" which, is different from the ownership of the rest of my wpmu files ie own1:grp1
The error I got is something like ,
Unable to create directory - .../wp-content/blogs.dir/4/files/ Is its parent directory writable by the server?
When I sshed, there was no blogs.dir folder created. I changed the permission for wp-content to 777 and tried uploading an image. This time i got the same error ie
Unable to create directory - .../wp-content/blogs.dir/4/files/ Is its parent directory writable by the server?
but when i sshed i saw that blogs.dir was created however owned by 'apache:apache' which is why the folder 4 was not getting created.
I did a 'chown usr1:grp1 blogs.dir' to change the onwership and tried uplaoding again. I got the same error ie
"Unable to create directory - /wp-content/blogs.dir/4/files/ Is its parent directory writable by the server?",
but this time the folder 4 was created but owned, again, by apache:apache. Did a 'chown usr1:grp1 4' again to change it and tried uploading an image.
Now it gave me a different error. it was ...
The uploaded file could not be moved to the upload folder.
In the terminal, I saw that "files" folder was created but again owned by apache:apache. I changed that and uploaded an image and this time it worked.
Thanks guys on top, and hope this helps someone later.