It doesn't quite disable the visual editor though, although WordPress does degrade nicely.
If you have "Use visual editor" ticked and you remove the tinymce folder, then you don't get an editor at all. You simply get two text boxes (no javascript errors though). However, the checkbox disappears from your User Profile page. So you can't then disable the visual editor.
Whereas, if you disable the visual editor first, then remove the tinymce folder, you still get the old fashioned javascript editor.
Thus, it's not quite a perfect solution to simply remove the folder. It would be better to be able to uncheck the option automatically for all users.