The MU forums have moved to WordPress.org

Need info @ downloader robots protection (6 posts)

  1. xrun
    Member
    Posted 15 years ago #

    Hi.
    The site I've set up is a video/media site, and there are several GB of media files there (audio/video).
    I'm getting rather high traffic on some days, like today before noon CET I'm seeing traffic as high as 4 normal whole days.
    Hotlink protection is on, but I'm guessing this guy today is using some crawler robot, testing every link for media files, and downloads it when it's found.
    Additionally, he shows up with 2 user agents, one for Chrome, and one for IE.
    Is there a way to block such downloader crawlers/robots?

  2. VentureMaker
    Member
    Posted 15 years ago #

    1. Make downloads available to registered users only
    2. Block guy's IP :)

  3. andrea_r
    Moderator
    Posted 15 years ago #

    Block the user agents.
    Block the specific crawlers in the htaccess.

    Google's a big help here, and it's just general website info, not specific to MU or even WP.

  4. xrun
    Member
    Posted 15 years ago #

    Yea I understand that.
    Had a chat with the server providers, and blocked the IP of the crawler in question about 3 hours ago. It's been quiet since, so we'll use this method next time as well I guess. Did manage to ramp up about 90GB of traffic before it was stopped though.
    I know it's unusually high, the site contains a number of popular videofiles.

  5. tdjcbe
    Member
    Posted 15 years ago #

    Have you complained to the owner of the IP address or even their host? Most folks would consider 90 gigs of traffic within a short period of time an DDoS attack and look into legal recourse.

  6. xrun
    Member
    Posted 15 years ago #

    I agree.
    However, this is in Norway, same internet but different jurisdiction. The ISP for the offending IP is not required to disclose anything, so as long as it can be blocked I won't waste time trying to get them to recognize the problem in the way it is f.example in the US. Telenor is not widely known as the most customer-friendly company out there. There are means and ways to deal with situations like this though.

    We're talking about a runaway fan of one of the interactive TV hostesses depicted on the site, his IP is blocked and he'll have to wait for his renewal, often getting the same IP which will still be blocked, or he'll have to take his laptop out for a spin and piggyback on someone elses wifi which will be much slower than his own fiber line. If he's an advanced user he may use proxies at other IP's. I concider the problem solved for now, allthough it's not an ideal solution.

    I'm guessing it's possible to make a script to block mass downloaders IP's automatically, locking down the actual media folderstructure. Making it inaccessible while still being able to see the site in general. I wonder if that would be somewhat frustrating, for a runaway fan that is. ;)

About this Topic