No Guests?

[JonnyMac]

Hey, gang, we're getting pounded by web spiders, especially from Yahoo; today as many as 19 "guests" from the same Yahoo location in New York.  I'm getting close to closing our forums to guests.  Before I do, I'd like to hear your thoughts.  What I don't want is server bandwidth being consumed by Yahoo when it's clear they aren't buying any Prop-1s from us!  ;D

[gadget-evilusions]

I see no reason why not, if someone wants to access the super knowledge in these forums, let them register.

[menehune]

Another forum I am a member in had to disable features due to a flood of spiders.  I say block'em with a robot.txt file.  I block all crawlers on my websites due to the constant visitations.

[JonnyMac]

Okay, now you've lost ME...  I can block IPs with the forum admin features, but I have to add them one at a time -- a very tedious task.  Am I missing something?  Any suggestions are welcomed.

For the time-being I have disabled all guest permissions.  Once I know how to block the spiders without blocking valid interested parties, I will restore guest permissions.

Oh, look, this was my 666th post -- the devil made me do it!

[ufo8mycow]

Jon you can block a group of IP addresses by useing * say if you want to ban all ips that start with 192 you can enter 192.**.***.**  and you can refine it down as far as you need to but far as I know the google and yahoo guests are a good thing. they are bots that automaticly brouse the forum for there search engines.

[ufo8mycow]

by the way if you do ban a group of Ips be careful and dont go to broad with it. I was trying to ban a dialiup user on my forum and ended up also baning one of the mods.

[JonnyMac]

If it was one or two Yahoo IPs on our site at a given time I'd be okay with that, but 10, 12, more -- we don't need that much "help" from Yahoo, thank you.  I've banned *.search.yahoo.net to see if that stems the tide.

[menehune]

If you have root access to the server, you can add a "robots.txt" file to stop spiders from crawling the site.  Most spiders are nice and identify themselves and ignore the webserver directories listed in the robots.txt file.

You can read up on blocking robots (search engine spiders) here:
http://www.robotstxt.org/wc/robots.html

If you don't have root access, you can add a META tag (also documented at the quoted site) at the top of each webpage to exclude robots from scanning the webpage.