Previous Thread
Next Thread
Print Thread
Rate Thread
#183023 01/23/2004 1:06 AM
Joined: Jun 2001
Posts: 2,849
Spotlight Winner
Spotlight Winner
Offline
Joined: Jun 2001
Posts: 2,849
Maybe warning is a bit strong, let's say a bit of advise to people that have added the spider hack to their boards.

In the text file with instructions for implementing the spider bait hack you are given a warning about the possible load that it can place on your server during the time your boards are being crawled. This is a serious issue for large boards on shared servers. Another warning warns about impact on your boards if you don't have the accelerator enabled.

Please listen to these warnings. I've helped more than one site troubleshoot problems that ended up being caused by using this hack without enabling the accelerator and being on a shared server. UBB already has a reputation for being a resource hog and a lot of hosting companies no longer allow it as part of their less expensive plans.

We all know that UBB Classic is a great product. The spider bait hack is a must have for anyone that wants their forum content to be searchable. Just remember that it's easier to set your site up properly than it is to have to troubleshoot problems later or find a new host because they booted you for using more than your share of the server resources.

I've also seen cases where the server crashes with a ton of perl processes.

Sponsored Links
#183024 01/23/2004 11:09 AM
Joined: Feb 2001
Posts: 817
Moderator / Kingpin
Moderator / Kingpin
Joined: Feb 2001
Posts: 817
.classic isn't much as a resource hog as it used to be with earlier 6.x versions but I do agree, you must have the accelerator enabled and you also need to consider the potential problems when enabling the spider hack on an over-shared shared server.

Good post. I'm featuring this topic.

#183025 01/23/2004 12:38 PM
Joined: Jan 2000
Posts: 5,073
Admin Emeritus
Admin Emeritus
Joined: Jan 2000
Posts: 5,073
Any script has the potential to be a resource hog when malicious spiders come crawling along.

Google, Inktomi, and others intentionally limit the crawling rate in order to prevent hitting the server too hard, even for static pages. Not all spiders are as nice. A single malicious spider can easily make a hundred requests a minute, which can promptly bring any server to its knees.


UBB.classic: Love it or hate it, it was mine.
#183026 01/23/2004 8:10 PM
Joined: Jan 2000
Posts: 5,833
Likes: 20
UBBDev / UBBWiki Owner
Time Lord
UBBDev / UBBWiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,833
Likes: 20
Agreed, I've seen some serious impact on some boards running spider hack when spiders come along, a lot of hosts don't like it when that happenes now adays due to spiderable url's in the beta...

Perhaps look at a new host before you enable spidering... HostNuke hosts both of my boards and Al's, and we both have spidering enabled... Works like a dream...


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
#183027 01/27/2004 3:11 AM
Joined: Jun 2003
Posts: 60
Member
Member
Offline
Joined: Jun 2003
Posts: 60
What do you think of LunarPages in this regards?

I've heard (and seen a little) that their servers do bow to spiders at times.


REaMERE
Sponsored Links
#183028 01/27/2004 6:32 PM
Joined: Jan 2000
Posts: 5,833
Likes: 20
UBBDev / UBBWiki Owner
Time Lord
UBBDev / UBBWiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,833
Likes: 20
I never really liked LunarPages... The only post I like anymore is HostNuke; their systems support my board when 3 other providers turned me away...


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
#183029 01/28/2004 4:20 PM
Joined: Jun 2003
Posts: 60
Member
Member
Offline
Joined: Jun 2003
Posts: 60
Hmmm. good to know, thanks.
I use LunarPages (obviously) for all of my clients and I really have no complaints. Of course, I have been on worse providers. Oh...the stories I could tell!

But DrkKnight and I just moved our boards over to Lunar and are doing well. I do about 3000 unique visits per month but he does well over 20,000 (crazy traffic), he's using Ubb.threads now, and I'm using classic, again no complaints.

But we do not have the spider hack installed. I was wondering if anyone had any experience with UBB and the Spider Hack on Lunar Pages as a host.

Thanks for the feedback


REaMERE
#183030 01/28/2004 6:49 PM
Joined: Jan 2000
Posts: 5,833
Likes: 20
UBBDev / UBBWiki Owner
Time Lord
UBBDev / UBBWiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,833
Likes: 20
We were using a spider mod when on there, see above, they didn't like it lol...


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
#183031 01/28/2004 8:53 PM
Joined: Jun 2003
Posts: 60
Member
Member
Offline
Joined: Jun 2003
Posts: 60
Ahhh, I didn't realize you were using them.
I suppose that either the server crashed or they told you that they were dropping the account?
Either way, thanks, good to know, because we were both thinking of using the hack.


REaMERE
#183032 01/28/2004 8:57 PM
Joined: Jan 2003
Posts: 3,456
Likes: 2
Master Hacker
Master Hacker
Offline
Joined: Jan 2003
Posts: 3,456
Likes: 2
if you wait a week or two, the next version will have it, so no need to alter your boards.


As for drknight, I believe that threads 6.5 will also have it

Sponsored Links
#183033 01/28/2004 9:28 PM
Joined: Jan 2000
Posts: 5,833
Likes: 20
UBBDev / UBBWiki Owner
Time Lord
UBBDev / UBBWiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,833
Likes: 20
Correct, as AL sais, no need to use a hack when it's standard in a few weeks in 6.7.

And I was running my board there and they had told me that I was recieving too much traffic and creating too much of a load with their systems.

So I simply moved to HostNuke and never looked back...


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
#183034 01/28/2004 10:54 PM
Joined: Jan 2003
Posts: 3,456
Likes: 2
Master Hacker
Master Hacker
Offline
Joined: Jan 2003
Posts: 3,456
Likes: 2
OK, I'm now taking bets on how many times Gizzy is gonna mention Hostnuke in this thread.

#183035 01/28/2004 11:02 PM
Joined: Jan 2000
Posts: 5,833
Likes: 20
UBBDev / UBBWiki Owner
Time Lord
UBBDev / UBBWiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,833
Likes: 20
you counting my sig in this count? lol...


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
#183036 01/29/2004 12:57 PM
Joined: Jun 2003
Posts: 60
Member
Member
Offline
Joined: Jun 2003
Posts: 60
Heheh tipsy
Now you say it's gonna be standard, but will it be the same hack. Meaning, will as many pages be spidered or will it be kind of like a Spider Hack Light?
My obvious concern is that (when) we upgrade to 6.7 Classic and 6.5 threads respectively will this cause major problems with our current host.

You know how much effort it is moving boards around. (you know, to places like HOSTNUKE wink )

Throw me your opinions and advice please!


REaMERE
#183037 01/29/2004 5:16 PM
Joined: Jan 2003
Posts: 3,456
Likes: 2
Master Hacker
Master Hacker
Offline
Joined: Jan 2003
Posts: 3,456
Likes: 2
the one is the base code is written by Charles Capps, who wrote the mod. And now that it's built in, it's much better, with .html extensions for spiders the require.

#183038 01/29/2004 5:54 PM
Joined: Jun 2003
Posts: 60
Member
Member
Offline
Joined: Jun 2003
Posts: 60
Nice. Thanks guys for all of the quick answers.
So basically we should:
1.) Make sure the PHP Accelerator is on and functional
2.) Monitor the Apache Servers during peak hours (among other times) to make sure they aren't getting overloaded.
3.) Consume several Gin & Tonics

Am I missing anything??


REaMERE
#183039 01/29/2004 6:11 PM
Joined: Jan 2000
Posts: 5,833
Likes: 20
UBBDev / UBBWiki Owner
Time Lord
UBBDev / UBBWiki Owner
Time Lord
Joined: Jan 2000
Posts: 5,833
Likes: 20
Don't forget the vodka; other than that, it's lookin good.


UBB.Dev - Putting Dev into UBB.threads
Company: VNC Web Services - UBB.threads Scripts and Scripting, Install and Upgrade Services, Site and Server Maintenance.
Forums: A Gardeners Forum, Scouters World, and UGN Security
UBB.Threads: My UBB Themes, My UBB Scripts
#183040 02/01/2004 9:18 PM
Joined: Jan 2003
Posts: 3,456
Likes: 2
Master Hacker
Master Hacker
Offline
Joined: Jan 2003
Posts: 3,456
Likes: 2
I found this website that will have an htaccess that prevents most malicious spiders.

Code
# source: http://www.webmasterworld.com/forum13/687-9-15.htm
# some of these are commented out, they are possibly legitate download agents
# if you don't want anyone downloading your sites, uncomment them

RewriteEngine On
RewriteCond %{HTTP_REFERER} q=Guestbook [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot mailto:[email protected] [OR]
RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^GornKer [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} Indy Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^Irvine [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC Web Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline Navigator [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} dloader(NaverRobot) [OR]
#RewriteCond %{HTTP_USER_AGENT} ^puf [NC,OR]
#RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^SearchExpress [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^Siphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web Image Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebBandit [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebGo IS [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website eXtractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website Quester [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus [OR]
RewriteCond %{HTTP_USER_AGENT} ^ZyBorg

RewriteRule ^.* - [F,L]

#183041 02/02/2004 7:31 PM
Joined: Jun 2003
Posts: 60
Member
Member
Offline
Joined: Jun 2003
Posts: 60
Nice find. I'll be adding that information to mine.
Excellent site also, lots of good info.

Question: Is there any way to block spiders from within any of the .cgi or .pl files?

Thanks in advance


REaMERE
#183042 02/02/2004 9:50 PM
Joined: Jan 2003
Posts: 3,456
Likes: 2
Master Hacker
Master Hacker
Offline
Joined: Jan 2003
Posts: 3,456
Likes: 2
just disallow access to the cgi-bin and the noncgi/templates folder

#183043 02/03/2004 12:05 PM
Joined: Jun 2003
Posts: 60
Member
Member
Offline
Joined: Jun 2003
Posts: 60
Thanks again Al. Cant vote for ya again, but you have my thanks!


REaMERE

Link Copied to Clipboard
Donate Today!
Donate via PayPal

Donate to UBBDev today to help aid in Operational, Server and Script Maintenance, and Development costs.

Please also see our parent organization VNC Web Services if you're in the need of a new UBB.threads Install or Upgrade, Site/Server Migrations, or Security and Coding Services.
Recommended Hosts
We have personally worked with and recommend the following Web Hosts:
Stable Host
bluehost
InterServer
Visit us on Facebook
Member Spotlight
isaac
isaac
California
Posts: 1,157
Joined: July 2001
Forum Statistics
Forums63
Topics37,573
Posts293,925
Members13,849
Most Online5,166
Sep 15th, 2019
Today's Statistics
Currently Online
Topics Created
Posts Made
Users Online
Birthdays
Top Posters
AllenAyres 21,079
JoshPet 10,369
LK 7,394
Lord Dexter 6,708
Gizmo 5,833
Greg Hard 4,625
Top Posters(30 Days)
Top Likes Received
isaac 82
Gizmo 20
Brett 7
WebGuy 2
Top Likes Received (30 Days)
None yet
The UBB.Developers Network (UBB.Dev/Threads.Dev) is ©2000-2024 VNC Web Services

 
Powered by UBB.threads™ PHP Forum Software 8.0.0
(Preview build 20221218)