Portal Forums Member List Calendar Active Threads Staff UBBWiki
Previous Thread
Next Thread
Print Thread
Rate Thread
Block robots from certain parts of the UBB w/ CP interface.
#54786 11/15/2002 5:32 PM
Joined: Dec 2001
Posts: 699
Bath || Reading, England
Dave2
Offline
Member
Dave2
Member
Offline
Joined: Dec 2001
Posts: 699
Bath || Reading, England
the title says it all wink
if you could specify UAs [as regexps] or IPs to block from any area of the ubb, that would be useful to stop search engines indexing uter crap.
Sponsored Links
Re: Block robots from certain parts of the UBB w/ CP interface.
#54787 11/15/2002 7:01 PM
Joined: Mar 2000
Posts: 21,079
Likes: 3
Texas
AllenAyres Offline
I type Like navaho
AllenAyres
I type Like navaho
Offline
Joined: Mar 2000
Posts: 21,079
Likes: 3
Texas
You can insert robot.txt in the headerinsert for style templates associated with the forums you want blocked smile

- Allen wavey
- What Drives You?
Re: Block robots from certain parts of the UBB w/ CP interface.
#54788 11/16/2002 9:43 AM
Joined: Dec 2001
Posts: 699
Bath || Reading, England
Dave2
Offline
Member
Dave2
Member
Offline
Joined: Dec 2001
Posts: 699
Bath || Reading, England
Well, it's not in particular forums, just things like calendar, memberlist etc.
I don't think I can block them via robots.txt (although I'm not sure) and I know I can't via .htaccess (which I currently use to block spambots).
Re: Block robots from certain parts of the UBB w/ CP interface.
#54789 11/16/2002 10:07 AM
Joined: Dec 2000
Posts: 730
Germany
joking-down Offline
deutscher moderator / v5 specialist
joking-down
deutscher moderator / v5 specialist
Offline
Joined: Dec 2000
Posts: 730
Germany
You can use this code to stop spiders abd robots:
Code
if (($ENV{HTTP_USER_AGENT} =~ /Spider/i) || ($ENV{HTTP_USER_AGENT} =~ /bot/i)) {
    print qq(
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML>
<head>
<meta NAME="robots" content="noindex, nofollow, noarchive">
</head>
<body>Do not index this page!!</body>
</html>);
 
    exit;
}
A list of Spiders and Robots you can find here
Re: Block robots from certain parts of the UBB w/ CP interface.
#54790 11/17/2002 5:14 AM
Joined: Mar 2001
Posts: 172
nn
Gladius Offline
Member
Gladius
Member
Offline
Joined: Mar 2001
Posts: 172
nn
You actually have bots indexing your UBB? That's a first. Mine's been up for a year (since it's all cgi) and it's never been indexed by any of them. And I'm not doing anything to prevent it.
Sponsored Links
Re: Block robots from certain parts of the UBB w/ CP interface.
#54791 11/17/2002 10:52 AM
Joined: Jun 2001
Posts: 729
CT
1QuickSI Offline
Coder
1QuickSI
Coder
Offline
Joined: Jun 2001
Posts: 729
CT
We are visited daily smile I wonder if it might be the way you have your meta tag statements setup?

Robots/Spiders visitors

Hits Bandwidth Last visit

Road Runner: The ImageScape Robot 54 172.93 KB 17 Nov 2002 - 04:02

Pioneer 44 106.59 KB 17 Nov 2002 - 01:21

Googlebot (Google) 37 362.41 KB 17 Nov 2002 - 02:48

Inktomi Slurp 9 145.29 KB 17 Nov 2002 - 04:06

MSIECrawler 5 3.84 KB 17 Nov 2002 - 04:27

Scooter (AltaVista) 2 16.96 KB 17 Nov 2002 -00:25
Re: Block robots from certain parts of the UBB w/ CP interface.
#54792 11/17/2002 2:53 PM
Joined: Dec 2001
Posts: 699
Bath || Reading, England
Dave2
Offline
Member
Dave2
Member
Offline
Joined: Dec 2001
Posts: 699
Bath || Reading, England
Yeah - spidered, but Google gives no PageRank to .cgi pages so I have a PR of 0 for ultimatebb.cgi smirk (4 for .php). But it seems to like the memberlist tipsy .
Re: Block robots from certain parts of the UBB w/ CP interface.
#54793 11/18/2002 12:08 AM
Joined: Mar 2000
Posts: 21,079
Likes: 3
Texas
AllenAyres Offline
I type Like navaho
AllenAyres
I type Like navaho
Offline
Joined: Mar 2000
Posts: 21,079
Likes: 3
Texas
When I search for ubb's, it almost always links to the FAQ page too smile

- Allen wavey
- What Drives You?
Re: Block robots from certain parts of the UBB w/ CP interface.
#54794 11/18/2002 1:53 AM
Joined: Jun 2001
Posts: 729
CT
1QuickSI Offline
Coder
1QuickSI
Coder
Offline
Joined: Jun 2001
Posts: 729
CT
Been using different search criteria to learn about my system, where it is linked from and what people are saying about it. Seems individual boards are linked to from Google.

Hostboard
... Users Online: THIS FORUM ONLY 1slowZ, 4PASNU and 2 guests. Help support
Hostboard by visiting our advertisers. ... Why PepBoys sucks ass!!! ...
www.hostboard.com/cgi-bin/ultimatebb.cgi?ubb=forum&f=346 - 72k - Cached - Similar pages

Hostboard
... Users Online: THIS FORUM ONLY 1 guest. Help support Hostboard by visiting our
advertisers. Topic, Topic Starter, Replies, Views, Last Post. ... Dragon Ball SUCKS! ...
www.hostboard.com/cgi-bin/ultimatebb.cgi?ubb=forum&f=836 - 17k - Cached - Similar pages

Hostboard
... Help support Hostboard by visiting our advertisers DRINK UP. Topic, Topic Starter,
Replies, Views, Last Post. this place sucks without ski, Firebird98 October 03 ...
www.hostboard.com/cgi-bin/ultimatebb.cgi?ubb=forum&f=419 - 26k - Cached - Similar pages

Hostboard
... Help support Hostboard by visiting our advertisers ... Johnny Storm sucks, why would
you want to write about him?," or "Hey, what the hell's wrong with you? ...
www.hostboard.com/cgi-bin/ultimatebb.cgi?ubb=forum&f=157 - 18k - Cached - Similar pages
Re: Block robots from certain parts of the UBB w/ CP interface.
#54795 11/19/2002 12:00 PM
Joined: Mar 2001
Posts: 172
nn
Gladius Offline
Member
Gladius
Member
Offline
Joined: Mar 2001
Posts: 172
nn
Wait... I don't know if I even have any meta tags set on it. Duh. Is there an option to do it somewhere?
Sponsored Links
Re: Block robots from certain parts of the UBB w/ CP interface.
#54796 11/19/2002 12:12 PM
Joined: Jun 2001
Posts: 729
CT
1QuickSI Offline
Coder
1QuickSI
Coder
Offline
Joined: Jun 2001
Posts: 729
CT
Meta Tags is not an option within UBB but directives you place within your main web page such as index.html Feel free to visit my home page, view source and look at the way I have the meta tags setup.
Re: Block robots from certain parts of the UBB w/ CP interface.
#54797 11/19/2002 2:30 PM
Joined: Dec 2001
Posts: 699
Bath || Reading, England
Dave2
Offline
Member
Dave2
Member
Offline
Joined: Dec 2001
Posts: 699
Bath || Reading, England
Uhh... header insert? tipsy
Re: Block robots from certain parts of the UBB w/ CP interface.
#54798 11/19/2002 4:27 PM
Joined: Mar 2001
Posts: 172
nn
Gladius Offline
Member
Gladius
Member
Offline
Joined: Mar 2001
Posts: 172
nn
Geez, talk about missing the obvious... thanks.

Link Copied to Clipboard
Donate Today!
Donate via PayPal

Donate to UBBDev today to help aid in Operational, Server and Script Maintenance, and Development costs.

Please also see our parent organization VNC Web Services if you're in the need of a new UBB.threads Install or Upgrade, Site/Server Migrations, or Security and Coding Services.
Recommended Hosts
We have personally worked with and recommend the following Web Hosts:
• Stable Host
• bluehost
• InterServer
Visit us on Facebook
Member Spotlight
badfrog
badfrog
somewhere on the coast of Maine
Posts: 94
Joined: March 2007
Forum Statistics
Forums63
Topics37,575
Posts293,932
Members13,824
Most Online6,139
Sep 21st, 2024
Today's Statistics
Currently Online
Topics Created
Posts Made
Users Online
Birthdays
Top Posters
AllenAyres 21,079
JoshPet 10,369
LK 7,394
Lord Dexter 6,708
Gizmo 5,835
Greg Hard 4,625
Top Posters(30 Days)
Top Likes Received
isaac 82
Gizmo 20
Brett 7
WebGuy 2
Morgan 2
Top Likes Received (30 Days)
None yet
Forum Rules · Mark All Read Contact Us · Forum Help · UBB.Dev
The UBB.Developers Network (UBB.Dev/Threads.Dev) is ©2000-2025 VNC Web Services

 
Powered by UBB.threads™ PHP Forum Software 8.0.1
(Snapshot build 20240918)