Previous Thread
Next Thread
Print Thread
Rate Thread
#191859 06/28/2002 6:43 AM
Joined: Oct 2001
Posts: 114
Content Kidman
Content Kidman
Offline
Joined: Oct 2001
Posts: 114
What you need is a crawler page or set of crawler pages.

One page that lists the forums just as plain links, i.e. forums.htm

The links on this page will link to other pages called f1.htm, f2.htm, etc. One for each forum.

Each f??.htm page will then have a list of links on them to the actual topics on the site.

You can then submit the one forums.htm page to the search engines and let it rip.

Most search engines are experimenting with dynamically generated pages or they only spider a few pages. They also have a tendency to get lost inside a forum with the many, many links.

It's easier to setup a crawler page and use the robots meta tag, to tell the crawler to follow the links on the crawler pages but not to index them. And then on the forums, add a robots tag to the forum header that says (index, no follow). This should mean your content is indexed, the crawler pages are not and the spider doesn't hammer your site following all the reply and edit links on all the forum pages.

This is a theory at the moment but one I'll be trying shortly. I want all my pages indexed internally and externally and this seems the easiest way around it. However, I can't write Perl so I'm going to use PHP to generate the pages from the UBB files.

If I get this working I'll share it here.

Sponsored Links
Entire Thread
Subject Posted By Posted
Search Engine Listings stupidboy.com 06/27/2002 8:30 PM
Re: Search Engine Listings AllenAyres 06/27/2002 8:43 PM
Re: Search Engine Listings stupidboy.com 06/27/2002 9:19 PM
Re: Search Engine Listings AllenAyres 06/27/2002 9:49 PM
Re: Search Engine Listings AllenAyres 06/27/2002 10:12 PM
Re: Search Engine Listings Paulus Magnus 06/28/2002 1:43 PM
Re: Search Engine Listings Paulus Magnus 06/28/2002 11:15 PM

Link Copied to Clipboard
Donate Today!
Donate via PayPal

Donate to UBBDev today to help aid in Operational, Server and Script Maintenance, and Development costs.

Please also see our parent organization VNC Web Services if you're in the need of a new UBB.threads Install or Upgrade, Site/Server Migrations, or Security and Coding Services.
Recommended Hosts
We have personally worked with and recommend the following Web Hosts:
Shock Hosting
Stable Host
bluehost
InterServer
Visit us on Facebook
Member Spotlight
Nettomo
Nettomo
Germany, Bremen
Posts: 417
Joined: November 2001
Forum Statistics
Forums63
Topics37,583
Posts293,955
Members13,825
Most Online151,614
Nov 14th, 2025
Today's Statistics
Currently Online 29723
Topics Created 0
Posts Made 0
Users Online 1
Birthdays 3
Top Posters
AllenAyres 21,080
JoshPet 10,369
LK 7,394
Lord Dexter 6,708
Gizmo 5,834
Greg Hard 4,625
Top Posters(30 Days)
Top Likes Received
isaac 82
Gizmo 20
Brett 7
WebGuy 2
Morgan 2
Top Likes Received (30 Days)
None yet
The UBB.Developers Network (UBB.Dev/Threads.Dev) is ©2000-2026 VNC Web Services

 
Powered by UBB.threads™ PHP Forum Software 8.1.0
(Snapshot build 20260108)