UBB.Developers Forums UBB.classic V6.4 - 6.7 Modifications V6 Mod Ideas Crawler Page?

How about a hack or bit of code that will generate a crawler page for search engine submission.

Not just for Internet wide engines but site specific ones too. I just need a page with links to the topics and the multiple pages within them.

I'll have to write my own PHP based one at some point as I'd like to integrate the forum contents into my site wide search engine.

Just an idea.

Oh well, if nobody fixes a problem for you...

http://www.liversidge.net/paul/crawler.zip

Two PHP scripts that generates set of crawler pages that allow you to submit the contents of your UBB to search engines.

As long as the search engines obey the robots meta tag, most do, you can use these pages. The ReadMe file tells you what to change for your board and also how to make sure your site doesn't get hammered when the spiders come-a-crawling.

If you have any problems, let me know [email protected].

The scripts work, whether this encourages the spiders to crawl your forums pages is unproven. Please let me know.

thanks

I've just finished setting up my site's search engine and I've been using these PHP scripts to test if my theory works.

It does.

I've setup Fluid Dynamics Search Engine on my test site and pointed it at these scripts. It does crawl all the topics in the forums and doesn't get bogged down flicking through all the reply and get profile links that pepper the forum pages. Therefore, it should work perfectly with the Net's search engines too. I'll give a few of them an opportunity to chomp through my forums in the next few days and see what they manage to find.

BTW FDSE is a wonderful bit of software and I heartily recommend it for building search pages. It's also free if you're limiting the scope to a single site otherwise it's US$40. A small price to pay.