Mod Name / Version: Spider Modification 1.1

Description: Trick search engines spiders into reading your forums by telling them pages are plain HTML.
Right now it will only work on home, forums, threads

Working Under: UBB.Threads 6.3

Mod Status: Beta

Any pre-requisites: Running Apache, your host must have mod_rewrite enabled

Author(s): Charles

Date: 08/24/03

Credits:

Files Altered: ubbthreads.tmpl,postlist.tmpl

New Files:

Database Altered: no

Info/Instructions: This is basicly the same I posted in UBBdev chitchat forum. I am just fooling around and am sure cleverer people will tell me how to achieve this better <img src="/forum/images/graemlins/tongue.gif" alt="" />

the actual threads were added even though I am not satisfied with the name.

you can download it; removed, as Josh improved spidering

This mod has been downloaded []http://www.marsdev.net/mods/count2.png[/] times

You can now link to ubbthreads.html instead of PHP, the spider will not know he is accessing dynamic content.

Disclaimer: Please backup every file that you intend to modify.
If the modification modifies the database, it's a good idea to backup your database before doing so.

Note: If you modify your UBB.Threads code, you may be giving up your right for "official" support from Infopop.If you need official support, you'll need to restore unmodified files.

just for info:
right now threads are linked like this
thread.t1_10_0_collapsed_5__1.html t being threaded mode, 1 the cats and so on.

a bit ugly I agree. I will see if I can improve this before posting updated file

hmmm... for a spider mod it might be better and simpler to just use flat mode, great start tho, we were just talking about something similar last night

Yes, Allen and I were batting this around.

yes, extending it to showflat would be great for search engine content as well.

[]AllenAyres said:
hmmm... for a spider mod it might be better and simpler to just use flat mode, great start tho, we were just talking about something similar last night [/] well I think threaded or flat is not as problematic as the number of arguments you have to fit in the name

*edit* just got your point, actually you can use both, threaded is thread.t and flatmode thread.f but I might need some infos ob how search engines will react to the name I gave.

ack, just saw the first '.' - you might really wanna consider a naming scheme like showflat_92826.html or showflat-92826.html - for spidering purposes (and for a general decent looking archive as well) you don't need all those extra arguments. For general linking purposes, something like:

https://www.ubbdev.com/forum/showflat.php?&Number=92826

works as well as:

https://www.ubbdev.com/forum/showflat...;o=&fpart=1



You'll probably wanna include the pages if there's more than one, so you might want p2 etc in there, but all those other arguments are a waste for something like this.

One of the regulars here (name escapes me) has a way of using Rewrite, but not sure it is server efficient on high traffic sites..

well I was kinda suprised because from what I have seen the page number is actually not used. And at first glance I was not sure what all the other options do.

I could use only the thread number but the forum has to remain usable to people I think I don't know what kind of side effect this could bring.

Yes, I had someone interested in doing something like this. They suggested that using slashes (in place of &) was a better route. I wonder if there's any difference.

http://www.zend.com/zend/spotlight/searchengine.php

this would be the best thing ever! eace:

That page at Zend suggests if you use slashes instead....

Instead of
showflat.php?Cat=&Number=1234
Use:
showflat.php/Cat//Number/1234

That populates the $PATH_INFO variable in PHP with
/Cat//Number/1234

Then you can use explode() to poulate the variables:

If that method worked.... it could be done in the scripts..... and would not require mod rewrite.

So the idea here is to actually re-write the url's into spider-friendly url's? hmmm... would be a heck of a job updating your forum I'd think, tho I may be wrong.

I would suggest something else, to either create an 'archive' script that would give a seperate page for spiders to crawl like vb does, or do something like Micah did for ubb.classic and write a script that can make it appear like the urls are spider-friendly

http://www.praisecafe.org/boards.php

looks like this to the spider:

http://www.praisecafe.org/boards

leaving all forum files untouched:

http://www.praisecafe.org/ubbcgi/ultimatebb.cgi

http://www.praisecafe.org/ubb/ultimatebb.php

Yeah, you're right about updating. Alot of changes I think.

I'm not sure I know where to start with an archive script. but that would be best.

I saw your example but didn't follow exactly how it was being done.

ARe the arcives dumped and recreated each evening? What happens if someone adds to a thread.

"archive" is basically a "recent posts" mod with .html extensions... you see an example at the top of the photopost support forums. Here's an example of a seperate page:

http://www.destee.com/forums/archives/

forums are forum_number.html and threads are thread_number.html

http://www.vbulletin.org/hacks/index.php...&hackid=843

It does a similar action of micah's modification in that it's using mod rewrite to make the pages appear to have .html extensions and short url's

[]JoshPet said:
That page at Zend suggests if you use slashes instead....

Instead of
showflat.php?Cat=&Number=1234
Use:
showflat.php/Cat//Number/1234

That populates the $PATH_INFO variable in PHP with
/Cat//Number/1234

Then you can use explode() to poulate the variables:

If that method worked.... it could be done in the scripts..... and would not require mod rewrite. [/]

OK this is what will be used for classic so we might as well go this way. let me see what I can find on the subject

I've actually got this complete and we did some testing on one of my client's very large sites. 20,000 pages were successfully spidered the first night. mod rewrite was not required.
Once I get it tested a bit more I'll write it up and post it.

actually you can improve the process with mod rewrite by hiding the PHP file

Yes... they were concerned about the overhead of mod rewrite. I don't know much about it, but their server guy told them it added load to the server. Wouldn't be a big deal on most sites.... but this site already takes 2 servers.

ok, so lets do one version for them and a better one for the rest of the community lol

yes, let's do people, c'mon now, MOVE IT...MOVE IT...MOVE IT!..........:D

I've got something in play here.....

We always had 380 pages in google.... today we now have 11,800.

This works just fine on 6.2.3 as well

I have small problem

when i modify htaccess
i have mod_rewrite enabled


please help


Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, [][email protected][/] and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.



--------------------------------------------------------------------------------

Apache/1.3.27 Server at www.forum.tarnow.pl Port 80

are you sure mod rewrite is enabled by apache ?

otherwise contact joshpet he did a similar hack without requierment for mod rewrite

yes i,m sure mod rewrite is enabled

Silly question probably - how do I find out if I have mod rewrite enabled?

A little off-topic... but how can I see how many of my pages are listed in google ?

go to google and in search box type this

site:www.sitename.com sitename.com

The first line of my .htaccess file is a redirect:

Redirect /index.html http://www.mydomain.net/forum/ubbthreads.php

So will this still work if I add those 5 lines of code below that ?

hey,

can we get rid of this mod please? I don't want any of my competitors to find out

Wando, try this....

Test a very basic rewrite first:

rewriteEngine on
rewriteRule ^doesnotexist\.html$ /index.html [L]

Put that in .htaccess in the web root directory of a working site with an index page called "index.html" (you can change the rewriteRule as needed to use the right index page URI). Now, request "doesnotexist.html" from that site. The server should provide the contents of index.html.

If it doesn't work, then you have a problem with the way mod_rewrite is installed or configured on your server.

Josh.. you got this for 6.2.3?

I don't know, haven't looked at this.

This isn't what we are using here.

It would be great to have a working version of this

mine works just fine! Went from having 454 pages to having over four thousand and counting...

Okay, maybe it's something I am not getting right when appending my .htaccess file. Here is how it reads after installing the mod and my boards do no display correctly.

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://www.vhfans.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.vhforums.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://vhfans.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://vhforums.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^https://ubbdev.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^https://ubbdev.com/.*$ [NC]
RewriteRule .*\.(gif|jpg)$ http://www.bundymania.com/images/steal.gif [R,NC]
RewriteRule ^vhfans(.*)\.html http://www.vhfans.com/boards/ubbthreads.php?Cat=$1
RewriteRule ^forum(.*)_(.*)\.html http://www.vhfans.com/boards/postlist.php?Cat=$1&Board=$2
RewriteRule ^thread.t(.*)_(.*)_(.*)_(.*)_(.*)_(.*)_(.*)\.html http://www.vhfans.com/boards/showthreade...&fpart=
RewriteRule ^thread.f(.*)_(.*)_(.*)_(.*)_(.*)_(.*)_(.*)\.html http://www.vhfans.com/boards/showflat.ph...&fpart= [L]

The instructions read: You can now modify the links in your website to point to ubbthreads.html instead of
ubbthreads.PHP, and the spider will not know he is accessing dynamic content.

Where (or what links) exactly do I change from php to html?

yes hows thos coming? does it work? where do you change to html from php?

Where is the attachment?

Go Here to learn more about it.

This must be a freebie by now?? go on you know you wanna give it to us for free.. since 6.5 is going to have spider friendly urls...

6.5 does have a version of it - although I don't think it's quite as extensive nor does 6.5 have the ability for forum specific META tags.