UBB.Dev
Posted By: OziBen Googlebot and friends - 07/08/2003 7:58 AM
We have an very active forum and wish to get this content listed on search engines

Infopop tells me that dynamic content is not listed?

anybody know how to get googlebot to list forums

I have read of spider hack
http://www.totalwebpackage.com/ubb-spider-hack.html
any experience with this mod or any other ideas please

Also does anyone have a script to archieve threads to html, or some way of converting old threads to html content for a new site

We could manage a small fee for someone to assist with these problems
Posted By: lbritish Re: Googlebot and friends - 07/08/2003 7:09 PM
Not sure on listing forums but google is an odd thing. My site is like 80% dynamic. Google basicly has the following rules from what I can see.

Dynamic content IS ok... but BIG BIG BUT.... it can only have one querystring variable... in otherwords:

mywebsite.com/dynamic.php?key=stuff is fine.....
mywebsite.com/dynamic.php?key=stuff&category=things&type=12 is NOT ok...

in the case of ubb.... I would ASSUME.....

https://ubbdev.com/ubb/ultimatebb.php?ubb=get_topic;f=8;t=003571 is bad because the ; counts as a break and new variable...

BUT I'm not familiar with how UBB processes the query strings... maybe if you made it take all the variables and smash them together with some specific pattern that doesnt happen in ubb lines and do some sort of programatic loop to find the variables by extracting them from a single ?ubb= line then that would probably fix it...

in otherwords... if you made it send
https://www.ubbdev.com/ubb/ultimatebb.php?ubb=get_topic___f=8___t=003571 using ___ as a separator... doing some sort of regular expression based string selection looking for patterns of ___ separated stuff and then processing those..... *Shrug*

I'm guessing this would be a pretty major rewrite though..... but really only effort should be placed on the display of messages and forums.. the rest does not matter so much I wouldnt think.. possibly even some sort of google sensing code to feed out maybe a different page and when the user shows up it redirects them back into the regular stuff based on an "encrypted" string of jibberish that can be translated into normal keys? aka if ?ubb=ksmdlk2moim2#%2imomfgklm233ijorfm2klml23@#fkmo2 somehow gets translated into ?ubb=gettopic;f=8;t=15534 etc.... see "vernam encryption" coding samples found on the net as an example of what I mean if someone wants to try programming something to do something like that...just hard code the encryption key into the source since we're not really trying to secure anything.. just jam it all together... lol... but I still think that would need some sort of regular expression string manipulation...

Outside of all that stuff above... I have no other idea other then making your ubb generate static pages for the messages like the old ubb's used to do which would eat the hard drive space up big time on large forums....

And in regards to how to do the above... I dont really know... lol... Just my idea based on what I know but all bets are off on how I'd go about implementing that... lol... I dont know CGI and PHP well enough to tweak UBB to do something like that... just know that's probably how you can get around it...

you would only wanna jam that into something bots would see though...
Posted By: AllenAyres Re: Googlebot and friends - 07/09/2003 6:07 AM
Would you not do better then to submit your forum_summary and get_daily pages for the widest range of topics?

Charles will always say there's mod_rewrite smile
Posted By: Charles Capps Re: Googlebot and friends - 07/09/2003 9:09 AM
You don't want a spider *EVER* hitting ultimatebb.cgi, even through mod_rewrite. There's a good reason why polite spiders don't try looking at dynamic pages...

At best, a cache builder and the Accelerator + mod_rewrite would be a solution.
Posted By: OziBen Re: Googlebot and friends - 07/09/2003 2:16 PM
Can any of you guys get these forums listed in search engines We will pay
Posted By: AllenAyres Re: Googlebot and friends - 07/14/2003 2:56 AM
good question oziben... the problem is even if you get some pages spidered, they don't go any further because all links from there have the ; in them.... vb has a decent mod where you can have so many t124236.html for topics... I see it at photopost. Maybe something can be done to at least do that for the get_daily page.
Posted By: Gizmo Re: Googlebot and friends - 07/15/2003 10:05 AM
I'd kill to be able to have my content spidered :x... who cares if our bw is 50gigs as of last month! lol...
Posted By: OziBen Re: Googlebot and friends - 07/15/2003 10:49 AM
spider hack is working
googlebot is already in my forums!!!!!
also UBB pages are now valide html
for once of fee of $150 you too can have your forums listed
Posted By: AllenAyres Re: Googlebot and friends - 07/15/2003 4:37 PM
It says $50, not $150 tipsy

Nice work, may be worth it then smile
Posted By: OziBen Re: Googlebot and friends - 07/15/2003 5:39 PM
resellers need a markup
Actually I paid for installation, support and validation of code also
Posted By: Gizmo Re: Googlebot and friends - 07/16/2003 12:32 AM
Out of sheer curiosity I'm entering into an affiliation with the coder of this addon just so i can review it :x... I'll let you all know how things go...
© UBB.Developers