I've seen the subject brought up before.. It would be really handy if you had a script that could spider your forums, saving the pages locally on the server as static HTML pages. This would allow users to browse a super fast copy or you could make CD-R's of your site which are browseable offline. I know there are programs out there to save webpages but they never seem to work all that well with forums..

The script would have to:
* Modify the links in the pages to point to other static pages instead of the usual links to the dynamic site.
* Save images locally that are linked to and change the corresponding img tags.
etc..

It would be quite a project but could be done.. Ideas?

this is what your looking for

https://www.ubbdev.com/forum/showflat...5/o/all/fpart/1

To me it looks like that just converts the URL to a spider friendly one and doesn't actually save a new copy of each page as an HTML file... am I wrong? Their page doesn't exactly list what all features it includes..

I guess it would make it easier to spider with archiving software but I would hate to have to buy that as well as the archiving software..

I beleive it converts the pages to html through this hack

Here is a link from Allen's board which uses this

http://www.praisecafe.org/boards/ubb-get_topic-f-9-t-000240.html

No I think it just uses rewrite to make them "appear" as HTML pages. I have not ever seen a hack that physically exports the pages to HTML so that you could download and browse offline etc....

[]3DSHROOM said:
To me it looks like that just converts the URL to a spider friendly one and doesn't actually save a new copy of each page as an HTML file... am I wrong? Their page doesn't exactly list what all features it includes..

I guess it would make it easier to spider with archiving software but I would hate to have to buy that as well as the archiving software.. [/]

You can get more info on this here https://www.ubbdev.com/ubbcgi/ultimatebb.cgi?ubb=get_topic;f=10;t=002400;p= and if you write the developer, Micah [][email protected][/] he will most likely be happy to answer your questions.

I suppose it is possible that one could construct a script that would scan forum pages that are open to the public, extract links and then send a GET request for those links and then store the returned information as an html file.

What a nightmare that would be. lol

heh, yeah

wget is your friend.

Type this in a unix shell account:



And you will have a complete working local copy of threadsdev. After a while, it'll take some time downloading everything. =]

If you are on windows, you could use something like cygwin to use the wget program. Don't ask me how cygwin works though, because I don't know that.

There is a windows gui available on this site: http://www.jensroesner.de/wgetgui/

A nice thing of archiving would be that once archived database could trimeed, reducing server load.

Garderner, out of curiosity... so you are saying wget could be run on the server side? I sort of tested wget (never worked for me) on my Firebird, as an extension, but it needed some updating so it didnt work... But being on the server side sounds very interesting.

wget seems to have a few problems converting the urls to ones that are valid filenames.. you can't have question marks in the filenames... using the Spider script, wget would probably work quite well.. and yes he probably means server side, although it could be done from any machine..

Well, I ran wget with the arguments I gave and I got a complete local copy of my own site. It changes all links so that they work with the saved files, whether there are question marks or not.

It's not really server side, since wget is a http client (like a browser) but I ran it on the same machine as the site was on. But as long as you have wget you can run it from any place you want to download the files, it'll just take a bit longer to transfer everything over internet.