UBB.Developers Forums UBB.threads 7 Development How Do I? Any way to scrape a UBB Threads forum and create an import?

We have an issue w/ our current vendor hosting company and he's turned off telnet and ftp. We have a new install of UBB Threads going on in another box. Is there a way to "scrape" the forums & discussions without losing much of the discussions after we get the present box turned off? There are 2 separate Linux boxes and 2 different hosting companies. Yes I have phpMyAdmin installed in our new install but I would need a compressed file to import. Are there any utilities that can do this? I know with phpMyAdmin you can try the "Database Command" window or do a Select * from w3t_Users; it's going to try to return six thousand plus rows returned from the database in a very raw format. But then I suppose I'd have to convert that into a delimted format, e.g., tab-delimited. But then I'd have to import it some how into the new install. Any suggestions?

I suppose I'd lose all of the users as well?

All your data is in the database. From the posts to the users, it all stored in the database. If you can get a copy of your database you can easily get it moved over. If not, I don't think that there is any way.

Unfortunately due to issues beyond my control and basically a dispute between 2 parties, it doesn't look probable that I will get my hands on the DB. Even if I could save 80% of the threads I'd be a hero so I'm open to any suggestions.

Do you have a Control Panel that you can access? You can also get it out of there in most cases. Without access to the DB you are out of luck

I believe that was turned off as well...

Can you login to your old threads install? If so, you can make a DB backup from the admin area of UBB Threads.

Continuation of a discussion from Infopop community.

I don't know what the "issue" is with your hosting service, but it would be much easier if you could either get your telnet/FTP access turned back on temporarily, or if they could provide you with a dump of the MySQL database.

Another alternative is to access the old MySQL database remotely. That can be done from your new server either using phpMyAdmin (with the procedure I posted at Infopop community), or with the Linux shell command mysqldump. Actually, if you have telnet access, Linux shell commands are preferable, since it avoids the problem of CGI timeouts. The catch is that MySQL on the old server has to be configured to allow remote access. I don't suppose your old hosting service would be willing to set that up for you?

If none of the above is possible, I don't know of any scripts lying around to do what you want, but maybe someone else here can help.

Actually, I did something very similar to this a few years ago when forced to vacate my server due to an "uncooperative" hosting service. I wrote a quick & dirty C++ program to "scrape" the web pages and reformat them for my new server. But that was for a completely different web script, so I doubt if it would be of much help here.

KYDD,

What access is left to your old forum?

[]Omegatron said:
What access is left to your old forum? [/]

The forum is running. Just the passwords to CP, FTP & Telnet has been changed while this--ahem--disagreement is taking place. And unfortunately it doesn't look like it'll get resolved any time soon.

So can get ftp in to get the software?

Well FTP is not closed--the login and password have been changed and are being withheld until this dispute is settled (which doesn't look like it will ever happen).

Well there are a few ways to get your database.

It looks like you need to SETTLE the dispute or start over.

1.Use PHPmyadmin on old server and dump database.---YOU CANT DO

2. Use admin menu in threads to do backup and then ftp in to get them in backup directory provided you set this up.
YOU CANT DO EITHER

3. Telnet in and get database YOU CANT DO.

4. Have them send you a MYSQL backup BASICALLY WITH THE DISPUTE HIGHLY UNLIKELY.

5.Remotely link PHPMyAdmin and dump database. Possible but HIGHLY UNLIKELY AS MOST HOSTS DO NOT ALLOW REMOTE LINKS.

Basically you are being told by your HOST WE WIN you settle or start your site over.

There is may be an option here. If you adjust your config to put your database dumps into your html tree, then you can run the backup. Then you should be able to type in the url of the sql files and save them locally. Then you should be able to restore the dumps.

[]Omegatron said:

5.Remotely link PHPMyAdmin and dump database. Possible but HIGHLY UNLIKELY AS MOST HOSTS DO NOT ALLOW REMOTE LINKS.
[/]

How do I check if we can do this?

Also, if I get the login/password for the control panel/admin, will it let me do a backup AND download the file through the interface/GUI or do I have to Telnet in to get it after it's created?

PHPMyAdmin lets you download the file.

BTW, what I described above, that is through the UBB Threads interface.

[]slayer60 said:
There is may be an option here. If you adjust your config to put your database dumps into your html tree, then you can run the backup. Then you should be able to type in the url of the sql files and save them locally. Then you should be able to restore the dumps. [/]

OK how do I find the url of the sql files? I take it that creates some sort of "dump" that I will be able to import? Sorry if this is obvious as I'm new to myPHPAdmin & MySQL...

Slayer he does not have PHPMYADMIN on the old serve

Kydd if you have no FTP ACCESS to grab the files created by the threads admin interface then that is not good.

REMOTE LINKING. you most likely would have to ask the host if they could set this up. If you are in a dispute with them this would be highly unlikely.

If I am misunderstanding and this is a dispute with another admin then you can call your host and have them do a dump and email it to you or make other arrangements. However my assumptions are based on this being a DISPUTE with the host in basically they have cut off all access for you.

When was the last time you backed up your forums (and downloaded the backup onto your hard drive)?

Perhaps you can change the settings of what files are allowed for upload on the old forum and then upload some scripts to create backups of your database. Perhaps uploading a script that would grab a preconfigured phpMyAdmin package from your new server and unpack it on the old one and then use it to access your database as needed?

Just some thoughts.

In config in ubbThreads find this:
Directory to store database backups in. Only necessary for using the database backup/restore function in the admin section. This directory will need to be writeable by the webserver.

Set this to a path within your html tree. Such as:
/home/wwwtsbiu/public_html/ubbthreads/
/home/wwwtsbiu/public_html/ubbthreads/images/

You should be able to figure out what this needs to be based on your other paths in config.

Save that, then go to DB Management and run the DB backup. Backing up all of your files.

Then you should be able to type in the url to the sql files. Eg:

http://www.2-pop.com/ubbthreads/devbo_Posts.sql

That is what I would try. This is dependant on you have the correct value for:
Path to mysql. Only necessary for using the database restore function in the admin section.

I think that should work even if you don't PHPMyAdmin access.

Good thought Slayer!!

That may just work.

Since it has to be a writeable directory, it would have to be something like the /includes, /stylesheets, images/graemlins, or images/icons folders because you proabably can't change permissions any more w/o ftp or telnet. As long as you correctly set permissions at installation and haven't changed them, there should be several writeable directories in your ubbthreads tree.
EDIT:
If you have a directory for attatchments, that should also be writeable already as well

I think we have his workaround.

I don't know how you do a backup from the admin control panel. Does it drop one table at a time? It looked that way from slayer's example. If so, get users and posts first. Then maybe forums and categories. That's assuming you can't just do the whole database at once. but it is a big site. Must be very sly....

It drops them as individual tables. It has a link to Backup all tables, but it still outputs them as individual tables. With a large site this will probally be best anyhow.

I personally don't see why he does not settle his dispute with the host.

Yes Threads Backup drops them one at a time.

Slayer is right as backing up the individual tables is alot better on bigger sites. I can't tell ya how many times I have seen PHPMyAdmin time out on larger databases trying to be backuped in a large chunk.
This process for him will be time consuming.

KYDD,

If any of this is over your head I suggest you post a post in our developer for hire forum and they may be able to do something for you.

I don't what host he has or what the "dispute" is, but given that the host severed telnet(I've never had this) and FTP to hold the site hostage, there must be some seriously bad blood there for some reason. Could be a billing thing, could be a move to try to force him to renew for a year or whatever to keep him from moving. Those are the only specultations that I can offer that might be unresolvable.

Just to clarify, the dispute is not my doing. It's something that the billpayers and the current admin (not his hosting company) are hashing out and I was told that my task was to save as many of the threads & users as possible since we're locked out and the possibility of a mutual resolution (w/out legal proceedings) is slim and nil.

I'm not worried about how long it'll take as long as I can get a good amount of the threads & users in. I'll give Slayer's idea a shot tonight.

I just wanted to write that all of you guys are awesome. I've been making the transition from UBB Classic and perl to Threads so I'm taking the MySQL & PHP stuff slowly. Thanks again and I'll be back tomorrow to report what happens.

You are not the sites owner? You are not the owner of the Threads software? Am I reading this right?

No we are absolutely the owner of the software, the license, the content, etc. The dispute is between the Sr. VP of our company and the current outsourced admin. Not sure about the details actually but it is $$ related.

OKay thanks for clarifying.

IANAL - But in all reality they can not withhold your database as it is your property. Any legal issues between both parties would need to be settled in court if need be but keeping your data and causing downtime and other related incured costs wouldn't be a good idea on their part. This would be grounds for a counter suit.

But then again I have no idea... where's Lisa_P when ya need her... lol

you have ssh or telnet access on the new machine, but not the old?

Here is a fast way to see if you have remote access to the old MySQL server


$bash 2.0> mysqldump -u username -p -h oldhostmachine.com --opt databasename > databasename.sql

You will be promted for a password. then if it connects you're in luck.

Good Point by both Dave's to the rescue one with free legal advice and one with a nifty telnet command to use on the new server.

[]navaho said:
you have ssh or telnet access on the new machine, but not the old?

Here is a fast way to see if you have remote access to the old MySQL server


$bash 2.0> mysqldump -u username -p -h oldhostmachine.com --opt databasename > databasename.sql

You will be promted for a password. then if it connects you're in luck. [/]

Unfortunately to answer your 1st question--yep.

As for the login attempt, using an educated guess on the correct variables from navaho's suggestion:



I guess that implies SSH is not available from my IP and may be limited to an IP range or turned off. I have to use SSH since that's what the new server is using.

Then your only left variable would be backing the threads backup directory to within Threads directory in a directory which has write permissions. Your attachment directory is perfect and then downloading them one at a time via your browser.

Also while browsing your forum I see you are in violation of Infopop's copyright rules. You need to place the Infopop link back on your site.

[]slayer60 said:
In config in ubbThreads find this:
Directory to store database backups in. Only necessary for using the database backup/restore function in the admin section. This directory will need to be writeable by the webserver.

Set this to a path within your html tree. Such as:
/home/wwwtsbiu/public_html/ubbthreads/
/home/wwwtsbiu/public_html/ubbthreads/images/

You should be able to figure out what this needs to be based on your other paths in config.

Save that, then go to DB Management and run the DB backup. Backing up all of your files.

Then you should be able to type in the url to the sql files. Eg:

http://www.2-pop.com/ubbthreads/devbo_Posts.sql

That is what I would try. This is dependant on you have the correct value for:
Path to mysql. Only necessary for using the database restore function in the admin section.

I think that should work even if you don't PHPMyAdmin access. [/]

Well I'm pretty sure I got all of the settings correct on the new box. But I'm trying to hunt down the url to the sql files. The 1 suggested above obviously doesn't work/ Any other suggestions?

You set the backup directory on the OLD BOX to something within the directory tree.

If you allow uploads into something like /uploads the web page would be http://www.2-pop.com/uploads/

and the sql tables would backup there.

This would be on your current board not new one.

I have offered in private to help twice and you have ignored me in both instances.

Another Gentleman was having trouble with his forums and I got them up and running for him. Obviously you do not need any since you have ignored my attempts to help twice.

[]Omegatron said:
You set the backup directory on the OLD BOX to something within the directory tree.

If you allow uploads into something like /uploads the web page would be http://www.2-pop.com/uploads/

and the sql tables would backup there.

This would be on your current board not new one.

I have offered in private to help twice and you have ignored me in both instances.

Another Gentleman was having trouble with his forums and I got them up and running for him. Obviously you do not need any since you have ignored my attempts to help twice. [/]

Sent you a PM...saw that you read it

But I DO really appreciate your help and have been "studying" everyone's suggestions. Now I have to wait until AM to see if the admin still has admin privliges to set the current (i.e., soon to be old) board. And I'll have to make some educated guesses about the location of mysql or mysqldump. It's most likely the default or /usr/local/bin.

/usr/bin /usr/local/bin

usr/local/mysql/bin

usr/mysql/bin

I would just try the default

/usr/bin

I sent you a PM on this but just so your clear If I where you I would inquire at Infopop the true license holder. In a corporate structure like yours if so and so's name is on it then they are the rightful possessor. If it is held in a corporate name and they are no longer with the company then the company is the rightful heir. I would check to see how it was licensed.