UBB.Developers Forums UBB.classic V6.4 - 6.7 Modifications V6 Mod Ideas A brief introduction to the 6.7 Import/Export tool

Brief. Heh. This document is 3,200 words long. Happy reading!

---------

For a long time, we at Infopop have had a very annoying and persistant problem. New customers keep moving over to us from other products, but we didn't have the resources to develop custom import engines.

The main problem with writing importers is that everyone does something differently than everyone else. You can't just write one block of code and expect it to work for more than one product... or in some cases, even more than one version of any one product. Trying to keep multiple importers from multiple other products up to date with three of our own products would almost be a full time job.

A few months ago, the Infopop developers sat down and pondered this major problem. After looking at the database structures of a handful of other products, we determined that we could write a "generic" import engine - it would be given the data to be imported, along with a file that declares how the data is arranged. The main problem would be that because it would need to be so generic, it could only bring over categories, forums, posts, and users. Almost everything else, including settings, styles, permissions, private messages, etc, is done so differently that it can't be done by a generic engine.

UBB.classic 6.7 is the first beta product to have the new generic importer. Eve/UBB.x will be the next, followed by UBB.threads. I'll be writing the UBB.threads importer engine. (Note, May 3, 2004 - this was true when I wrote this document, but is no longer. Please take this fact into account when reading the rest of the doc. Rick did a good job with the 6.5 importer...)

By the end of the beta cycle, we hope to have format declarations for Invision, vBulletin, and phpBB. We already have experimental formats for all of our own products.

So, what's a format declaration? We settled on using YAML to store format data for the importer. The YAML spec will contain a list of dumped files in the export, what format the file is, and then the identifiers for each field, along with a set of possible transformations that need to be applied to the data.

Files can be in two formats: full CSV (the MS Excel style: unquoted,"quoted","embedded """quotes""" here"), or delimited by a single character (i.e. "this|that|these|those"). Fair warning: The UBB.classic CSV parser is very stupid. It does not handle CSV with much grace at all. It's based on the Text::CSV module from CPAN, which was last updated in 1997. All development since 1997 has gone into an XS version of the module... and as we can't compile anything on the server...

The UBB.classic and UBB.threads import engines can also natively read from a MySQL database. This feature is not in the original specification, and is not officially supported. Information on exactly what needs to be done to make that work will be released at a later time. The official method of importing is going to be through flat files.

Each file is expected to contain a series of columns with data. The data doesn't need to be processed in any way - the YAML spec will indicate what needs to be done with regexes. Let's take the following user file as an example...

Column one is the username; two is the password; three is the email. If the password is blank, then the importer will make one up. If the email is blank, then the importer makes one up. If there's no unique identifier for the user (i.e. a user number), then the user name is used instead.

This file would be represented by the following entry in the YAML spec:

The 'IP_Name' field is used to determine what the field really means internally. A complete list of IP_Name fields and their definitions is forthcoming. Probably next week.

The 'Index' field is used to determine which index the field is contained in, if the line were split into an array, starting with index 0.

The 'Key' field is where things get fun. Key fields are those that are referenced by other files in the data set. Let's use the following three files as an example:

The first file contains the user ID, and the user login name. The second file contains the user PDN, and the user number. The third file contains the user login name, and the email address.

To import this data correctly, we have to ensure that the importer knows that both the user number AND the user name will be referenced in other files. We would do this by marking both the number and name fields in the first file as Keys. In the second file, only the user number would be a key. In the third file, only the login name would be a key.

(The UBB.classic importer can handle multiple levels of keys - the email could be a key to another file, which could then have a key of its own that's used in yet another file, etc. Thankfully, such impossibly stupid data structures don't really exist in the real world. The Eve/UBB.x importer can only handle key references one level deep, such as the one above. Also note that for all intents and purposes, any ID IP_Name is always treated as a key (Forum.ID, Category.ID, Post.ID, User.ID))

The last field as shown above is 'Required'. This field is ignored by the UBB.classic importer at this time.

To demonstrate the next possible field, let's declare some forums and categories:

Confused? You should be. This file declares both forums and categories at once. Both Eve/UBB.x and vBulletin do things this way. The only difference between a forum and a category is the second indexed field - it's "yes" if it's a forum, and "no" if not. Let's declare how we'd process categories from this file:

The new field is 'Validate'. It takes multiple possible arguments, as a string. The string is basically a query string, and must be URL-encoded. Possible arguments for the query string are:
- equals=$string - where $string is a literal string that this field must match exactly
- notequals=$string - where $string is a literal string that this field must NOT match exactly
- regex=$regex - where $regex is a string in the form "/$pattern/$modifiers", where $pattern is a normal regex, and $modifiers are any modifiers to the m// function. An example might be "/^no$/".

Multiple Validate tests can occur. For instance: "regex=/w/®ex=/d+/"

If any one Validate test fails, then the entire record as read from that file is discarded. (This is something of a bug in the UBB.classic implementation - the Validate check currently occurs during a file read, but it really needs to occur when all of the data has been read, as to ensure that the ENTIRE record gets discarded, not just what's listed in this one file.)

Here's a snippet of the YAML for importing forums out of that same set of data:

Note how the Validate check on index two has changed.

As an alternative, I could also have ignored index two entirely, and did a Validate on index three. In this case, if the forum category ID is blank, then it's a category... and if it contains numbers, then it's a forum.

Also of note, while I'm in the area - the relationship between Category.ID and Forum.CATEGORY_ID is assumed. Forum.CATEGORY_ID does not need to be marked a key.

Let's move on to the final type of field. As an example, let's use a topic file:

Field one is the topic ID. Field two is the post ID. Field three is the post ID to which this post was a reply to. Field four is the topic subject. Field five is the topic body. Field six is the user number that made the post. Field seven is the unix timestamp for that post. The last field is the forum ID.

Here's how this would be expressed in the YAML:

Here we see the new 'Pattern' field. Like a Validate field, it takes a URL-encoded query string. The available arguments are a bit different, however:
- regex=$regex - takes a regex as would be passed to the s/// function. In the above example, the regex is "/
/n/g", which turns all
tags into newlines.
- epoch=(S|MS) - defines this field as an epoch date. If the argument is S, the epoch is seconds since 1970, like in Perl. If the argument is MS, it's milliseconds, like in Java.
- sdf=$string - defines this field as a Simple Date Format date string. Note that UBB.classic can not currently parse SDF - instead, any SDF fields are passed off into the newly distributed Date::Parse module, which can make sense of just about anything. SDF support is being built into the next release.

All regexes are run before date processing. All Patterns are run before Validates.

Also note that the regexes should be in the "///" format. Alternative delimiters (i.e. "~~~" or "{}{}") are not supported by Eve/UBB.x. Sorry. This rule also applies to Validate regexes.

As for the topic data itself... some items of note. Like Forum.CATEGORY_ID and Category.ID, the relationship between Post.USER_ID and User.ID (or User.USERNAME) is assumed. The same is true for Forum.ID and Post.FORUM_ID.

Also, the relationship of Post.ID and Post.TOPIC_ID is assumed. All posts must have a TOPIC_ID. If it's missing or blank, the post is assumed to be the first post in a new topic. If there are multiple files containing post data, the one containing keys for ID <=> TOPIC_ID will try to get imported first to try to keep things consistant.

REPLY_TO_ID support is limited in UBB.classic. All topics must be identified distinctly. I.e. you can't just have ID and REPLY_TO_ID - you must also have TOPIC_ID. In fact, you can skip REPLY_TO_ID altogether at the current time. This is considered a bug, but the overhaul required to get it working was too much this late in the game. Look for proper REPLY_TO_ID support in the next major release, and in the UBB.threads implementation.

So, that's about it as far as the importer is concerned. If you're confused and overwhelmed, don't worry... that's not entirely unexpected.

For more on the importer, I suggest reading the code - it starts at line 1140 of cp2_content.cgi

Now, for the exporter... it produces a simple |-delimited dump of all the data that it is capable of exporting. The first line of the first of each type of dumped file will contain a list of fields. (The files are size-limited to half a meg each. Only the posts files will ever grow that large. Only the first line of the first file in the set will contain a list of IP_Names.)

You should get a good feeling for what can be supported by the importer by running an export and examining the files. Everything that gets exported can also be imported again. Here's the YAML spec to import data from one UBB.classic into another:

I think that about does it. Some final notes...

The importer assigns completely random category/forum/post/topic/member IDs. Importing an export will result in completely different user and post numbers. This is not an effective backup tool for that reason.

Duplicate member records will not be imported. If a user with the same login name (User.USERNAME) exists, then all imported posts will be attributed to him/her.

Sorry, there are no date restriction controls on the import/export. It's an all-or-nothing thing. Likewise, there are no forum controls (i.e. "export this forum only" or "import this forum only").

The export process will increase your disk space use by about 2/3. The import process will increase your disk space use by about 300% while it's running, but only about doubles it when it's done deleting all the temporary data.

Sorry, there is no support for data compression, either ingoing or outgoing.

Importing about 50,000 posts and 5,000 users takes about an hour on a decent sever.

The UBB.classic import process is not memory-friendly. Restrictions on some service providers may cause imports to fail. Unfortunately, I don't have too much control over this. The UBB.threads implementation will not have this issue.

If you need to split up the number of messages being imported, you must attempt to import members from both batches. Only the first batch will actually import members. You need to attempt to import members during all subsequent batches so that user data can be properly reassociated. (For instance, if everything goes by a unique user ID, there's no way to associate an imported user ID with the post-import user ID once the import completes. Because members aren't duplicated, the association with the existing imported member will occur instead.)

Clear as mud? Good.

Questions?

Did I scare everyone? %)

not anymore than usual My head still hurts from the cp modification thread

This may be a bit off subject and should be posted elsewhere but here goes. I was thinking of switching to threads from classic at some point. My biggest consern is do links users left pointing to other posts on the board get un-useable because they still point to the old board. Or are they found and updated as well?

This has been a problem over the years with just upgrades.

Thanks
Bill

there is no easy solution to that problem as you would need to have a cross reference table with the old and new links then use an htaccess rule to foward people to the new thread. (if you are running under linux and apache)

now this is possibe but will need a bit of coding and modification of the actual thread importer.

you might want to post a job offer in the dev for hire forum when you do actually decide to migrate the forums. (or contact me directly )

Let me run this one by you...

Would it be better before I export/import classic to threads to run a script to change the links to what they need to be on the new board? Then export/inport to threads. I've got a script I've used in the past to change the old .html links to the current .cgi links. I'm sure it could be modified easy.

bill

The new topic numbers won't be known after the import is complete, which is why a modification of the current Classic -> Threads importer would be best. It can keep track of the previous forum and topic numbers... then a replacement ultimatebb.cgi (and ultimatebb.php if you use the Accelerator) can look the requested topic up in the database and point the user where they need to go.

I am sure you are talking about the old Infopop script that was used for the move from 5.x to 6.

The problem with a move to threads is that a thread number is defined as the overall thread number and not just the thread number in a specific forum.

In threads all posts are in the same table whereas in classic the next thread number deponds only the threads in that single forum.

Thus to do a link redirect you need to add (in a database for example) a reference of the old forum/thread number and the new threads thread number as there is no way to know it so it really has to be done during the import.

A good reason to stay with classic, I guess.

I figure that would be the problem.

Thanks, at least I know my choices.

Bill