UBB.Developers Forums UBB.threads 7 Development How Do I? Remove Orphan Attachments

I don't know if this is at all possible.
Maybe someone would know how to make a script to do this. I don't have a clue.

But here what happened... and it happened a long time ago...but I'd love to free up the space.

Our board originally had some photo forums... with about 2000+ pictures. We switched to photopost, and reposted them all there.

This was back under threads version 5 something. So I wanted to delete the posts, and the attached files (as they were taking alot of space), so I went under admin to expire threads.

I got some error there, and the threads were removed, but not the attachments. Looking back... it was probably a permissions thing of some sort... as some were transfered from another server.
So I thought... If I delete the forums....that deletes all posts. Yes... .but it didn't touch the attachments either.

So now I am stuck with 2000+ orphan files in my uploads directory...with no matching post that I'd love to get rid of.... but there are many legitimate files there... can't go by date. There's no way to tell.

What got me thinking is that photopost has an admin fuction, that goes through the database and finds posts with missing images, and images with no posts and deletes them... sort of a database clean up function.

I don't know if this would be possible.... probably it would be more possible to have something which scanned the post numbers with file attachments.... and copied those files (that I want to keep) to another direcotry... then I could delete the "orpans" that were left over and are not attached to any post. But of course, I don't have a clue as to where to start.

Any ideas/ advice?

No urgency.... it's just I'm wasting several hundred MB of storage space... and my server is fast reaching capacity.

If they are the oldest files in the directory, perhaps you can delete them by looking at the date the file was created/modified/written?

1. You would need to figure out what existing posts contain attachments and build a list from that, setting them into some sort of an associative array for refference.

2. Read the files in the directory and create an array also. (simple numerical one will suffice)

3. Cycle through the numerical array using the value (the file name) to check if the associative array has a value. (1 = post exists) Then delete the file if not related to a post.




Perhaps I'll just try and show the scripting... lol

I'll try and put something together after dinner

Here's the algorithm for your script:

code:
for (each file in the attachment directory) {
   parse filename
   if (filename contains a post number) {
      query database to see if post exists
      if (post exists) {
         continue
      }
   }
   delete file
}

---

1) For initial testing, I would have the script print the filename to a log file, instead of deleting it. Then you can spot check to see if it picked the correct files.

2) As an alternative to deleting the files, the script could move them to a backup directory. Then they could be mass-deleted after you're sure they're no longer needed.

Placing the file names from the directory and from the posts into arrays and comparing them keeps the queries down to only 2 per deletion run. Otherwise it's 1 query per file in the directory. I have a limit of 50 file unlinks (configurable) per updating to limit the steady stream of unlinks. (one disk seek/access per unlink so if he has 2000 files to delete I don't think we want to send 2000 unlinks to the system) LoL

It's a simple script. I have it almost finished.

If you want to check user status then that will take an aditional query per run making it 3 total.

I hadn't really planned on making it look nice but perhaps this may be another utility that could be placed into the admin section?

Attached is a simple script that will display the number of total files in the attachment directory, the number of files that belong to existing posts and the number of files to drop as orphans.

This will not delete anything though. Just wanted to see if the numbers look right... lol


I don't have anything to test this out on.

I tested this here at ThreadsDev and got these results:

You have 431 files in your attachment directory.
399 files belong to existing posts.
32 files are orphans...





This sounds plausable... ?

The by date thing doesn't quite work for me.... as the forums I deleted weren't all the attachments we had.... there are attachments in that date range which need to stay.

I am forgetting a second forum... thus I was now thinking my orphan attachments may be closer to 4000......

Dave,
What you described sounded awesome.... but I didn't know where to begin.

But I run you checker:

You have 5196 files in your attachment directory.
1513 files belong to existing posts.
3683 files are orphans...

Shocking isn't it?

But guess what... it IS right.
I checked the number of attachments using attachment pal... and 1513 is correct.... yet the number of files in the directory is greater than my ftp program will even display.

So your script is working.

Now.... Mr. PHP Genius... can you tweak it to either delete the orphans... OR copy the "real" ones out? Whichever would be easier.... if I copied the good ones out... I could delete the contents and then copy the real ones back in.

The would make a cool utility for the Admin area. As I said, Michael (extremebikini) introduced this in the latest version of photopost. The average user wouldn't have such an extreme case as mine.... but as you found with threadsdev.... it does happen.

I appreciate your help so much! Thank you.

I should be done with the utility before the night is over. Will post soon.

Just trying to make it controlable as far as the impact on the server goes.

You are awesome!

I guess you can see by the number... why I'd be happy to get these dud files off my server. They have all been duplicated on the photopost side of things.

Just something interesting I noticed here. The attachment list is returning 590+ attachments but there isn't that many in the uploads directory. I would venture to say that there are missing files as well as orphaned ones.

I have attached the utility that *should* do the trick. You can set the $delmax variable to what ever value you wish but I wouldn't recommend dumping 3,600+ unlinks to your server

The file is attached. Place it in your admin directory and create a link in menu.php under "Post Management" or where ever convenient. Example:

<a href="{$config['phpurl']}/admin/delorphanfiles.php" target="mainFrame">
Delete Orphan Files
</a>



You may want to test this out somewhere first... LoL but I think it should work.

This should probably be posted in the beta mods but I wanted to make sure it worked first.

Dave,

First and foremost... as always... YOU are A W E S O M E!

This all seemed to work fine.

The numbers don't quite match... but it did a fine job at trashing the junk files.

The delete script hiccuped with 3 files and returned a "Unlink Failed" error. Those files had a ' in the name...which it didn't like. No problem... I deleted those three manually.

The script said (when it was done) there was 1513 files in the directory... which is correct.

When I checked the Pal list... it had 30 full pages of 50... there were no more pages. So maybe that stops at 30 pages????

When I searched the database (using myphpadmin) for % in the file field, it returned 1554.
So there are some messages with missing files... which is very possible.
Before I changed the upload limit in PHP.ini file, many users used to time out during large video uploads.

But really.... it accomplished my goal... and got rid of all those junk files!

[:"red"]Thank You!!!!![/]

That's good to know it worked

The attachment list shouldn't have any page limits that I'm aware of. I hadn't thought about special characters being in the file names so I'll adjust the script to include escaping them in the names when unlinking them.

I'm not sure what could be going on with the file count vs. the attachment count though. I did make the file count skip over directories and those files that didn't have an extention in them as I figured these would also be directories.

I'll tune on it some when I get the chance.

I'm not worried about it....
for the most part it removed all that "dead weight" we've been hanging on to. Thank you.

It works, but it also deleted everyone's picture that they use in their posts if they've uploaded it. I found that out the hard way.

Ah... yeah... If you are storing user pictures in the same place as uploads... it gets rid of any file not associated with a post. Doing the photo uploads for user pictures to a separate directory is best. But I think I had a bit of the same problem.

Hence comes in the 'seperate upload folder for different forums' suggestion.

I had to improve this just a tiny bit. I like to know what I am deleting at times. This will make each file a link and even open in a new window for you.



Change line 71 to this....


$dellist .= "<tr><td align="left" class="$style"><a href="{$config['fileurl']}/{$delfile[$i]}" target="_blank" />{$delfile[$i]}</a></td></tr>";

[]JustDave said:The file is attached. Place it in your admin directory and create a link in menu.php under "Post Management" or where ever convenient. [/]

Dude, this script rules! I was looking for something to clean up my attachment and avatar directories. While this doesn't clean up orphaned avatar images (hint? ), it was very cool to clean up my file attachment directory.

Thanks!

FYI, I ran it on 6.42, and it worked fine.

ah I really need something like this!

this is my problem:

I use an uploader script that stores pics in a folder, that is all fine and dandy

But now what to do when the original post is gone but that picture is still sitting on the server?

So what I need is a way to scan all pics and posts and compare to the list of pictures then delete the ones that are not called via the [ image ] tags in any posts

Can it be done?

Bump.

Doesn't work in 6.5. At first I thought it was because 6.5 moved the location of main.inc.php. I edited the file to the proper location, and then I got this error:

[]Fatal error: Call to undefined function: admin_table_header() in /usr/home/web/forums/ubbthreads/admin/delorphanfiles.php on line 85[/]

Any ideas?

i try this to run under 6.5. After changing the includes path I get:

Fatal error: Call to undefined method html::admin_table_header() in (path)/ubbthreads/admin/delorphanfiles.php on line 85

What to do ?

This would need to be reworked quite a bit - as the way the admin pages are generated are different in 6.5 - it's trying to generate admin_table_header - which is a function which no longer exists in version 6.5.