i went with a 'total inbox size' method. 'size' is calculated as the actual DB used for all messages in the inbox PLUS the attachments (yes, attachments is a mod).. also takes into account number of peeps in the PT and divides accordingly.
now attachments also end up being shared, so i don't kill the poor bastard with the whole attachment either. i also limit attachments to images (png|jpg|gif). if they wanna send zips and rars, they can use email..
reason i went with attached images is to make it easy to upload screenies and all that directly into the PM.. it auto inserts the [img] tags after upload so they can add text before / after as part of the post.. this replaces the current attachment system, where it's just appended at the end and really doesn't seem to be part of the post
i'd recommend this only if your hosting plan has the room (HDD space), otherwise, just kill attachments as the option and just tally up the PM_POSTS size amortized against all the participants.
also, i added this calculated field to the online table (per user_id), so it's easily grabbed. then viewmessages.inc.php does a recalc whenever it's invoked. this saves always computing, while sacrificing the small possibility that a user barely might go over their limit for a short while.