It's possible that someone could spam your board from another domain, but I've not had it happen to me since Ihad to disable it on my site.
Many new firewall programs (Zone Alarm, McAffee, etc) remove the referer information from http requests as a "privacy" feature. Nothing but marketting BS -very little, if anything, can be gained by watching where people are coming from to get to your site.