I noticed it was slower too when enabling the poll feature, but for my site it's a hit and the polls have never been more active, so I guess the trade off is worth it. Maybe IIP 5.4.x will address this issue, unless of course the speed is purely dependent on the server.
Donate to UBBDev today to help aid in Operational, Server and Script Maintenance, and Development costs.
Please also see our parent organization VNC Web Services if you're in the need of a new UBB.threads Install or Upgrade, Site/Server Migrations, or Security and Coding Services.