What's the URL to your board?
Have you installed any of the various PHP accelerators/compilers/caches?
You've also pretty much answered your own question above... you probably need more memory, but as your server can't be expanded any further, it's time to look at a new one. You might also consider buying a second box just to host the database, if you can afford to do so with your host.
