Thanks for all the comments so far.
I'm definitely not looking for power above all. I've driven both and found them both fun. Problem is, I can't remember the non-S feel since I drove it first. I don't have access to one (non-S) in the local market to do some more test driving.
I agree with the comments about always looking for more power. I went from a 325i which certainly had decent power to the 330i which has noticeably more. After a few months with the 330i I started wanting more!
I'm guessing that the non-S would be a bit slower than my 330i and the S would be a bit faster.
Finally, which ever way I go, I'll probably always wonder if the other way was the better choice!