I have an '05 S, but have not driven an earlier version to compare it to. I did drive a Mercedes SLK for comparison, and the Boxster blew it away. My dealer now has left over '04's, '05's on the way, and you can order '06's; so an '04 would probably be a deal if you are looking for value.
I love the car. Clutch, shifter, steering, brakes are right on.