Both my Boxsters were flawless cars despite 2 prior owners each. Boxsters are cars that fit a certain point in one's life for most of us. A good PPI from a great mechanic who is going to service the car with the info gained will reassure you. Having said that, you are looking at 7-8 year old cars and things do age (plastic/rubber) not just wear out. I'd rather have the well maintained 60k car than the garage queen 20k one.
Use
http://www.autotempest.com/ to look at many sources and see if you can find the right one for you. And locate that trusted mechanic first. He is your friend and savior.
No big differences between a '03 and '04 car, the big differences came at 2000 and 2003 and of course 2005.
Considering that you can get a well maintained '01S for $15k, the prices you are quoting seem high to me but then condition and locale do come into play and it is spring.