My top down point is 50 degrees F. I have driven in 90+ degrees with the top up simply because my arms were turning bright red from sunburn and I needed a spatula to pry the boys off the leather (sorry if this reference offends anyone)...
BTW - There is nothing more awesome than driving top down on the interstate on a clear night into Chicago with a hot chick in the passenger seat!