I only put the top-down during the summer when I don't have to really be anywhere and look presentable.
That, and don't have to deal with traffic lights (so I can keep a breeze going).
I have, though, put the top down when it was 34-degrees at night and everyone was looking at me like I was bats**t crazy.