I would roll top down in the winter, but I'm too scared of my plastic window breaking. Luckily, when it's too cold, it's usually accompanied with snow and therefore salt so it doesn't get driven too often in that weather.
I only put the top down at night since I get more privacy. 70 F degrees and warmer.
I like driving with the top down at night, but during the day I keep the top up to stay cool and avoid sunburn/skin cancer and avoid people from starring at me.
I agree with the other posters - the top only goes up when it's raining. Or, rarely, when the trip is so short that there's no point waiting to raise it at the other end. Cold weather? That's what heaters, heated seats, scarves and hats were invented for.