[Overdrive]

The original poster specifically said he does not track the car, nor does he drive it aggressively.

If you're just going out for Sunday drives, I can't see any general everyday issues with doing what you're considering. The tires will wear more evenly but your handling will be affected. I think the only time this may be of major concern to you is when making emergency maneuvers (swerving and such) on the road.