[Anker]

Bad toe-in alignment on either front or rear will give you exactly this feeling. As weight shifts from one side to another the car will have a tendency to drift towards the high load side. The reason why it feels worse at high speed is because the drift will be more severe as the speed increases.

Another possible cause, which I am sure you don't have is severe loss of tire pressure in the rear. The first indication of a punctured rear wheel is loss of steering stability. I was once a passenger in a Mercedes running at very high speed on a highway when one of the rear wheels had a puncture. The driver was barely able to control the car. After we changed the tire he then proceeded to fall asleep at the wheel. Scary trip.