Caster will have the opposite effect if you're going backwards. What does a shopping cart wheel do when you back up ? It flips 180°, not really what you want.....
When sideways releasing the wheel will allow it to correct, drifters do it all the time.