My cupholder "broke" on me and so I had to remove it too. The thing with the slim cup holder on 03-04 is that you not only need to remove the horse shoe, radio, and ac, but the damn frame that goes around those and remove the ac vents above. Look at this diagram:
Accessories Dashboard Trim 03-
See how 12 (the cupholder) is stuck behind 13 (the support frame)? So to remove the support frame, you have to remove all the things i've mentioned above. It $!@#! sucks, I know.
As for my cupholder being "broken", it wouldn't close, as if some plastic part or piece was stuck in it. I had ordered a new one, did all of this work to get it out, and then I flipped it over which loosened whatever was stuck and it would close properly again! And that's was my first foray into brilliant German engineering and learning not to buy parts until doing a proper diagnosis. Good luck with the removal!