The handle has to come off first.
I think fixating the complete handle by the hole and pulling the wooden handle grip with a hankerchief or something that gives you grip will help will help to pull it off.
Maybe heating up the wooden part of the handle will also help? Not too hot. Maybe 70-80 degrees celsius.
Here are some pictures. You can see, that the carbon part is overlapping the alcantara part.
