Test pipe vs gutted cat noise

[truegearhead]

Whew I'm going to get some heat for this question I think! I'm currently running test pipes, headers and a stainless exhaust and man is this car loud! It's not a street car but I feel bad whenever I start it up in my neighborhood, I'd like it a little quiter. With the cats on it's MUCH quieter, I'm thinking of drilling a hole through the catalytic material to free them up but hopefully retain some of the sound deadening. Has anyone ever done this? Is there a noticeable difference in volume? I'm thinking the remaining cat material would act as a glass pack.