Actually there are 4 retaining clips that hold the speaker grill onto the door panel. I'm thinking you probably popped one of those off, because they would pop off and fall in. The door panel clips are white plastic and don't really fall off the door.
You can see the metal retaining clips in this photo:
