I see those little teeny tiny sparkles (when the light hits the oil just right) when I change oil on all of my cars and have never worried because there are only two options: (1) tear down the engine to find the source.; (2) do nothing.
It seems like it would be hard to justify tearing down an engine because of these microscopic metal particles, but its your car and your budget so you will have to make whatever decision you feel is right.
If you're really worried, have the oil tested and then you'll know exactly what is in the oil.
For reference, this is what a real problem looks like:
