Correct. Pumps cavitate (the fluid stops flowing and creates voids
Article on wikipedia) when turning too quickly. Water pump, power steering pump, oil pump. Their pulleys are all designed to run at low(1000) to mid RPMS(4000). That's why when at a stop some cars forces the engine to rev a little higher when it senses that its not providing enough cooling or power steering help. Now I personally wouldn't replace it unless you are really having spiking temperatures when on the track. Most of us will get stuck in traffic more often then run 7K RPM constantly at the track. Then again its an easily replace part.
Example:
My Sr20det revs up to 8k RPM and I was chasing some power-steering and cooling gremlins. Past 7k for too long and I was running into some cooling problems. I under-drove the power steering and water pump and it mostly helped the cooling side. But its more of a band-aid. Finally installed some power steering cooling lines with a filter and that was able to keep the viscosity in check and help deaerate the oil.
Back to the power. The pulley doesn't have enough significant mass to provide any substantial effect on the inertial of the system. Actually smaller pulleys would make it harder for the engine at lower RPMs to turn accessories by the loss of mechanical adavantage.(lever arm)