Nope, you doing the right thing. Yes, shaving and clipping affect the hair the same way - you are cutting the thickest part of the hair and the blunt ends will always feel like stumble. When it grows back, it will be the same old hair structure whether you cut it 500 times or 5,000 times. It will not get worse.
Dee