Most of the "problems' with the system that are being referenced here aren't problems with the system at all; they are issues of personal integrity combined with insufficient oversight by clubs. And remember, the ultimate answer to better players who don't like losing to lesser players is simply, "Don't play in net events!"
But assuming not only players who "do the right thing" and proper oversight by clubs, here's the reality:
1. the handicap system works best for match play, either individual or two-man better ball.
2. the system works less well for larger fields, simply because somebody is going to catch lightning in a bottle and shoot an outlier score. Low index guys have less variation in their scores, higher index guys have LOTS of variation, and so the odds are that the lightning in a bottle guy is going to be a high handicapper. That's just the way it is.
3. Typically, the higher index guy who shoots the great round in competition has a couple of things going for him. He plays all his rounds by the Rules, including posting scores that include net double bogey, and a realistic "most likely" score if he picked up. In four balls, I see guys with a 15' putt for 5 pick up when their partner makes that score or better, and then take the 5, even though they haven't made a putt of that length all day. And the higher index guy who prospers in competition tends to play a lot of competitive events, and it just doesn't bother him to be under the gun.
4. There are a LOT more guys with vanity caps than there are true sandbaggers; a LOT more! And those guys NEVER prosper in competition because they don't play by the Rules and aren't as good as they think they are anyway. In my personal experience, those are the guys who do the most complaining about the system, endlessly so.