Elitist and exclusive are different things. Smugness is another thing. The first two are fine, if handled properly. The latter is not. From your descriptions, I think it's the latter you detest, and I agree.
Nothing wrong with striving to be better, to be elite. Some elite things are exclusive (Augusta National), some not (anybody with the scratch can tee it up at Pebble Beach or walk into Tiffany & Co. and buy a bauble).
Exclusivity can be bought the way a round at Pebble Beach can, but it's not necessarily elite. Bushwood CC, say.
To answer the question, no, elitism has not ruined golf. The game is everywhere. But smugness, which flows through the bloodstreams of too many in the game, has done nothing to help the game's image among those who don't know better.