I've had a lot of experiences in golf that incline me to agree with Jason: that golf isn't that elitist in many places.
But the fact remains that to non-golfers, golf is a very handy symbol for elitism, at least in America and especially the developing world, where it's thoroughly a 1% or .1% pursuit.
That said, I do think that every year, golf becomes more and more a game for the people, because municipalities are investing in their golf courses. I'm sitting about 20, 25 minutes from Wintonbury Hills GC and Keney Park GC, two phenomenal municipal facilities in Hartford County, CT. Wintonbury was built from scratch, but Keney has been converted into a barely-used relic into a hub of recreation for people from all walks of life, at least in the context of the general pool of golfers.
It may not be these municipalities' explicit intentions, but improving the golf courses that can be accessed by those on the middle and lower tiers of golf's economic spectrum shows a greater respect for golfers who are not in the top 10%, 1%, .1%, .01%. That one of the two or three best golf courses in the county can be played for $30 or less is as strong a counter-argument to the overly simple argument that golf is monolithically elitist as I have seen in practice. More of that, and in a few decades society might be willing to look a little deeper into golf as a pursuit for the masses.
(N.B.: Of course, there is the undeniable fact that many golfers don't want it to be a pursuit for the masses. These are the people who make non-golfers so sure the game is elitist, and it's hard to argue with them there.)