1) Folks have stated above, but golf is hard. Moreso, to be able to go out and enjoy oneself on a golf course takes a lot of time and money. I have been playing golf for 8 years, never posted a score below 83. I still lose one to two balls a round--unless I'm at Sheep Ranch
My college baseball skills translated to golf about as well as curling to basketball. What keeps me in it is the courses, the people, travel, and the occasional perfect meeting of forged steel and urethane.
2) Cost. It takes money to get sticks, get a tee time, practice, etc. Many folks spend money on skiing while on vacation and usually can get good enough in two days to say they enjoyed it and would spend that money again to ski. Can golf say the same? Two days of hitting balls justifies everything one spends to get into the game initially? I can honestly say that I would enjoy golf much less if I wasn't able to participate in the equipment obsessions, traveling to play great courses, etc.
3) Exposure to the roots and core of what makes it great. For my first three years of golfing, it was usually riding, usually with a cooler of beer, and usually involved no adherence to etiquette. Many folks learn golf during college when their athletic careers are over, and it's time to pick up an adult pursuit. I was no different. But many of my friends stopped playing because the way we played golf--see above--wasn't fun for them anymore. How many golfers would we as a golfing community retain if they were exposed to easily walkable, forgiving and fun golf courses with a knowledgeable group of regulars?
4) Time. In my opinion, the time spent on one round of golf in the US is killing our game more than any of the above reasons. People are busy and we live in a fast society. Who--outside of the core golfer--can justify the now all-day pursuit of golf in America. How abut this; 10 minutes of chipping and putting, a 3 to 3.5 hour round, one or two drinks at the grill, GO HOME!! Many spouses and family members would love this shift in paradigm.
On my honeymoon I played a round at a famous course with a great host. We hit 10-15 balls and teed off at 0800 or so. Had a candy bar between the 8th and 9th holes. Walked up the 18th at 1120. Quick lunch in the grill. My host even had time to work a half day after that. What a concept!!