Lou,
We don't shrink it, we transfer some of it, and grow it as a subset of golfers going to Top Golf or similar golf substitutes, or as it will be come to be known, "versions."
Think about it - Precedence - Japan. Convenience - takes away the negatives of time, investment, too much land, water, etc. (society views, not mine!) and leaves the essence of the game - hitting the ball well, friendly competition via computer chips in the balls measuring results, etc. It eliminates intimidation. It removes carts in favor of lounge chairs between shots! And adds in upscale food and beverage, reduces embarrassment, etc.
It is amazing how popular Top Golf is with non traditional golfers - last time I was there, a group of six Indian women in Sari's was there, sans husbands. I don't know if it was a coupon deal or what, but they were hooping and hollering as much as anyone. Also saw many groups of young single women, African American teenagers, etc. For them, it also removes the intimidation factor of golf.
Instead of shrinking or growing, I think some out of the box thinking like Top Golf, which in many ways is an update of "date night" at the driving range to rearrange golf for modern times is a more likely occurance.