This kind of gets into the "every day course" vs the "destination course." People need an out of the ordinary course to travel long distances to play it. But, they still want interest and don' like getting beaten up too often. If a course is too tough, I think they play it once to add it to their list and then look for other new experiences. I think there is a magic blend in there - like the Bandon courses - that are tough, but actually more interesting than tough. If they are relentless, then its because of the wind conditions that day, which I think golfers understand.
JKM - I think I answered your question. The once in a lifetime experience would probably factor in history, I think. Even then, I think many wouldn't join a club as tough as Oakmont or PV. Mot golfers want fun, not real tournment challenges.
I think most of us architecture buffs forget how much convenience, expense, and speed of play influence the vast majority of golf round purchases.
Pete Dye once attributed the success of Pinehurst to the fact that you couldn't lose a golf ball there and he is right. More than difficulty, I think adding the extra expense of a dozen balls to a round is a killer, moreso than three putting 18 greens.
There is that old story about some female members telling the gca that the ninth hole - a par 3 witha frontal bunker they couldn't carry and was too deep to get out of - ended their round. The gca said if he had known that, he would have done it on the first hole. Not PC, of course, and not good gca. But, it goes to show one unattainable forced carry is enough to stop people from playing a course if they can't make it.