Tom
Your question seemingly could inspire answers in multiple directions and has, thus far, only inspired one; an individuals competitive needs requirements. And that is something sure to vary with each response.
I think looking from the commerce and capitalism perspective, competition is the lifeblood of the golf industry. As it is for any industry I would say. Without the desire of the lesser player to beat the more accomplished player there is no demand for innovative equipment. That is not to say we would still be using sticks and stones in open fields, but we most certainly would not have the game, and its ancillary effects, we have today.
My individual feelings about competition driving my interest in golf is a stumper right now, as you can probably imagine. I grew up playing plenty of competition and any success I had probably played a large part in my enjoyment from day to day or week to week, but as I look back today as a 30 year-old amateur awaiting reinstatement my memories of golf as a young kid (12, 14, 16 years old) I am by myself playing 9 holes as the sun is going down and I am simply playing for the enjoyment of the shots and the environment. Was there some internal competitiveness to play my best? Maybe, I certainly wanted to, and took pride in hitting good shots or shoot a good score, but that would be comparable to my desire these days to cook a meal for my wife and have it come off well. I am certainly trying to do my best but there are no consequences for failure. To me, that is the heart of competition, a self imposed consequence for not succeeding.
When I play golf today I always have a match of some sort, I simply love the action, but the minute I have the time to do so I will be back out there as the sun is setting enjoying the game for itself.
You ask in your first post I believe:
"So is golf completely reliant on competition and its ever increasing rules and conventions and definitions?"
It all depends on what you consider golf to be. If, as I'll assume, you see golf as does Max Behr in the second paragraph below:
".....and the psychological tendency is to make further inroads upon nature's side of the balance, for once the human mind succeeds in overcoming natural hazards in life it never remains satisfied until it has devised a means to do away with them altogether.
I do not mean to imply that it is possible to return to those halcyon days of golf, or that it is even desirable; but what I do wish to emphasize is that unless we keep before us a true perspective of golf, a viewing of it always from its natural side, it will eventually degenerate to a known quantity, a true game, and will become robbed of those elements of mystery and uncertainty which make every round a voyage of discovery.
The fate of golf would seem to lie in the hands of the Royal and Ancient Golf Club and the United States Golf Association. Can we expect that they will protect and reverence that spirit of golf?"
Max Behr, 1922
I would say competition (in the commerce and capitalism sense) is the greatest threat to golf. The driving force behind all innovation is to make things easier in some way or another and that directly results in an effort to smooth out all of nature's "elements of mystery and uncertainty". It is this quality that Behr feels should be a lasting element of the game. I would have a hard time arguing.