Tim you are starting to sound like some of the consultants my company hires, citing Price's Law. Ranking systems in my view have inherent bias largely, but not limited to, the following:
First Mover's Advantage - Who was first into the segment? Whoever this was has a significant advantage for branding and possibly creating a status quo if powerful enough. Because the earlier golf course designers were creating courses when only a handful existed and it took a great deal of resources to build and maintain, the proponent for the courses cherished them and many were championed by wealthy influential people/clubs. Thus they have the first mover's advantage, and the status quo bias, which I'll talk about next.
Status Quo Bias - Anyone who has taken a basic psychology class has heard of SQB. It is the safe option, the established option, the way we have always done it option. Any choice other than the SQ is seen as less valuable and risky. It is mental, it is in your head. It is also why companies around the world spend millions of dollars for Change Management. We have to prepare people for the change coming and just saying this is what we want you to do is not enough, there is a science and subsequent process behind it. My company has a whole department of over 100 people just doing this, believe it or not.
In the golf rankings example Tim is pointing out, I think we can apply a strong bias as you look at the yearly rankings. It is almost like once your in, your in. It is tough to boot out a course that is designed by Mackenzie, Ross, etc. why? Because they have first movers advantage and have established themselves as the status quo for various reasons (lack of competition, got designs at the first private clubs, etc.). To get them out of the rankings will be risky and the SQB tells us that is tough to do, even if it is warranted.
Brand Premium - In investing when one company buys another for more than their tangible assets, i.e. I buy Microsoft and they have facilities, machinery, hardware worth 5 billion, but also have 25 billion worth of customer base, I.P., etc. of intangible assets that difference in accounting is called Goodwill. I like to refer to it as a brand premium, because it is easier to understand.
In golf course design intellectual property is huge, because doing the work can be done by many construction companies, but whose ideas are you going to use? The designer adds little value tangibly, but his ideas (I.P.) are what the client is paying for. If the designers in our case had First Mover's Advantage to get Status Quo Bias, then they have earned their Brand Premium (goodwill) in the market's eye. Thus, they can get more for their services.
They are then recognized on the rankings as where they are today, which is at the top of the heap. I can give you another example, College Football recruiting rankings. Is the player that wasn't ranked before he got an offer from Notre Dame, now all the sudden a top 100 player? Notre Dame is seen as a Brand Premium and if they are going to offer a scholarship to John Doe, then he must be good. It is a self fulfilling prophecy. Are all courses designed by Mackenzie great? Well some would say yes absolutely, it's a Mackenzie.
So great discussion and I love having to think critically and come up with a viewpoint.