Jim, Tom, and everyone else:
You've heard me make this point several times in the past. Having people give cardinal ratings to courses and then averaging those ratings is absolutely the wrong way to generate a ranking. Nobody thinks Ellerston belongs in the top 10, but they can't articulate why (except that not enough people have played it). Just look at the one person who publicly rated Ellerston (not to criticize that person): He gave 3 stars to 3 courses, which all ended up in the top 10. He have 2 stars to Ellerston, another top 10 course, and another course outside the top 10. Out of the other courses that he rated, none of them made the top 15. This rater didn't rate Ellerston ahead of a single other top-15 course. That's not to say anything negative about the rater, but the system is clearly flawed if this vote can launch Ellerston into the the top 10.
The right was to rank these courses, as I've argued here before, is through head-to-head matchups. Every time a person rates a course, it's as if they're casting a head-to-head vote between that course and all the other courses they've rated. If any single course would have beaten all other courses in a head-to-head vote, then that course should be number 1. After removing that course from the game, then you determine number 2 in the same way, etc.
The advantages of this system are many:
For one, if those who happen to have played Ellerston are particularly generous raters, then the traditional system will over-rate Ellerston, while the head-to-head system would not. In this sense, the head-to-head system accounts for the fact that some raters are more generous than others.
Also, under this system, raters have no incentive to misrepresent their true preferences. For example, under this system, Jim could have given Ballyneal 3 stars and given 0 stars to Sand Hills, Pacific Dunes, and Barnbougle Dunes, falsely inflating Ballyneal’s position. However, under the head-to-head system, this strategy would have done no good. This strategy would only make it more likely that other courses beat SH, PD, and BD without increasing the chances that Ballyneal would beat those courses (assuming that Jim really likes Ballyneal best!). In my own case, I’ve played 5 of the top 10 and I prefer Kingsley Club to all of them. I might have given falsely low ratings to those 5 courses and launched Kinglsey into the top 10, but I didn’t.
The only disadvantage of the head-to-head system is that it will not necessarily produce a clear ranking. It’s possible that Sand Hills beats Pacific Dunes, and Pacific beats Barnbougle, and Barnbougle beats Sand Hills. However, in that case, we could easily just go back to Jim’s system of averaging the cardinal ratings (or we could just all it a tie).
I am confident that abnormalities like Ellerston ranked #6 would not be present under this improved ranking system. Obviously, it’s a problem for any system when only 3 people have played a course, but the head-to-head system provides a much better way to deal with this issue.
Jim, I would be happy to look at the data and report back on the head-to-head ranking.
Anthony