Individual contributions to an aggregate crowd perspective are directly influenced by the reward system underlying the contributions. In financial markets, for instance, you put your money where your mouth is, and the collective view of varying and contrary opinions arrives at the crowd-sourced pricing of securities.
Specifically, there's great potential value to being a contrarian in cases where independent actors with local knowledge have reason to behave contrary to the crowd.
If you look at the reward system for magazine rating, there doesn't appear to be any significant benefit to buck the consensus, and significant downside for the same. Effectively, this disincentives indepedent, contrary contributions.
The rating system that places Jefferson park as the countries 3rd best course, while interesting, probably isn't the alternative, either.
Interestingly, horse race betting markets are a great example of the wisdom of crowds, as the collective estimation of the odds, as derived by the pubic, has been shown to be the best predictor of actual odds of any measure.
Too much "expert" influnece on the horse racing odds to be labeled "derived by the public".
It's that kind of expert information that helps make race markets efficient and the odds remarkably predictive.
Back in the day, my passion was horse racing. I scored a copy of the betting version of "The Confidential Guide", "The Efficiency of Racetrack Betting Markets",
http://www.amazon.com/Efficiency-Racetrack-Betting-Markets-Donald/dp/9812819185/ref=pd_sim_sbs_b_1This book is kind of a "Anarchist's Cookbook" for betting, and likely just as dangerous in the right hands. The very first article, from 1948, "Odds Adjustments By American Horse-Race Bettors", by R. M. Griffith, starts:
"In horse-race betting, the odds on the various horses in any race are a functioning of the proportion of the total money bet on each and hence
are socially determined.. On the other hand, the objective probability for winners from any group of horses is given a posteriori by the percentage of winners." (Emphasis added)
The primary conclusion of the paper is that:
"The socially determined odds on horses in races are, on the average, correct reflections of the horses' chances."
My own research, on 10's of thousands of races in the early 2000's confirms this research. In fact, in trying to develop betting models using 100's of independent variables, I could only improve on the predictive value of the public odds by including the public odds in the equations. I could find no combination of values that were more predictive than what the "wisdom of crowds" came up with. (including the public odds in the models, I could improve on the predictive model by incorporating under and over weighted variables, such as beaten lengths (over) and 4f and 5f workouts (under)).
this thread is about magazine raters, and I don't think this crowd has the dynamic required. First, the selection process for those qualified if only by dint of access seems to introduce a strong selection bias. Second, the concept of local information is more consistent with horse racing, moving money in when the barn knows something, than in golf course rating, where it would seem to reflect whether individuals have access to a course, or not. However, this could also reflect local information on the strategic nature of a course, in which case this expert knowledge would have a salutory effect on the ratings.
Third, if there are real or perceived disincentives to deviate from the consensus ratings, for example by rating Pine Valley 50th and risking losing your rating priveleges, then there is a risk of some social bias in the ratings. Given the volatility shown in the 30 year Golf Magazine retrospective, at least below the top 20, this may only be an issue at the top, or it may reflect an objective reflection of the course quality.
Unfortunately, unlike horse racing, there is no objective measure of course quality by which to review the question. Access to the ballot details could possibly tease out other biases: they do some pretty fancy things with statistics these days. That's beyond my skills, though.
Instead. Maybe line up all the raters and their editors at the gates of Pine Valley, yell go, then..
"May I have 10,000 marbles, please?"