Mike,
I think it would take much more than 'tweaks of a statistical nature' to fix the system.
For example, throwing out outliers most likely wont help and might hurt. I just cant imagine the sample sizes are big enough to justify throwing away the outliers. Doing so anyway reaks of manipulation. Plus, it may well be that the better raters sometimes have the outliers, and it seems a shame to throw away their scores.
Standard deviations might give off the aura of science, but unless we are dealing with data based on a system of similar valuation, the numbers are still worthless. Plus, my guess is that if we went to standard deviations, we'd find that we have hundreds of courses which are statistically inseperable.
What's the message to your readers when courses with a tight range of 5s and 6s (or 6s and 7s-- I dont know your scale) makes the list.
"No one had strong feelings about it one way or another. Never mind that noone loved it. Noone disliked it therefore it is one of the top courses in the US." Hardly a ringing endorsement and hardly aimed at identifying anything other than the courses at the high end of the mediocrity scale.
As far as the course with the 8s and 2s, a portion of the readers would probably greatly benefit from knowing that some of your raters absolutely loved the course. Likewise, other of your raters might benefit by knowing that others hated it. That is, if the likes and dislikes were explained. But to average the numbers renders everyone's rating meaningless, and especially neutralizes those that on the high end (the low enders have kept the course off the list, which is what they wanted.)
By definition, the latter course is more controversial on some level, probably takes more chances, is probably more inherently different architecturally than the norm, and someone playing there might love it or hate it. How would you suggest that type of thing gets factored in?
As I suggest above, fire almost all the raters, and have those that still rate explain themselves.
Raters Outings: I too think they are a good idea, except that they give the outing locals an unfair advantage over the rest. As I suggested earlier, have them at well-established courses which are unlikely to be hurt or helped by the rating. The raters could benefit without skewing the scores in favor of the courses which hold the events.
Regarding Bandon, I understand that a whole bunch of raters must have liked the course better than you. You looked very carefully at the course, and you
know it isnt the third best modern course in the country, and I'll take your opinion over their opinion. As you say, the ratings will only be as good as the raters, and here we have a good example of where the raters are perhaps not quite as good as they might be.
Who is to say that you are right and they are wrong? Let me put it this way, if they are 'right' and you are 'wrong' then there is something wrong with the ratings criteria. Call me elitist or biased or whatever you want, but there is something wrong with a system which calls Pumkin Ridge Ghost an elite course, and which cannot distinguish the difference in quality between Friar's Head and Bandon Dunes. (Dont forget, I like Bandon quite a lot, but come on!)
___________________
Tom, how many times are you going to post: 'well, the system isnt going to change?' How many times are you going to ask me how I would improve the system, only to dismiss the suggested changes as impossible? How many times are you going to ask me to 'flesh something out' only to later agree with it while dismissing it yet again?
Give me a break. I've said what I would do. If the magazines wont take my advice, that it certainly their perogative.
As for people preferring BD over FH, you make my point for me. The ratings dont work. They dont identify better golf courses over worse.