Would rankings be more credible with only one rater?

[Jerry Kluger]

It used to be explained that the reason for rating panels was the large number of courses opening every year which is no longer the case.  So let's say the magazines each chose one person to rate courses for them - Golf Digest: Whitten; Golfweek: Klein Golf: Morrissett. It could easily be argued that the rankings by each of them would be far more credible than some group which does not have anywhere the credentials of those three.   

[Garland Bayley]

The rankings would be worse due to the one-sided view.

[Tom_Doak]

Jerry:

One individual can't really do the rankings, because no one person has seen ALL the contenders.  Plus, what Machiavelli said.

However, it would be immensely helpful to most people to be able to look at the independent views of people whose opinions you respect, as opposed to a "consensus" of people you don't know and may not trust.  That was part of the success of The Confidential Guide.  I've been surprised there aren't more, similar books out there for just that reason.

[Greg Tallman]

As a facility that sees its fair share of ratings visits, panel summits.... etc. I would have to say that GOLF Magazine is as close to having it right as is possible.

-A pretty diverse group of panelists, most of whom have played or seen a majority of the courses up for consideration
-A relatively undefined system for identifying what belongs and what does not (not "checking the boxes" as Tom says)
-A nice mix of old and young
-A manageable number between whom comments and reviews are easily shared and digestable
-A ratings chairman that listens to "the right people" when considering tweaks (or nixing his own ideas for tweaking)

and last but not least... to a person those that I have met are wonderful people who I have been lucky to get to know over the years.

I am not going to get into the other magazines and what may be right and may be wrong but just wanted to throw that out there for GOLF.

[Jerry Kluger]

Tom:

I think that one rater could see most of the new courses and in the interests of candor and full disclosure, that person would list the courses that he had been to each year and could even highlight those which he had not yet seen.

I think your point about writing a book brings to mind an advantage of a panel as those in charge can blame the panel if anyone complains about a ranking.   When you wrote your book you weren't in a position where there would be ramifications beyond personal attacks against you.

I would suggest that Ran's reviews, including his photos, could make a wonderful book.  Now if I could only convince him to include photos and reviews of courses that he didn't like.  

[Peter Pallotta]

Jerry - I'm with you; if there are to be rankings, I think a very very small number of people - their names made public -- is the way to go, if for no other reason than that the subjective nature of the golfing experience would thus actually be recognized and celebrated instead of being buried/hidden away in a pile of numbers and statistics.  

I think that's the main trouble with the current ranking system, i.e. no matter what people say in defence of the GD or Golf or Golfweek systems, the rankings foster the (implicit) belief that 'math' leads to 'objectivity' i.e. that statistics don't lie -- that if you get enough different people filling in enough ballots in enough categories you will get something approaching fact/truth/objectivity.   

I'd rather see a small group of prominent and/or well travelled people with strong likes and dislikes make an open assessement (including a paragraph or two describing their tastes/philosophies) of the courses they think best.  Why not Matt Ward, Ran, Ron Whitten, Ian Andrew, and Greg Tallman for example?  Five people, five Top one hundred lists...

Peter

[Jerry Kluger]

Peter: I would like to see the raters as individuals who have written on the subject so that the reader can understand something about their views with respect to gca.

[Adrian_Stiff]

No. I think you need as many as possible to conclude a set of rankings but you need to give each ranker a sheet with strict criteria how they should mark. Provided that each scores reasonably fairly and you have 20 ranks for each course then it can be done reasonably but you have to out the crazy rater or his *ranks* will skew the system.

[Jerry Kluger]

Adrian: Your statement points out the reason why one person is better - if you get a few bad raters you have to throw out their scores - but who determines who is a good rater and who is a bad rater - is it determined solely by the fact that their score is different from the others - but if that is the case and most of the raters are going to come up with the same score why bother - the answer is to just have one really knowledgeable person do the ratings.

[JMEvensky]

Peter: I would like to see the raters as individuals who have written on the subject so that the reader can understand something about their views with respect to gca.

I'm the same.

Just give me a body of past work so I can see a particular rater's likes/dislikes and I can figure out the rest.To me,it's no different than a book or movie reviewer.

[Jim_Kennedy]

http://www.golfweek.com/news/2010/nov/30/raters-notebook-st-georges-gcc/?RatersNotebook
This feature is probably the distilled version of what actually gets jotted down when rating a course, but it adds flavor to the numbers.

I don't believe that the rankings are done willy-nilly by a bunch of incompetents, and I don't think that anyone goes out into the field to rate that doesn't have an understanding of the system they're using.
Having a reasonably large cadre of them is the only way to ensure that a broad number of semi-objective opinions can be assembled that will then be used to come up with a country-wide picture of what sits where in the rankings.
If there were only a handful of people doing it, and even if that handful were the most knowlegeable critics of architecture, their results couldn't encompass enough variety of thought to be seen as complete. Such a small number of 'raters' would probably have little trouble getting the whole numbers right, like a 7 or an 8 or a 9, but the fewer numbers that a limited group would produce can't give as precise of a figure as a much larger group when it comes to adding up the tenths and the hundredths, which are fairly crucial when the call is a close one.  

[Adrian_Stiff]

Adrian: Your statement points out the reason why one person is better - if you get a few bad raters you have to throw out their scores - but who determines who is a good rater and who is a bad rater - is it determined solely by the fact that their score is different from the others - but if that is the case and most of the raters are going to come up with the same score why bother - the answer is to just have one really knowledgeable person do the ratings.

Jerry - I think it would concentrate a rater to score properly, knowing if he was miles out of line with another his vote would be axed.
The idea is to collate good opinion not bad opinion, you are correct when you say "who decides what is good and what is bad" but when you look at the www.top100 site you see good opinion and you quite clearly see silly opinion, for instance the reviewer might give 1/10 because the day he played the course was undergoing maintenance. I have seen 10/10s for courses that would not get a doak 1. The problem you have with a solitary rater is no one has played every course in the world and even those that have played most of the top 500 could not have 'recent knowledge' so I think you have to find a multi route, what it needs is some form of document that can be scored as objectively as possible and perhaps the subtle aspects leave a margin for opinion.
I think if the 20 raters came up with the same score that would be good, it pretty much emphasises a true number of points for that course and if each course had an amount of points there could be a great way of ranking the top 10,000 if needed as the info could be imported into an easy program to sort.

[Matt_Ward]

Jerry:

Interesting proposition but a logistical impossibility unless the person were like the character George Clooney played in, "Up in the Air."

It's not about just making first time visits but staying up on what is happening.

A person could handle a specific area -- say the Northeast but would really need to stay current given the range and sheer qualities of the courses there.

One can knock down the army of raters that GD uses -- likely you could get an august body of something like 50 people.

The issue also presents itself that such people play more than just the elite of elites. Plenty of people who rate now don't even sample the obscure and public courses that are very good in certain instances.

Jerry, in today's viral world -- the real issue is what value the mags present anyway. I have my sources and many of these folks rely upon me for my comments .

[jeffwarne]

Tom:

I think that one rater could see most of the new courses and in the interests of candor and full disclosure, that person would list the courses that he had been to each year and could even highlight those which he had not yet seen.

I think your point about writing a book brings to mind an advantage of a panel as those in charge can blame the panel if anyone complains about a ranking.   When you wrote your book you weren't in a position where there would be ramifications beyond personal attacks against you.

I would suggest that Ran's reviews, including his photos, could make a wonderful book.  Now if I could only convince him to include photos and reviews of courses that he didn't like.  

Sure one rater could see all the new courses in this economy.
But that would leave no room for courses that have dramatically improved or gotten worse.
How relevant would a rating of Atlantic from 10 years ago be?
or evn oakmont 15 years ago
or erin Hills?
How about 100 raters each assigned to play 15-20 courses every year on a rotating absis.
that way they see all top 100 and contenders, and courses only have to entertain 1 rater annually from the respective magazine.

another set of 100 junior raters could play and nominate other candidates for the top 100 raters to visit next year.

[JMEvensky]

Jim Kennedy,you're probably right that a group would be better able to work out the decimal point distinctions.But maybe the whole exercise of deciding whether Course A is .001 better than Course B is the problem.

[Mike Benham]

... and many of these folks rely upon me for my comments .

Jerry -

Matt IS the answer to your question ...

[David Harshbarger]

Would anyone be interested in working with me to develop a Golf Course Architecture rating system? I'm thinking something along the lines of a tool or app you could take to the course to catalog the architectural features of each hole.  The feautures good and bad would roll up to a hole score, that would then roll up to a course score.

The results and catalog would be published on the web.  Magazines are dead, anyway 

[Jim_Kennedy]

JME,

I think it can be likened to a player's handicap, where a 21.10 index and a 21.42 will both play as 22s even though the 21.10 is the 'better' player numerically, and I'll bet the 21.10 is aware of that fact.  

[Adrian_Stiff]

Would anyone be interested in working with me to develop a Golf Course Architecture rating system? I'm thinking something along the lines of a tool or app you could take to the course to catalog the architectural features of each hole.  The feautures good and bad would roll up to a hole score, that would then roll up to a course score.

The results and catalog would be published on the web.  Magazines are dead, anyway 

Yes I would. I think collectively on here we need to come up with a method to score. I suspect the methodlogy would cause some oblique discussion in itself.

[David Harshbarger]

I suspect the methodlogy would cause some oblique discussion in itself.

You think?

Thanks for the offer, I'll put some notes down and put them out, as soon as I get my teflon suit on.

[Sean_A]

Adrian

Actually, if you are looking for good opinion and to weed out bad opinion, one person or a very small panel is the way to go as its much easier to control quality with a small panel.  If you want all and sundry to see if any consensus is reached, then a larger panel is good, but only if all segments of abilities etc are well represented.  Unfortunately, with the larger panel mags, they list courses which don't achieve a consensus - say rated highly ( a criteria set by the folks in charge) by at least 75% of the minimum number of raters (34 out of 45 in GD's case) or something like that.  Instead, they shoot for an even number rather than a list of true notables followed by a list of honourable mentions that may someday make the list proper.  

Pietro

As I state above, the goal of a large panel shouldn't be some sort of objective truth through numbers - it should be reaching a consensus.   Shit, I don't ever use any set list of criteria as each course is different and will somehow not fit the criteria exactly.  

In any case, I much prefer grouping/a star system where the reader can essentially draw his own conclusions within certain parameters rather than a dogfight over numbers.

Ciao

[JMEvensky]

JME,

I think it can be likened to a player's handicap, where a 21.10 index and a 21.42 will both play as 22s even though the 21.10 is the 'better' player numerically, and I'll bet the 21.10 is aware of that fact.  

True.But,the "players" at 0 or better don't even acknowledge the 22's whether they're at 21.10 or 21.42.

The sad truth is that the same people who'd argue over .01 of a handicap would argue over .01 of a course rating.

[Jim Nugent]

- most of whom have played or seen a majority of the courses up for consideration

This seems to me essential.  I wonder how many courses, on average, the GD panelists have seen? 

[Jerry Kluger]

Jim: I don't believe that the number of raters really gives any more credibility to the overall rating. I would say that the vast majority of regular GW readers would give more credence to a rating done by Brad than a rating by some anonymous group of raters. 

I would also bet that the vast majority of members of GCA would give far more credence to a rating given by Ran than any group of raters.

[Jim_Kennedy]

Jerry,
As I said in my first post, a handful of the most knowledgeable critics are enough to sort out the whole numbers. I'd 'trust' a Klein or a Doak or a Whitten before any group of anonymous raters, but those three fellows probably don't have enough time to see a few hundred courses every year.

It takes a large block of numbers to fairly make the hair-splitting decisions between a 9.56 and a 9.28 or between a 6.86 and a 6.82, and you need a large group to amass them.......and as that is the only option to date, the larger number's semi-objective opinions are the most credible yardstick when trying to arrive at the .1s and the .10s. that provide the necessary minor distinctions to create a list in the first place.