Would rankings be more credible with only one rater?

[Tom_Doak]

Having a reasonably large cadre of them is the only way to ensure that a broad number of semi-objective opinions can be assembled that will then be used to come up with a country-wide picture of what sits where in the rankings.
If there were only a handful of people doing it, and even if that handful were the most knowlegeable critics of architecture, their results couldn't encompass enough variety of thought to be seen as complete. Such a small number of 'raters' would probably have little trouble getting the whole numbers right, like a 7 or an 8 or a 9, but the fewer numbers that a limited group would produce can't give as precise of a figure as a much larger group when it comes to adding up the tenths and the hundredths, which are fairly crucial when the call is a close one.  

My emphasis added.

This is the main reason rankings are crazy ... because some people actually believe they are "semi-objective".  EVERYTHING IN GOLF COURSE ARCHITECTURE IS SUBJECTIVE.

[Steve_ Shaffer]

My thought has always been to let the raters rate but have an overseer- Whitten, Klein,etc- to rate the top 5 or so of the panel and rerate if necessary.

[Adrian_Stiff]

This is the main reason rankings are crazy ... because some people actually believe they are "semi-objective".  EVERYTHING IN GOLF COURSE ARCHITECTURE IS SUBJECTIVE.
[/quote]Tom - I disagree with that, I think there are some definite traits you will find in greatness, though I accept that by definition an opinion cant be objective. Taking my point to an extreme a golf course will 18 holes ticks the box and one with lesser holes is not so good, I think a great golf course needs to be 7000 yards (par 72) and I think you wont agree. To qualify that statement 6750 par 71, 6500 par 70 would equally fit, A course 6700 yards par 72 I would deem too short and it would lose points if it was 6300 yards par 72 it would lose more points. If you look at the worlds best 100 this is a common theme. I think you can have great architecture in golf courses that dont tick every box, but certain components missing stop a course from greatness, perhaps crossing holes, areas too tight, blindness, too easy or too difficult approaches could stop the greatness those aspects tend to lean more towards subjective, but there are some guidelines we all work too that are subjective and I think elements of Prestwick as a golf course are perhaps top 5 in the UK but it 'looses points' to say top 40 because it's a touch too short and its a bit too tight and its a lot too quirky. I could see an opinion liking the 5th and another hating it and perhaps its a good thing that way but I think there are fundamentals that would be semi-objective.

[David Harshbarger]

Jim: I don't believe that the number of raters really gives any more credibility to the overall rating. I would say that the vast majority of regular GW readers would give more credence to a rating done by Brad than a rating by some anonymous group of raters. 

I would also bet that the vast majority of members of GCA would give far more credence to a rating given by Ran than any group of raters.

Two GCA members were nice enough to invite me to play at Rustic Canyon and they too suggested taking Ran's ratings.  Too bad he doesn't scale.

[Mac Plumart]

Would rankings be more credible with only one rater?

Of course they would be, assuming that rater is you. 

[Andy Troeger]

Mac,
Ding Ding! We have a winner! My own list fits my preferences and likes far better than anything the big names mentioned in this thread are going to come up with, and every one of us could say the same thing. The fun thing about the lists is being able to see whether you like specific courses more or less than a consensus of peers.  


Adrian,
I think you prove Tom's point about everything being subjective. How can it be objective if we can't even agree on the objective criteria?

[Brad Isaacs]

Andy you beat me to the comment. Of course Mac beat me to the comment as well.

In medicine, the difference between subjective and objective is in the measurability of the description, not the validity of either. An example of that is an eyeball evaluation of an echo of the ejection fraction of the heart vs an objective measurement of the same echo. An experienced echocardiographer is just as good as the objective measurement and much faster.

Another example is pornography.... What is it? We know it when we see it.

I think a subjective evaluation has it's value and especially in one who has studied the subject and is experienced. Do a good job of defining what good arcitecture is and you will be able to quantify it. It is just as valid to subjectively measure the quality of the course.

One still has to be committed to always improving your knowledge and abilities.

[Mike Nuzzo]

It would be credible if you had the same tastes.

[Mike_Young]

where has anyone ever said ratings were credible?  As TD says...it's all subjective....I would use the same system Sports Illustrated uses to rate the swimsuit models....and then be done with it....cause there are still plenty of good looking swim suit women out there that were not rated and the same goes for golf courses....all you really need is one good one....

[Jim_Kennedy]

A rater has a set of categories for which he collects data. I don’t know how precise or complex the criteria are, but the categories are not subjective, they have headings and definitions. This is what I meant when I said semi-objective, it’s not just one person’s mind subjectively collecting his own set of data. The game plan has been pre-ordained by someone else.

But doesn’t rating take the rater’s subjective experiences/observations, throw them in with all the other rater observations, and distill the mash into a consensus of opinion and assign it a number? If so, once you have the consensus and the number it becomes an objective ‘fact’, and will remain as such until some other subjective interpretations which use the same pre-ordained criteria come along and create a new set of objective facts to replace it, like the 2012 lists.

[Brad Isaacs]

Subjective is not a bad word.

[Tom_Doak]

But doesn’t rating take the rater’s subjective experiences/observations, throw them in with all the other rater observations, and distill the mash into a consensus of opinion and assign it a number? If so, once you have the consensus and the number it becomes an objective ‘fact’, and will remain as such until some other subjective interpretations which use the same pre-ordained criteria come along and create a new set of objective facts to replace it, like the 2012 lists.

You just keep digging the hole deeper, Jim.  The sum total of a bunch of subjective opinions is not an objective opinion, whatever b.s. the magazines throw your way.

[Matt_Ward]

The issue for many of the mags is can they somehow be relevant in this process.

I don't see how the herd mentality which comes from a consensus-derived approach works so well.

No doubt the feasibility in having just one person rate them all is not possible logistically. That person would never stop moving in order to even attempt such a thing.

I'd like to see the mags list a few people and have them spell out their top courses. Clearly, no system is foolproof or a guarantee of clear and certain outcomes. Clearly, the closer one can come to a given area - such as a state it may be possible to have far less people involved -- and possibly even have just one for those states which are not too large to handle.

[Adam Lawrence]

I've written about this on here before, but if you believe everything in GCA is subjective then you are accepting that person A's view that Dogtrack Muni is a better course than Cypress Point has validity.

I don't believe that. Meaningful judgements of quality can be made in spheres that ostensibly seem subjective; they are in literature, music, art and so on. They are made by combining the considered opinions of qualified individuals to produce a canon. The canon doesn't have the status of fact; it just represents the best assessment that is available to us at the moment, and it's subject to refinement and even overturning if a future voice convinces informed opinion of his case.

This is pretty standard critical theory, and I don't see why it shouldn't apply to assessing golf courses.

On another theme, I don't know, because I haven't inquired, how any of the major magazine rankings are calculated from the piles of individual ballots. But mitigating the issues of regional bias, small samples and so on is not hard to do, using fairly basic statistical methods. You can give each voter a weighting, based on criteria such as the total number of ballots submitted, the geographical spread of those ballots, etc. You can depreciate the value of a ballot over several years, before finally eliminating it. The magazines may be doing this; I don't know.

[Dónal Ó Ceallaigh]

I've written about this on here before, but if you believe everything in GCA is subjective then you are accepting that person A's view that Dogtrack Muni is a better course than Cypress Point has validity.

Adam,

Person A's view may not have credibility (which is also objective and subjective), but one can't say that person A's view isn't valid, as it is the true opinion of person A.

It's all opinion whatever way you look at it, and I have long felt that ranking courses is an exercise in futility. I don't understand why it generates so much discussion on this board.

[Adrian_Stiff]

The objective aspect of rating a golf course comes from much more simpler fact:

a 4800 yard course is worse than a 6800 yard course, or by definition the 4800 yarder has a worse outset start position
b an 18 hole golf course is better than 9
c the course has no irrigation  (can be sub divided into greens, tees, fairways)
d the course has no practice ground (can be sub divided into practice greens, short game area, limited size of range)
e the course has returning nines

the list could go on, of + and - minus points

you could add points to a course if it has held, Majors, PGA, Amateur champs, County, Regional, etc..it is just a case of finding the correct formula. The above points you might disagree with but they are found at better courses
 

[Sean_A]

The objective aspect of rating a golf course comes from much more simpler fact:

a 4800 yard course is worse than a 6800 yard course, or by definition the 4800 yarder has a worse outset start position
b an 18 hole golf course is better than 9
c the course has no irrigation  (can be sub divided into greens, tees, fairways)
d the course has no practice ground (can be sub divided into practice greens, short game area, limited size of range)
e the course has returning nines

the list could go on, of + and - minus points

you could add points to a course if it has held, Majors, PGA, Amateur champs, County, Regional, etc..it is just a case of finding the correct formula. The above points you might disagree with but they are found at better courses
 

Adrian

I don't understand how the above criteria are objective. 

Ciao

[Adrian_Stiff]

Because those aspects are facts. A fact is objective. Whether you think that is important/not important becomes an opinion which is subjective. Some may be only subtle points and so deterime a small part in the formula.

I actually produced a GB & Ire rating using objective detail and collating the opinions of the magazines, but you wont like what I used as objective detail.... the results were very close to the Golf World ones and there were no real sillys.

[Kevin Lynch]

Jim: I don't believe that the number of raters really gives any more credibility to the overall rating. I would say that the vast majority of regular GW readers would give more credence to a rating done by Brad than a rating by some anonymous group of raters. 

I would also bet that the vast majority of members of GCA would give far more credence to a rating given by Ran than any group of raters.

Two GCA members were nice enough to invite me to play at Rustic Canyon and they too suggested taking Ran's ratings.  Too bad he doesn't scale.

David,

I had almost the exact opposite sentiment.  Thank God Ran doesn’t scale his courses.

If Ran put a number or ranking on his courses, I think people would spend too much time worrying about the comparisons, rather than absorb the wonderful commentary about each course.

I’m always looking for more commentary about a course, rather than some number (which is subject to “grading on a curve” issues).  From commentary, I can get a sense of which characteristics a particular reviewer enjoys or dislikes.  Then I can decide if those things same things matter as much to me.

Relative to the overall theme of this thread, I think it is true that we lend more credibility to reviews by someone whose tastes we understand, rather than some sterile “vacuum” of numbers.  Also, through this website, we discover people whose tastes seem to overlap ours.  That certainly doesn’t occur through numbers, but through a collective sense of their commentary.  If JNC Lyon suggests a course to me, I’m fairly confident that I’ll enjoy it.

I saw your idea on a different thread of cataloging holes at courses.  I thought that idea was good, in that it would give commentary on the holes.  However, less important to me would be the idea of “scoring it” based on how many boxes were filled out from a “technical elements checklist.”

[John Kavanaugh]

If you can't read this site yourself and figure out which courses are preferred by its members than you are better off not having an opinion of your own.  A road is never finished until we put a stripe right down the middle for people like you.

[JC Jones]

I've written about this on here before, but if you believe everything in GCA is subjective then you are accepting that person A's view that Dogtrack Muni is a better course than Cypress Point has validity.

I don't believe that. Meaningful judgements of quality can be made in spheres that ostensibly seem subjective; they are in literature, music, art and so on. They are made by combining the considered opinions of qualified individuals to produce a canon. The canon doesn't have the status of fact; it just represents the best assessment that is available to us at the moment, and it's subject to refinement and even overturning if a future voice convinces informed opinion of his case.

This is pretty standard critical theory, and I don't see why it shouldn't apply to assessing golf courses.

On another theme, I don't know, because I haven't inquired, how any of the major magazine rankings are calculated from the piles of individual ballots. But mitigating the issues of regional bias, small samples and so on is not hard to do, using fairly basic statistical methods. You can give each voter a weighting, based on criteria such as the total number of ballots submitted, the geographical spread of those ballots, etc. You can depreciate the value of a ballot over several years, before finally eliminating it. The magazines may be doing this; I don't know.

This is spot on and the first sentence highlights why THE BIG WORLD THEORY is garbage. 

The notion that it is all subjective and everyone has a "true opinion" as to themselves is nothing other than relativism invading the world of golf course architecture.

Sure, someone can say they "like" the dogtrack muni more than Cypress Point but that does not mean that the dogtrack muni is architecturally superior to Cypress Point.

I do think, however, that where preference plays a role is when one must choose among the great.  That is akin to entering the pantheon and pointing out your favorite whereas the other is saying that all courses can have representation in the pantheon.

[Sean_A]

Because those aspects are facts. A fact is objective. Whether you think that is important/not important becomes an opinion which is subjective. Some may be only subtle points and so deterime a small part in the formula.

I actually produced a GB & Ire rating using objective detail and collating the opinions of the magazines, but you wont like what I used as objective detail.... the results were very close to the Golf World ones and there were no real sillys.

Adrian

You must not be explaining yourself well.  What aspects are facts? 

I will admit that I am resistant to specific criteria in evaluating a course because I honestly believe that evaluation can't be a cookie cutter exercise.  Just looking at your five throw out criteria, I wouldn't accept any as a given objective fact, but then I am not sure that is what you are saying.  Well I hope not anyway or its time for you to hit the books and learn what objective and fact mean.

Ciao

Ciao

[Adrian_Stiff]

Sean -

A course with 18 holes is better than a course with 9 holes.

A course measuring 6800 yards is better than one of 4800 yards.

Do you dispute?

[Carl Rogers]

I would suggest to the golf world to try a very small committee (maximum of 4) and engage in more of a "Siskel & Ebert" dialogue and process with the critics clearly stating their agenda, pre-occupations, prejudices, fetishes in just a generally put it on the table format.  Then the reader can then assess, in their own way, the criteria as well as the result.

I am coming to an opinion that golf courses should be split up into different categories, Tournament, Resort High Dollar, Mom & Pop, Muni, Ultra Private, etc. and then other categories based on geographical location.  How does a course in the UK compare to one in Arizona?  I am lost on that one.

[Andy Troeger]

Sean -

A course with 18 holes is better than a course with 9 holes.

A course measuring 6800 yards is better than one of 4800 yards.

Do you dispute?

I would dispute that either one of these is ALWAYS true. The 9-hole Dunes Club in Michigan is better than 80% (or more) of the 18 hole courses I've played. I'd rather play a great course from the forward tees of 4800 yards than a lousy one from 6800 just because of length. And if you get into 6500 versus 6800 then I'd absolutely say the length issue carries no merit.