Who is qualified to rate golf courses?

[Andy Troeger]

Matt,
Next time I think about responding to one of your posts remind to beat my head into the wall instead--quicker, more efficient, and achieves the same result  

We'll agree to disagree that your arbitrary cut off provides any more benefit than mine...

[Steve Curry]

How do you qualify "Qualified"? 
Elimination of bias stands as the greatest challenge to the exercise, a spectrum of moron to genius confluent with scratch to duffer would be the best remedy.


Cheers,
Steve

[Kevin Lynch]

Andy:

Hello -- yes it's arbitrary - so is any number system one uses. I don't see how a collegiate football rating system works well. It was self-created by Digest and then all the other pubs followed the pied-piper approach so that every year or two you can create "news" that such and such course fell one position or another moved into the top spot -- see the fanfare that the ascension of ANGC to the top spot caused. In regards to the system you mentioned -- you can very well have them assigned a certain letter or number grade and go from there. But frankly the concept of "groupings" -- whether by an assigned number or a heading of say ten courses -- works better than having such a silly and preposterous notion that there is only one #1 course in the land.

I like the groupings because at some point there will be a cut-off -- the original Digest approach worked well in my mind.

Andy, let me state again aggregate ratings are meaningless -- they simply push numbers together and then ipso facto like some sort of cheap magician's trick we get the RESULT. There is no ryhme, reason or detailed analysis - it's just throw courses into the air -- have people vote without any meaningful wherewithal to cross compare from a similar pool of courses played. For example, if person A plays Oakmoint and person B plays Merion and neither has played both -- you have to assume that these respective people can apply the numbers in some sort of consistent fashion. That won't be an issue for the top top courses -- but it does becomes more of a problem the further from the top you slide down.

We do agree a well-researched listing will likely contain many fine courses but there are few raters who have the wherewithal to see the totality -- they often can only approach the process from a limited side of things. That's what made Doak's CG book so fascinating -- a clear and consistent analysis - albeit from his perspective -- but one that was well thought out and not polluted with the aggregate style that is nothing more than a hodge podge of this and that.

Matt,

I think you are right that the "Tiered" approach is more meaningful and realistic, but I wonder if "Tens" is really just the same problem (as Tom Doak mentioned earlier).  In reality, the tiers may need to be a little wider (such as Top 10, then 11-30, Then 40-80 and getting wider).  Like you said, is there really a difference between a #81 and #115 course?

Really, isn't that what Doak's Guide did?  The measure of quality is probably more of a "Bell Curve" rather than a linear, relative progression.  As we moved down from 9,8,7 - the numbers is each class grew.

But what you (and I) also liked about Doak's Guide was that he was the constant factor, rather than relying on many individuals having the exact same relationship with a proscribed scale.

if we are going to have ratings systems that involve multiple raters, I like the "head-to-head" methodology that Anthony Fowler was using for his "Re-Rating the GCA Top 100."  To some it extent, it shifts the "constant" factor back to the Individual Rater rather than a set "numerical scale" (but still provides some "guidance" as to what things to consider). 

Ultimately, there is no perfect answer / solution in a purely quantitative exercise.  Using Wider "Tiers" as I mentioned earlier could eliminate some of the obsession over "linear" rankings ("Woohoo - we moved from #52 to #41") and the "head-to-head" feature smooths out "inflationary grading." 

At the end of the day, I'll take the rankings with a grain of salt, because I've walked of Highly Ranked courses going "really?" and played Unranked Courses that I would play 10 times out of 10 given the option.  There's no replacement for qualitative discussion and comments which explain the rater's feelings, which is why I'd probably just turn to people here for suggestions / thoughts.

[Kirk Gill]

For my sake I'm far more interested in opinions of specific people that I know than I am in some sort of aggregate "opinion by committee."

It's the same way with movies. I may read Roger Ebert's opinion on a movie, take into account the degree to which his opinion and mine have agreed in the past, and then make my decision as to whether or not I want to go see it. I do not look at which movie did the most box office over the weekend and make my decision based on that.

An imperfect analogy, but you get the idea.

[John Kavanaugh]

I was actually playing golf with someone when they got the call they were accepted as a GolfWeek rater. You just never know when you go out in public who you are standing next to, could be a suicide bomber, a paroled rapist, a pedophile, or....a golfweek rater!

Did he get his green fee refunded?

[Matt_Ward]

Andy:

So be it -- but just remember you're the guy who thinks that Paa-Ko Ridge has the edge over Black Mesa too.

Kevin:

No doubt there is a cut-off figure -- in my case -- I'd have the first 50 rated courses broken down in sequences of ten. The 2nd 50 would just be listed alphabetically as GD previously did.

I undersand Andy's desire to break-off courses simply from a numerical rating -- a 10, 9, 8 and so forth. Even in his case there is a break point between a course getting a 9 and another getting an 8. In either case -- mine or his -- an arbitrary determination is made which filters courses to one side or the other.

Kevin, frankly many people would be hard-pressed to say what is significantly different from the 81st to the 115th course. If I remember certain rating numbers that Jonathan Cummings once posted the separation between such courses is very, very small indeed.

The larger issue is that aggregate ratings are nothing more than cumbersome and largely ineffective mechanism of group think. There is the assumption that all raters are equals. That is far from the case. Many raters are regional and often state-centered in their thoughts. Few really have played a very wide cross section of courses.

Kevin, the "constant factor" you highlighted with Doak's CG is his steady voice and analysis which is aplied from one course to the next from one country to the next. One is then able to get the kind of personalized cross-comparisons that are simply not a part of the gallp / zagat poll group think most magazines follow.

[Andy Troeger]

Matt,
After more rounds, I've come to the realization that Black Mesa is better than Paa-Ko.

How would your tiering system affect the state lists? Would you list ten courses without an order?

[RSLivingston_III]

"Who is qualified to rate golf courses?"

I'll go with, if you have to ask it ain't you. (for 100, Alex)

[cary lichtenstein]

Everyone who rates golf courses has a bias, even if they are unaware of it. How does it fit your game?

I remember playing in a 2 day member guest. The course was narrow and the trees on the right side of the fairway humg over the right side of the fairway. I alwayed started my ball out over the right rough and drew it back into the center of the fairway. Not here. It crashed into the trees 6 times that day. I never learnt how to hit a fade.

Not only did I tell my buddy to not invite me back the next year, I hated the course. I thought I was a good rater, but how could I not give this course low marks.

My buddy faded everything and never saw the overhanging limbs on the right side.

I loved courses with forced carries, eye candy, huge traps, and lots of quirk. In minimialists eyes, that made me a bad rater. I hated miminalist courses and guys who see all kinds of stuff that isn't really there, but when certain guys tell you what is in, alot line up like sheep to repeat the same words.

Funny how the best work of the miminalists isn't mininailism, but instead a great piece of land with great movement and a great routing to take advantage of it.

[John Kavanaugh]

This was all so much more fun 5 years ago before the bottom fell out of the golf market.  It saddens me to see even the raters themselves admitting that no standards are the best standard. I have seen far too many friends lose their jobs and or livelihoods to any longer see the humor in this farce.  The ratings are fine and fun and harmless by themselves but allowing a large percentage of what is left of the golfing traveling public to play without fee or membership is hurting those brave enough to remain investors in the game. It's time for all magazines to mandate a pay to play policy for each and every panelist and their friends, families and guests.  There is not a single club that needs a rater to survive yet every remaining club requires revenue to stay afloat.  This does not need to be a permanent policy, just long enough to give golf a shot in the arm. 

I will personally donate the postage it would take for Golfweek to mail out to their 1700 nominated courses in the U.S. a letter requiring each visiting rater be charged the published guest fee until election day of November 2012.  This simple act would infuse a much needed $2,000,000 in a starving golf market.   

[Bart Bradley]

John:

This is voodoo economics.  I played 75 rounds this year.  38 new courses.  Informed the course that I was a rater 3 times and was comped 2x.  Your assumptions may be completely and totally false.  I am happy to pay for my rounds; in fact, I already do, nearly all the time.

The raters are a large percentage of posters but a miniscule percentage of golfers.  Sample error.

Bart

[Mac Plumart]

John...

I am with Bart in that I think you are way overstating this "raters play for free deal", but maybe I am ignorant and/or stupid. 

Either way, I am with you that rater's should pay a guest fee to play. 

[Adam Clayman]

Kelly, He wrote the book "How to win friends and influence people".

So an architect should try to find out who the raters are and then win them over and influence them? Does that really happen? I know an architect who bought a very nice camera for the head of all raters at GD. Is that the kind of thing architects should be doing?

Kelly, No! That's not anywhere near what I was saying.

My original comment was in jest, and the new was in quotes.

It's similar to John K's way of getting invited to the best courses. Say something completely degrading and some member will IM him and invite him to prove him wrong.

But, clearly, calling out raters and associating them with pedophiles and other of life's scum, is an interesting way to become recognized as an architect.

[Tim Martin]

This was all so much more fun 5 years ago before the bottom fell out of the golf market.  It saddens me to see even the raters themselves admitting that no standards are the best standard. I have seen far too many friends lose their jobs and or livelihoods to any longer see the humor in this farce.  The ratings are fine and fun and harmless by themselves but allowing a large percentage of what is left of the golfing traveling public to play without fee or membership is hurting those brave enough to remain investors in the game. It's time for all magazines to mandate a pay to play policy for each and every panelist and their friends, families and guests.  There is not a single club that needs a rater to survive yet every remaining club requires revenue to stay afloat.  This does not need to be a permanent policy, just long enough to give golf a shot in the arm. 

I will personally donate the postage it would take for Golfweek to mail out to their 1700 nominated courses in the U.S. a letter requiring each visiting rater be charged the published guest fee until election day of November 2012.  This simple act would infuse a much needed $2,000,000 in a starving golf market.   

Do you really think the rater that goes in his pocket for travel expenses to go far and wide to sample different courses is getting over on the golf course industry for a comped round of golf? Without the service of the raters there are no rankings.

[Mac Plumart]

I thought it was in jest and so was my original comment. As for your last comment above I don't take raters or the rating business seriously, in fact I think the rating business is a sham and a negative in the business  and it has no impact on me as an architect because there is not a rater alive that cares what I do. There is a certain level below which they will not waste their time and I am far down the ladder!

Kelly,

It looks like from your website there is at least one living rater that cares what you do!!  

http://kellyblakemoran.com/article.html

[John Kavanaugh]

Tim,

Would you turn in your rater card if comps were not part of your compensation. If so you are the first rater I have ever met who shares your opinion. If you would choose not to travel to a course because of the green fee perhaps the owner would soon discover his fee is too high. In that case the golfer wins. What is wrong with that system?

[Tom_Doak]

 Without the service of the raters there are no rankings.

Tim:

You should take back that last quote before anyone else sees it.

This is the problem with rating panels ... some eventually start to think that THEY ARE THE SHOW, rather than just the audience.

One of the nice things about the GOLF Magazine panel [in the old days] was that nearly everyone was in the golf business, so none of them identified themselves as "panelists" at all, or needed to look to be comped when they visited a new course.  It was my idea (and my mistake) to suggest putting a handful of traveling amateurs on the panel, not thinking they would use the position for access ... but they certainly have done so over the years.


Everyone else:

Y'all are delusional if you don't think there is a lot of influencing going on behind the scenes.  Much of it is innocent -- for example, I am friendly with many panelists from many publications, many of whom I've known long before they were panelists.  [It wasn't such a large group of people 25-30 years ago when I started meeting people in golf.]  

On the other hand, I've met lots of panelists who have introduced themselves to me AS panelists.  I'm sure that some are just proud of the fact.  Others probably assume I will think more of their knowledge of courses because they are panelists, even though that's not the case.  Some clearly hope to arrange a game at one of the more private courses I've built [I've had 30-40 requests from panelists I don't know to play Sebonack, for example].  And, unfortunately, there have been at least a couple who gave the impression that I should be nice to them, because as panelists they have some control over how my work is judged.  They're the exception, not the rule ... but the longer some people serve in this capacity, the more they tend to think they're Important.

There is no architect out there who doesn't know a fair number of panelists, whether we try to or not.  And I hear panelists all the time telling me that architect A or B is "a really nice guy."  You really don't think that influences how the panelist rates architect B's courses?  And you don't think architects understand that?

[Sean_A]

 Without the service of the raters there are no rankings.

There is no architect out there who doesn't know a fair number of panelists, whether we try to or not.  And I hear panelists all the time telling me that architect A or B is "a really nice guy."  You really don' think that influences how the panelist rates architect B's courses?  And you don't think architects understand that?

Tom

Of course not!  Raters are as pure as the driven snow and knowledgable as Einstein. 

Its laughable to think panelists aren't influenced by freebies, being treated well and hob nobbing with whos who in the industry.  Yes, of course its only the architecture that matters.

Ciao

[Tim Martin]

 Without the service of the raters there are no rankings.

Tim:

You should take back that last quote before anyone else sees it.

This is the problem with rating panels ... some eventually start to think that THEY ARE THE SHOW, rather than just the audience.

One of the nice things about the GOLF Magazine panel [in the old days] was that nearly everyone was in the golf business, so none of them identified themselves as "panelists" at all, or needed to look to be comped when they visited a new course.  It was my idea (and my mistake) to suggest putting a handful of traveling amateurs on the panel, not thinking they would use the position for access ... but they certainly have done so over the years.


Everyone else:

Y'all are delusional if you don't think there is a lot of influencing going on behind the scenes.  Much of it is innocent -- for example, I am friendly with many panelists from many publications, many of whom I've known long before they were panelists.  [It wasn't such a large group of people 25-30 years ago when I started meeting people in golf.]  

On the other hand, I've met lots of panelists who have introduced themselves to me AS panelists.  I'm sure that some are just proud of the fact.  Others probably assume I will think more of their knowledge of courses because they are panelists, even though that's not the case.  Some clearly hope to arrange a game at one of the more private courses I've built [I've had 30-40 requests from panelists I don't know to play Sebonack, for example].  And, unfortunately, there have been at least a couple who gave the impression that I should be nice to them, because as panelists they have some control over how my work is judged.  They're the exception, not the rule ... but the longer some people serve in this capacity, the more they tend to think they're Important.

There is no architect out there who doesn't know a fair number of panelists, whether we try to or not.  And I hear panelists all the time telling me that architect A or B is "a really nice guy."  You really don't think that influences how the panelist rates architect B's courses?  And you don't think architects understand that?

Tom- Isn`t that a completely factual statement that "without raters there are no rankings"? If a certain rater`s sense of self importance is inflated by such a statement then the magazines hired the wrong guy. Maybe I am naive but I can`t imagine that every rater that learns he won`t ever get comped again is going to walk away as JK says in his previous post. See Bart Bradley`s reply #88 as an exception to this line of reasoning. Additionally it`s a shame if architect A or B is only perceived as a really nice guy not because he is but because he is subliminally trying to influence a rater. As with all things in life when someone has the ability to effect the outcome of something there will be some measure of influence peddling. Ratings put courses on people`s radar and I don`t see how that hurts the golf industry.

[Jud_T]

 Ratings put courses on people`s radar and I don`t see how that hurts the golf industry.

This quote alone is worthy of it's own seperate thread....Tim,  I would say that it certainly helps the given course. How and why it's on people's radar and what the given course represents in terms of a host of issues may or may not be a positive for the industry long term...

[John Kavanaugh]

It is important to note that Golf Digest put together their best new issue this year without the help of any raters. Ron Whitten did it all by himself.  Also note that journalists are not raters and raters are not journalists.

Tim,

I do not believe a single rater would quit just because he would have to pay the same fees as any other guest. If any of you would please let me know and I will pay your postage to submit your resignation. Certified return receipt, no less. 

[Phil McDade]

See Bart Bradley`s reply #88 as an exception to this line of reasoning.

Tim:

Bart may or may not favor a course because he was comped, but a course that's in the business of comping raters may view it as a good way to get favorable ratings. How many Barts are out there getting comped as raters? If, say, a dozen are comped at the same course, aren't the odds that that course will end up with more favorable ratings, all other things being equal?

[Adam Clayman]

Jk, Do you continue to ignore the fact that comps are NOT part of any compensation just to be controversial?

I'll say it for the last time, there is no expectation of a comp.

Every person is different in how they view the process.

The reality is that when someone (a normal person, not you lot) finds out I'm a rater (because I certainly don't tell them) they almost always mis-understand. They assume I write in the magazine.

It's been a long time since we had good old fashion rater bashing thread. It was my mistake to think Carl's query was genuine. That won't happen again. GN.

[JC Jones]

I think there is a need for raters for no other reason than to serve as foils in the John Kavanaugh narrative.

[John Kavanaugh]

JC,

I can honestly say that I have played every great course I have ever desired because of my many friendships with raters. The majority comped to boot. I love raters, I love golf and believe my proposal of pay to play will go a long way to help both.