Course Raters- status quo or upset the apple cart

[Dan Byrnes]

I am still a newbie here but have read a lot of the posts regarding rating courses and was interested in how the courses have changed from the NGLA not being ranked in the 80's highly. That thread inspired me to post these questions.

How much of the ranking suff is follow the crowd, a relatively homogenous group of people doing the rating, folks with a vested interest in keeping the status quo as both they and the courses where access is prized and rare seems to benefit from the rankings remaining high?

Are raters typically moneyed and connected old school private golf members or a more random mix of avid
golfers?

How does those raters wanting to play the Pine Valley's, Shinnecocks, and other difficult to access places keep the ranking of those places high and exclude lesser names from consideration?

As raters drop out and new ones are brought in.  Does that change the ratings landscape?

I have only played a dozen or so in the top 100 and typically walked away impressed.  I don't consider myself knowledgeable enough to be a ranker but certainly I could understand the desire to play all the greats? 

Is being a rater the ultimate club to be a member of,  great access to the worlds great courses with out the financial outlay and connections to actually become a member of such storied places?

Can a ranker really call up Pine Valley and ask to play? Or do they too need to find access through a member?

Anyways not tying to stir a pot nor criticize any raters here just want a better understanding of what makes this rating game tick.

[John Kavanaugh]

Raters are like abortion doctors.  Some do it for the money and some feel like they are providing a noble service.  Many of my best friends in the golfing world are raters.  One of the reasons I never ask a man what he does for a living or if he is a rater is because I once met a man who performed abortions.  It shook me that I chose to remain his friend.  Sometimes it is better just not to ask.

[Adam Clayman]

Lots of assumptions implied in those question. Many are inaccurate. The free golf is B.S., there's no expectation of comps. Yes a rater can call PV but I doubt it would get them on. I have never tried, though. Of the top four courses I have played and given very high votes/scores, not one of them was as a rater. And 3 of them were before I was ever a rater. And I'm not that well connected, or, do I try to access "the greats".

Everyone is different, and everyone has their opinion. Which is what is desired.

The entrenched staples on the lists appear to have a high number of votes, so new votes won't move the needle.

Apparently, for a long time now, marketing people have tried to figure out the system by inviting specific raters to boost their hype. It might work in the short run, but, imo, the cream rises and the hype goes by the wayside and those courses fall precipitously. Often off the lists completely.

[Terry Lavin]

Can a ranker really call up Pine Valley and ask to play? Or do they too need to find access through a member?

Anyways not tying to stir a pot nor criticize any raters here just want a better understanding of what makes this rating game tick.

First off, they're a sensitive lot, so calling them "rankers" as you mistakenly typed above, might inspire "rancor", if you allow the low-hanging-fruit of a pun.  Secondly, what makes it tick is selling magazines, so the different magazines take a different approach in coming up with a system that best helps them sell magazines.  It's a commercial, not an altruistic, enterprise.

[Dan Herrmann]

My name is Dan, and I'm not a rater.

[Dan Byrnes]

Also like Harbor Town are there other courses that were highly ranked 20-30 years ago that may have fully dropped out of the top100 and may have now closed or fallen on hard times?

[John Kavanaugh]

Can a ranker really call up Pine Valley and ask to play? Or do they too need to find access through a member?

Anyways not tying to stir a pot nor criticize any raters here just want a better understanding of what makes this rating game tick.

First off, they're a sensitive lot, so calling them "rankers" as you mistakenly typed above, might inspire "rancor", if you allow the low-hanging-fruit of a pun.  Secondly, what makes it tick is selling magazines, so the different magazines take a different approach in coming up with a system that best helps them sell magazines.  It's a commercial, not an altruistic, enterprise.

Terry,

For Golfweek it is much more than selling magazines.  Each rater pays annual dues and is required to attend rater camp that is also profitable to the magazine.  Raters are cash flow.

[Martin Toal]

Can a ranker really call up Pine Valley and ask to play? Or do they too need to find access through a member?

Anyways not tying to stir a pot nor criticize any raters here just want a better understanding of what makes this rating game tick.

First off, they're a sensitive lot, so calling them "rankers" as you mistakenly typed above, might inspire "rancor", if you allow the low-hanging-fruit of a pun.  Secondly, what makes it tick is selling magazines, so the different magazines take a different approach in coming up with a system that best helps them sell magazines.  It's a commercial, not an altruistic, enterprise.

'rankers' also invites usage of rhyming slang to create an alternative descriptor.

[Tom_Doak]

The free golf is not b.s. for a portion of the panelist class.  [At the same time, it's generally superfluous for those of us who are actually in the golf business.]  But when people long to be panelists, what they crave is access, and influence.

Every club has its own policy of dealing with them.  The most prestigious clubs just leave their guest policy as it is -- panelist or not, you need to be the guest of a member.  However, it's scary to see how few clubs nowadays are confident enough not to make some accommodations to panelists.  I am shocked at how accommodating some elite clubs are.  It's clearly because they have a lot of ego wrapped up in being #34 and worry that they might fall to #42 if a disgruntled rater takes it out on them with his vote.

Most panelists follow the crowd to a large extent ... you're allowed to have one or two personal exceptions to the top ten, but if you had seven, you probably wouldn't get on the panel at all.  [Some of the magazines discount "outlier" ballots which is the height of status quo hypocrisy, if you ask me.]  More worrisome is that in many instances, panelists communicate amongst themselves and decide which new courses are deserving or not, instead of just voting their own thoughts and seeing how it turns out.  That's what gives rise to politicking the system, and obviously biased results.

[John_Conley]

Also like Harbor Town are there other courses that were highly ranked 20-30 years ago that may have fully dropped out of the top100 and may have now closed or fallen on hard times?

Haig Point and Bonita Bay are examples of courses that were ranked highly upon opening and are now not appearing on the list.  Grenelefe West and many others garnered recognition and then fell off.

Important to note is that Golf Digest began with the 200 Most Difficult courses and the list morphed into the Greatest in the 70s.  It took several iterations to shake out some of the courses that are just hard and not much else.

[Garland Bayley]

The free golf is not b.s. for a portion of the panelist class.  [At the same time, it's generally superfluous for those of us who are actually in the golf business.]  But when people long to be panelists, what they crave is access, and influence.

Every club has its own policy of dealing with them.  The most prestigious clubs just leave their guest policy as it is -- panelist or not, you need to be the guest of a member.  However, it's scary to see how few clubs nowadays are confident enough not to make some accommodations to panelists.  I am shocked at how accommodating some elite clubs are.  It's clearly because they have a lot of ego wrapped up in being #34 and worry that they might fall to #42 if a disgruntled rater takes it out on them with his vote.

Most panelists follow the crowd to a large extent ... you're allowed to have one or two personal exceptions to the top ten, but if you had seven, you probably wouldn't get on the panel at all.  [Some of the magazines discount "outlier" ballots which is the height of status quo hypocrisy, if you ask me.]  More worrisome is that in many instances, panelists communicate amongst themselves and decide which new courses are deserving or not, instead of just voting their own thoughts and seeing how it turns out.  That's what gives rise to politicking the system, and obviously biased results.

Thanks Tom.

[Kalen Braley]

My name is Dan, and I'm not a rater.

Ha ha,

I like the spin off version..


"Hi, my name is Kalen, and Im not a rater...but I did stay at a Holiday Inn Express last night"

[JC Jones]

 [Some of the magazines discount "outlier" ballots which is the height of status quo hypocrisy, if you ask me.]

Discount is too favorable a word, the word that should be used is "omit."  Of course, the omission of outlier ballots perpetuates the status quo inertia and allows for such things like Tullymore to stay in the Top 100 longer than what is acceptable and allows for courses like Ballyneal to stay out of the Top 100 longer than what is acceptable.

Worst of all is that the omission of outlier ballots assumes that the rating criteria will produce a certain ranking of a course regardless of the rater and that the one submitting an outlier ballot is not in full understanding of the rating criteria, otherwise their ballot would have been within one standard deviation of the "group."  Therefore, if you are outside of the "group think," you don't understand the rating criteria.  Which means that the rating criteria are designed to produce the same rating of a course among all members of the group.

[Brent Hutto]

Mr. Jones mentions in passing the key assumption underlying this whole ratings business. It is assumed that there is some underlying reality which the raters are trained to discover with only a limited amount of individual-rater variability and perhaps a certain amount of individual bias. All of which should "average out" if you set it up correctly, once you get enough ratings of each course. Sort of a Platonic view of golf course quality you might say.

Nothing wrong with that assumption if that's how you view the situation. But there could be another equally defensible assumption that each rater brings to bear a slightly different (or perhaps in some cases qualitatively different) "ideal" of what a golf course should be. In that case your training mostly serves to either dissuade the quirkier raters from honesty about their assessment or to week them out altogether from your rater pool. Discarding "outliers" is sort of last ditch backstop in case an occasional rater with a very different "ideal" happens to submit ratings reflecting that version of reality instead of the agreed-upon one.

[John Kavanaugh]

Perez Hilton ballots demand that outliers be omitted.

[Anthony Gray]

  How could a qualified rater deliver an outlier ballot?

  Anthony

 

[JC Jones]

  How could a qualified rater deliver an outlier ballot?

  Anthony

 

It depends on your definition of qualified.  Apparently, "qualified" in the rater sense means "sufficiently indoctrinated to a certain ideal such that a rating of a course using criteria set to serve that ideal will result in a rating within one standard deviation of the 'group.'"

[Brent Hutto]

Or put more colloquially, a qualified rater is "on board" with the outfit he's rating for and knows how to apply the criteria his is given.

[Tom_Doak]

  How could a qualified rater deliver an outlier ballot?

  Anthony

 

Anthony:

Some of the most interesting ballots I saw in my time of adding them up were from the panelists who were women.  They see a completely different course than we do [by design], and they have very different golf games, too.  A few staples of the top 100 were almost universally panned by women panelists -- and I think it had everything to do with the design and not the clubs' attitudes toward women golfers.

I agree completely with Brent's idea that the ideal panel should not be a bunch of similar people, but a variety of golfers with different eyes and different perspectives.  Sadly, none of the magazines are very good at this anymore.  GOLF Magazine probably still has the most diverse group, but there are more and more "career panelists" every time, with increasingly questionable results.

[Anthony Gray]

  How could a qualified rater deliver an outlier ballot?

  Anthony

 

It depends on your definition of qualified.  Apparently, "qualified" in the rater sense means "sufficiently indoctrinated to a certain ideal such that a rating of a course using criteria set to serve that ideal will result in a rating within one standard deviation of the 'group.'"

  Count me in.

  Cruden Bay

[John Kavanaugh]

Perez Hilton was qualified to judge a beauty pageant but then he got his panties in a bunch over a personal agenda. It happens.

[Brent Hutto]

Tom,

I don't know how much exposure you've had to the corporate or academic research worlds but the perfectly valid and important tactic of using "focus groups" or similar qualitative research panels can suffer from the same problem. If you're not real careful, as the person setting up the panels you'll be tempted to reuse (and reuse and reuse) the people from whom you've elicited "good input" on previous projects.

Of course what's really happening is that certain panelists are getting better and better at finding what the researcher wants to hear and the researcher is getting more and more used to hearing it. The endpoint can easily be a minimal effort path for gaining very high quality feedback that appears to occasionally "challenge" the status quo but in the most important elements actually reinforces it.

Occupational hazard in that kind of work. No different really than the principle which says "If you don't want to go under the knife, then don't go asking a surgeon for his opinion". Or golf course designs, renovations, restorations, etc. etc. We're all only human and it takes a superhuman effort not to Get What We Want even if we don't want to.

[Jim Franklin]

  How could a qualified rater deliver an outlier ballot?

  Anthony

 

Anthony:

Some of the most interesting ballots I saw in my time of adding them up were from the panelists who were women.  They see a completely different course than we do [by design], and they have very different golf games, too.  A few staples of the top 100 were almost universally panned by women panelists -- and I think it had everything to do with the design and not the clubs' attitudes toward women golfers.

I agree completely with Brent's idea that the ideal panel should not be a bunch of similar people, but a variety of golfers with different eyes and different perspectives.  Sadly, none of the magazines are very good at this anymore.  GOLF Magazine probably still has the most diverse group, but there are more and more "career panelists" every time, with increasingly questionable results.

But Golf Magazine includes Torrey Pines South in their Top 100. That is plain awful. I know, I know, each magazine has their own cross to bear, but TPS just blows me away that a diverse panel like GM would include it.

[John Mayhugh]

Maybe some would find this Golf Digest Course Critic app a better approach?
http://www.golfdigest.com/magazine/digital-subscription

For me, the most valuable rating service is hearing feedback from people that I know that like the same sorts of courses as I do.  Fortunately, GCA has given me contact with a way more traveled group than I ever dreamed possible.  Without that sort of resource, magazine rankings and books would have a more significant influence on me.  It's not surprising that many people use rankings, however flawed they may be.

[Dan Herrmann]

PS - Laura is able and willing to be a rater - they need more women! (she calls Sand Hills and Pacific Dunes the best courses she's every played, so she definitely gets it!)

Ms. Redanman (Renee) is, I think, a Golfweek rater.