'Tis the (Ratings) Season - General Rater Question

[Kevin Lynch]

Since the GD lists are out and have already generated-digit pages of comments, I thought I'd ask a general question of the raters (and others).

A few years back, I had an exchange with Brad Klein regarding my perceived under-rating of Ballyhack.  In that exchange, a comment came up regarding the severity of the terrain, which could cause confusion in how to play the course and assess multiple options.  Paraphrasing, Brad said "no doubt on subsequent plays the angles and slopes reveal themselves, as do the possible lines of play. But such an approach to design can also spawn some critical responses."


That comment stuck out to me as I felt it highlights a critical underlying premise for a rating system. 

Is the rating intended for the person who may only get to play the course once, or for someone who would like to play the course frequently?  It seems to me that these two populations may have diametrically opposed desires with respect to course being "right there in front of you" vs. "a mystery to be solved over time." 

Whichever way the course goes in that regard, should that necessarily be viewed as a "negative" or should it simply be understood as a different philosophy (and judged on that standard)?


Just by the nature of the ratings system (usually one visit), I would expect raters to favor the "right there in front of you" approach.  Or stated the opposite, the "mystery course" may not be appreciated as much during a one-time visit.

For people who perform ratings, is the consideration of "one-time" vs "repeat" play discussed in your "training?"  Is it possible that bold or multiple-variety course have to fight more of an uphill battle to be appreciated?


Please understand, I'm not trying to be critical of raters.  I admit my own inability to fully comprehend a course.  I just wanted to get some thoughts from people who do this.

[Andy Troeger]

Just by the nature of the ratings system (usually one visit), I would expect raters to favor the "right there in front of you" approach.  Or stated the opposite, the "mystery course" may not be appreciated as much during a one-time visit.

For people who perform ratings, is the consideration of "one-time" vs "repeat" play discussed in your "training?"  Is it possible that bold or multiple-variety course have to fight more of an uphill battle to be appreciated?

I've never really heard discussion of rating for a specific audience in the way you've mentioned. Golf Digest rates based on the viewpoint of a scratch or low-handicap golfer but not based on whether they are viewing a course for the first time or the 100th time.

I would argue that bold courses often make better first impressions and might do better in the ratings, unless there is the possibility that the bold features might be deemed "unfair" by a visitor that does not understand how to interact with them. But, I think it probably depends on the rater.

[Jason Topp]

Kevin:

It is definitely a flaw in the system that many raters run around and squeeze as many rounds as they can at as many courses as possible on a trip.  I have found that I enjoy a trip much more if I take my time and play courses multiple times.  The downside of my approach is that it is difficult to see many courses that way.  Despite my preference for multiple plays, I think one is able to discern 90-95% of what is there the first time around and I have rarely had a radically different view of the quality of a course based on the second, third or even fifth round.

The most difficult thing to imagine is how a course plays in different weather conditions, and multiple plays on a trip do not necessarily address that concern. You often get similar weather conditions if you play a course on consecutive days.  For example, you will hear many on here criticize Brandon Trails because the finish is so brutal.  I was surprised at that reaction but when I played the course, the wind blew from the South which is opposite the normal summer wind and means that the holes play downwind.  I was there four days and the wind was the same each day.

[Jeb Bearer]

Doesn't this problem sort of work itself out over time? That is, courses that make a big splash but don't contain the depth of strategy/subtlety tend to drop off the list after a few years, when the raters have played them multiple times, while truly great golf courses will stand the test of time and remain on the list as raters find the course equally engaging to play year after year. So maybe the solution is to pay attention, when looking at any rankings, to how old the course is. If a course like CPC or PV is on there, it's a safe bet that it's a good course. But if a brand new course pops up out of nowhere, it's probably best to watch for that course in subsequent years to determine the true quality of the design.

[Tom_Doak]

Kevin:

It's a good question, and my first thought [having spent the past day taking apart the GOLF DIGEST rating system] is that GOLF DIGEST, the only one of the magazines that defines what their criteria are, has no criteria which address the difference you are citing.

Meanwhile, the other magazines just leave it up to the individual panelist.

In the end, as we've said many times in the past couple of days, the quality of a ranking is only as good as the judgment of the people who are voting on it.  When I cast my own vote, I am always thinking about the long haul and whether the appeal of the course will grow or wane over time.  That's why I've got St. Andrews so high on my personal list.  But I can't speak for other voters.

[John Kavanaugh]

I play most of my golf where I am not a member with raters for the various magazines.  You can get a pretty good handle on their opinion based on where they choose to be members themselves.  

One of the,classic instances was when I met the only person to ever bad mouth Sand Hills in a fervent open manner.  He was a member of Milwaukee CC.  Great guy, just loved something different.

[Joe_Tucholski]

I always thought of the lists targeting a one time player.  The reason being the raters generally see the course once and if I'm playing a course on a list, chances are it won't be for repeated play.

[John Kavanaugh]

I always thought of the lists targeting a one time player.  The reason being the raters generally see the course once and if I'm playing a course on a list, chances are it won't be for repeated play.

Every rater I have played with, with one exception, is perfectly capable of judging where a course should be placed on their list after one play.  Hell, most of us know where a course should be before we play it so on that special case where it takes a brain they all, with one exception, got it.  note:  I have met several raters that I have not played with.

I mean really, do any of us need to play the Streamsong courses more than once to submit a rating sufficient to benefit any given panel?  I used to think the courses didn't need to be visited at all until the recent overload of photo tours has me convinced that pictures make courses far less interesting than what they are.  The people who are posting these pictures are doing the courses a great disservice which may lead to some early artificially low rankings.

Even the clueless rater just submits a number in line with what the course has received in the past.  No harm is done.

[Nigel Islam]

I always thought of the lists targeting a one time player.  The reason being the raters generally see the course once and if I'm playing a course on a list, chances are it won't be for repeated play.

Every rater I have played with, with one exception, is perfectly capable of judging where a course should be placed on their list after one play.  Hell, most of us know where a course should be before we play it so on that special case where it takes a brain they all, with one exception, got it.  note:  I have met several raters that I have not played with.

I mean really, do any of us need to play the Streamsong courses more than once to submit a rating sufficient to benefit any given panel?  I used to think the courses didn't need to be visited at all until the recent overload of photo tours has me convinced that pictures make courses far less  interesting than what they are.  The people who are posting these pictures are doing the courses a great disservice which may lead to some  early artificially low rankings.

Even the clueless rater just submits a number in line with what the course has received in the past.  No harm is done.

I think this is an excellent point. I think you get a very good feel for the quality of the course with one play and some research. Where several plays help is with the ability to shoot a score.

[Joe_Tucholski]

I always thought of the lists targeting a one time player.  The reason being the raters generally see the course once and if I'm playing a course on a list, chances are it won't be for repeated play.

Every rater I have played with, with one exception, is perfectly capable of judging where a course should be placed on their list after one play.  Hell, most of us know where a course should be before we play it so on that special case where it takes a brain they all, with one exception, got it.  note:  I have met several raters that I have not played with.

I mean really, do any of us need to play the Streamsong courses more than once to submit a rating sufficient to benefit any given panel?  I used to think the courses didn't need to be visited at all until the recent overload of photo tours has me convinced that pictures make courses far less interesting than what they are.  The people who are posting these pictures are doing the courses a great disservice which may lead to some early artificially low rankings.

Even the clueless rater just submits a number in line with what the course has received in the past.  No harm is done.

As others have said elsewhere I guess I wouldn't make a very good raters because I have different opinions on many courses after my second play.  For me I have yet to play a fazio course that I did not enjoy after the first play.  That being said there are also courses I've played that I didn't think were all that special on the first play.  In the latter group I include a course I play regularly, rustic canyon.  I didn't enjoy it all that much my first time around but with each play I enjoy it more.  I suspect the old course will fall into the same category, Unfortunately the price will likely prevent me from playing it over and over. 

I should also note I primarily look at the top public courses lists, not sure if that makes a difference.

[Kevin Lynch]

Thanks for the feedback guys - I appreciate the first-hand insight.

As I may have expected, much of it does depend on the rater and their own preference.  I just wondered if it was "coached" into the raters to set aside their personal preference for "right there in front of you."


Jason wrote - "I think one is able to discern 90-95% of what is there the first time around and I have rarely had a radically different view of the quality of a course based on the second, third or even fifth round."

As much as I respect many of the raters here, I have to believe there are a number of under-appreciated courses out there that don't reveal themselves immediately.  While you may not have a "radically" different view of a course after the multiple-plays, it seems like the rankings are so tight that a more "mysterious course" is negatively affected.

To throw out a few MacKenzie quotes:

"A good golf course is like good music or anything else: it is not necessarily a course which appeals the first time one plays over it; but one which grows on a player the more frequently he visits it."

"A first class hole must have the subtleties and strategic problems which are difficult to understand, and are therefore extremely likely to be condemned at first sight even by the best of players."


If MacKenzie is right about that, I understand how it can be so difficult to come up with a rating on one visit, especially when you consider the "changing weather" scenario Jason described.  I think we saw that on the Streamsong thread in recent days, where an opposite wind made the hazard placements confusing.

[Jeb Bearer]

According to Mackenzie, who worried at the lack of controversy created by CPC, the best courses would be the most harshly criticized at first.

[Tom_Doak]

Every rater I have played with, with one exception, is perfectly capable of judging where a course should be placed on their list after one play.  Hell, most of us know where a course should be before we play it so on that special case where it takes a brain they all, with one exception, got it.  note:  I have met several raters that I have not played with.

I mean really, do any of us need to play the Streamsong courses more than once to submit a rating sufficient to benefit any given panel? 

John:

I've always felt it was in my best interests to try and get people to come and play my new courses TWICE on consecutive days ... so they could see whether the course plays differently with different hole locations, and so they might forget their double-bogeys from Day 1.

I agree that most raters could play Streamsong once and decide whether it belongs in their list of greats or not.  I am NOT convinced they can determine where it ought to fit on that list.  In fact, that is the main reason that virtually all of the panelists today get together and compare notes on a regular basis ... even here on Golf Club Atlas.  Hell, that's what they are doing when they are playing golf with you!  They are scared to go against the grain.  And for years, the Best New rankings reinforced this by telling them after one year what they should think.

[Ben Sims]

Every rater I have played with, with one exception, is perfectly capable of judging where a course should be placed on their list after one play.  Hell, most of us know where a course should be before we play it so on that special case where it takes a brain they all, with one exception, got it.  note:  I have met several raters that I have not played with.

I mean really, do any of us need to play the Streamsong courses more than once to submit a rating sufficient to benefit any given panel?  I used to think the courses didn't need to be visited at all until the recent overload of photo tours has me convinced that pictures make courses far less interesting than what they are.  The people who are posting these pictures are doing the courses a great disservice which may lead to some early artificially low rankings.

Even the clueless rater just submits a number in line with what the course has received in the past.  No harm is done.

Your best post ever?  May be. 

[Ronald Montesano]

Ben, I found that post to be disheartening. A call to "mail it in" in lieu of doing one's appointed work. I do recognize that some golf course raters get "it" the first time through.

When I played CCTroy this summer, it stunned me, as Mohawk had done the previous year. I tend to react with strength and passion, two responses that would be detrimental to a detached, calculated rating of a golf course. I would need a second tour of an important course to clarify things.

I am married to a woman who cannot comprehend that others don't "get" what she perceives/understands instantly on myriad subjects. I  am convinced that Online JK is similar; I hope that offline JK has more patience for us mortals.

[Kevin Lynch]

On a similar but separate note, I also wonder how much a course with numerous blind shots is affected.  

I recall the Tobacco Road locker room had a "treatise on blind shots" which was taken from the St. Andrews yardage book.  I can't locate it now, but the general gist was that blind shots were meant to keep the course thrilling for the repeat play member so that the course never became too familiar.  Again, blindness seems like a characteristic that could be loved in the context of repeat play, but loathed by the infrequent visitor.

I think everyone here has a personal feeling about blind / semi-blind shots in design.  As raters, are you told to set these preferences aside during the ratings process?  Do you have many ratings where you say "I don't like that style of course, but for people who do, it's a great design?"

Are there many courses rated highly with a proliferation of blind shots?  Or is it possible that these types of courses, while liked by many, have their averages dragged down by those who cannot get past that preference.  

I remember that Tom Doak also made an observation about severe-terrain courses in general, noting that they are poorly rated overall.  I have to believe that some of this is due to personal preference creeping in, even if just for a smaller subset of raters.

[Ben Sims]

Ron,

I found it to be a solid bit of satire from John.  If you cannot see the humor and grinning truth of parts of his post, then you are taking this stuff a bit too seriously.

I am admittedly down on golf course rankings right now.  I think it once served a purpose.  To me it feels like it may be diluting the pool.  Homogeneity doesn't feel right to me when it comes to golf.   

[John Kavanaugh]

Ben, I found that post to be disheartening. A call to "mail it in" in lieu of doing one's appointed work. I do recognize that some golf course raters get "it" the first time through.

When I played CCTroy this summer, it stunned me, as Mohawk had done the previous year. I tend to react with strength and passion, two responses that would be detrimental to a detached, calculated rating of a golf course. I would need a second tour of an important course to clarify things.

I am married to a woman who cannot comprehend that others don't "get" what she perceives/understands instantly on myriad subjects. I  am convinced that Online JK is similar; I hope that offline JK has more patience for us mortals.

Ron,

How often is your gut reaction about a child in your class wrong?  I don't mean marginally wrong like between a Juco or State College grad.  I mean how often does a child surprise you by being truly great or evil after you have made it through the first grading period and met their families?  Golf courses are far less complicated than children in my estimation.

I know McVeigh went to high school near you.  He would have been a tough nut to crack.

[Ronald Montesano]

If I had as much daily experience reviewing and rating golf courses, I would agree. Few do.

At times, I refer to certain students as a hostile clientele. Even the most experienced teachers peg good kids as bad and bad as good, in the initial evaluation. In fact, we have a teacher in my department who makes an analysis of students during their freshman year and does not change it ever, even in the face of mounting evidence. This teacher is respected, albeit perhaps not for this unwavering lunacy.

If I were to rate, I would feel as though I owed a debt of thoroughness to someone, certainly to multiple someones. My goal would be to give my personal best evaluation and ranking to the course. I do agree that I would know from the get-go if the course were a top fifty, second fifty or outside the top 100, but the precision of it all would require a second tour (although perhaps not a golfing one.)

Just Manny being Manny.

[John Kavanaugh]

Often times it becomes rude even to spend to much time with a car salesman.  Just make up your mind and move on.  Does anyone really go back for a second test drive to see if they missed some nuance that might help in their decision.  Drive the car in the wind, rain and snow?

If it is rude to reduce the living wage of a car salesman by bogarting his time with your needy little insecurities imagine what it is like for a golf pro dealing with raters who require a return visit. 

[Jud_T]

Maybe the raters should volunteer to work on the grounds crew gratis for a week while they get multiple plays in...

[Ronald Montesano]

Fair points well raised. My thinking would be to spend the day at the joint and head out for a drive round before I actually play (depending on the professional's wishes) and take notes. It would probably take me 1-2 hours. I would then have ample opportunity to play the shots I suspected would work and then would request a second drive-round post-golf, for any other house cleaning. I might even drive the course backward (unless it were walkers-only.) See, I really like golf courses.

If professionals find raters to be pariahs, then this plan would be amended.

[John Kavanaugh]

Fair points well raised. My thinking would be to spend the day at the joint and head out for a drive round before I actually play (depending on the professional's wishes) and take notes. It would probably take me 1-2 hours. I would then have ample opportunity to play the shots I suspected would work and then would request a second drive-round post-golf, for any other house cleaning. I might even drive the course backward (unless it were walkers-only.) See, I really like golf courses.

If professionals find raters to be pariahs, then this plan would be amended.

At a private club this type of intrusion on the golfers enjoying a round is uncalled for.  One of the little things I hate most in life are the members who eat Sunday Brunch and then drive the course backwards with their young grandchildren in tow. 

I'm glad you really like golf courses but I have to question your respect for golfers privacy.  Would you really go through this exercise at any of the top ten courses in the country?  Should any other course or its members be respected less?

[Ronald Montesano]

This is where our roads diverge in the woods. I seldom set foot on the type of courses you frequent; there are one or two exceptions.

If I were invited to rate a course, I would expect a modicum of civility from the staff and membership that invited me. At least, they should want me there to provide a free service to them (on my own dime, let's remember.)

I would make my pre-round drive at 6:30 am, when I doubt that there are members on the course. I would make my post-round drive later in the evening, when your membership has ceased its play.

Unlike your aged tour guides, I would make every effort to move the cart in as inconspicuous a manner as possible. And I would be driving solo, so there would be no conversation nor brightly-colored taffeta dresses.

[C. Squier]

Give me a break. Nobody on this website has ever played a course for the first time, walked off 18 and said, "boy, I really don't know if that was good or not."

Ron, a course owes raters nothing. Certainly not a day full of a non-member driving all over unescorted. It's rating a golf course, not discovering the purpose of life.