Golf Digest's 2013-14 Top 100 Rankings

[Mark Bourgeois]

Are there any magazines that DON'T tell their raters what to look for?

Jeff,

Pretty astute observation.  I don't understand how the process is so "strong" when raters are being told what to look for.

Criteria simply are definitions. The more thoroughly and well explained those criteria are, the less potential for misunderstandings and misinterpretations.

You guys would pillory Golf Digest if rater interpretation of the criteria varied wildly. Explication reduces unintended variation. Golf Digest's criteria and the length to which they explicate them indicate a well-thought, time-honored approach. As befits golf's authoritative ranking system.

[Mac Plumart]

Are there any magazines that DON'T tell their raters what to look for?

Golf Magazine

[Kevin Lynch]

Are there any magazines that DON'T tell their raters what to look for?

Jeff,

Pretty astute observation.  I don't understand how the process is so "strong" when raters are being told what to look for.

Criteria simply are definitions. The more thoroughly and well explained those criteria are, the less potential for misunderstandings and misinterpretations.

You guys would pillory Golf Digest if rater interpretation of the criteria varied wildly. Explication reduces unintended variation. Golf Digest's criteria and the length to which they explicate them indicate a well-thought, time-honored approach. As befits golf's authoritative ranking system.

Mark,

I don't have a problem with Golf Digest giving the criteria to their raters.  As Tommy Williamsen illustrated with his Bulle Rock example, having those definitions can bring you some objectivity when a course doesn't fit your subjective style of course. 

Unfortunately, seeing the overall ratings by category, it is pretty obvious that the ratings for some criteria are being influenced by an overall impression (e.g. Augusta being the firmest, fastest in the land). 

If everyone objectively rated the courses like Tommy did with Bulle Rock, I don't think Augusta would be #1 in conditioning.  But, I understand that it is human nature that there will be some "creep" towards an overall grade, regardless of how defined the "independent" criteria.

Perhaps Golfweek has the right idea, where the criteria are defined and rated individually, but the "overall" rating is still left up to the rater (not the function of math).  The criteria gives them things to consider, but it doesn't dictate a mathematical weighting of each factor to overall importance.

Again, none of this is meant to bash raters.  It is a nearly impossible exercise to quantify architecture.  I was just struck by the statistical curiosity of ANGC leading so many "independent" categories.   I respect their opinions and almost wish that Golf Digest would do a huge compilation of their qualitative comments for each course.  That would probably be the most meaningful system of all.

[Sean Leary]

The next 100 is now up on the GD website for those without tablets or a dollar or both.

[John Kavanaugh]

The next 100 is now up on the GD website for those without tablets or a dollar or both.

I was once accused of joining Dismal River out of spite but now with it officially being recognized as one of the top 200 courses in the nation my true identity as a belt knotcher has been revealed.  Every time I think I have had it with raters and ratings they pull me back in.  Going from double secret probation to the Dean's list in one semester.  Wow.

[Mike Sweeney]

The next 100 is now up on the GD website for those without tablets or a dollar or both.

I was once accused of joining Dismal River out of spite but now with it officially being recognized as one of the top 200 courses in the nation my true identity as a belt knotcher has been revealed.  Every time I think I have had it with raters and ratings they pull me back in.  Going from double secret probation to the Dean's list in one semester.  Wow.

To be honest, I still think/hoping you might resign from Dismal on the day that the Doak course opens out of spite. 

[John Kavanaugh]

The next 100 is now up on the GD website for those without tablets or a dollar or both.

I was once accused of joining Dismal River out of spite but now with it officially being recognized as one of the top 200 courses in the nation my true identity as a belt knotcher has been revealed.  Every time I think I have had it with raters and ratings they pull me back in.  Going from double secret probation to the Dean's list in one semester.  Wow.

To be honest, I still think/hoping you might resign from Dismal on the day that the Doak course opens out of spite. 

These are odd times.  I had thought that I would at least have the Old Course to myself while all the lookylous were checking out the Doak.  Now I might not even be able to get a room.  I hope those memberships at Awarii Dunes are still cheap.

[Chris Johnston]

The next 100 is now up on the GD website for those without tablets or a dollar or both.

I was once accused of joining Dismal River out of spite but now with it officially being recognized as one of the top 200 courses in the nation my true identity as a belt knotcher has been revealed.  Every time I think I have had it with raters and ratings they pull me back in.  Going from double secret probation to the Dean's list in one semester.  Wow.

To be honest, I still think/hoping you might resign from Dismal on the day that the Doak course opens out of spite. 

These are odd times.  I had thought that I would at least have the Old Course to myself while all the lookylous were checking out the Doak.  Now I might not even be able to get a room.  I hope those memberships at Awarii Dunes are still cheap.

Lookylous   

[Mac Plumart]

I had thought that I would at least have the Old Course to myself

Not a chance, if I'm there JK.  I love that course.

[Chris Johnston]

Old Course?

You guys lobbying for names?

Love the DRUG&AC!

[Bruce Wellmon]

Is the 201 marker at The Dismal River Golf and Yacht Club going to be replaced with a 199, moved, or am I going to be 1-2 yards long? Or short? I'm so confused
 

[Kevin Lynch]

Sven,

  This spreadsheet has the category scores for each course: https://docs.google.com/open?id=0B9I2sGAMtMU9Y0R6NDhwZTJMSnc

Another interesting anomaly when comparing certain categories for Winged Foot West & East:

Aesthetics:  West - 33rd,  East - 95th
Conditioning:  West - 25th, East 89th
Ambience - West - 6th, East - 32nd

Is there a separate staff for the East Course than the West Course?  Those East employees better get to work on upping the ambience in their section of the locker room and clubhouse (if they want to keep up).  Better fire the East maintenance crew and hire some guys from the West crew while you're at it.  A little more proof that these last three criteria are the "fudge factors" influenced by the overall fame of the course.



And, should I cease my plans to travel to Bandon Resort?  I like firm & fast courses, but they rated 79th (PD), 81st (OM), 91st (BD) and 94th (BT) on the Conditioning Scale (supposedly a measure of F&F).   How over-watered is that place?    Perhaps I should start looking for nicely browned courses like Muirfield Village (6th) and Alotian (7th).

[Mac Plumart]

Sven,

  This spreadsheet has the category scores for each course: https://docs.google.com/open?id=0B9I2sGAMtMU9Y0R6NDhwZTJMSnc

Another interesting anomaly when comparing certain categories for Winged Foot West & East:

Aesthetics:  West - 33rd,  East - 95th
Conditioning:  West - 25th, East 89th
Ambience - West - 6th, East - 32nd

Is there a separate staff for the East Course than the West Course?  Those East employees better get to work on upping the ambience in their section of the locker room and clubhouse (if they want to keep up).  Better fire the East maintenance crew and hire some guys from the West crew while you're at it.  A little more proof that these last three criteria are the "fudge factors" influenced by the overall fame of the course.



And, should I cease my plans to travel to Bandon Resort?  I like firm & fast courses, but they rated 79th (PD), 81st (OM), 91st (BD) and 94th (BT) on the Conditioning Scale (supposedly a measure of F&F).   How over-watered is that place?    Perhaps I should start looking for nicely browned courses like Muirfield Village (6th) and Alotian (7th).

Excellent work, Kevin.  Excellent!

[Kirk Moon]

Statistical concern re the GD ranking methodology:

I am going to assume that raters are using a 1-10 scale for each category and that they report their scores for each category using integer values with no decimal points (e.g. each score is a whole number between 1-10).  If this is wrong, I would appreciate it if a GD rater could tell me how they actually do it.  But for discussion's sake ......

I notice GD is reporting out the mean scores for each category at each course (and the final summary score) to a level of "accuracy" of four digits to the right of the decimal place.  The other thing I notice is that the scores are fairly tightly grouped with relatively small numerical differences between courses (both in the individual scoring categories and in the overall scores.)

The rules of statistics indicate that the final result of a statistical evaluation cannot have a meaningful level of accuracy any higher than the level of accuracy of the least accurate input value.  In other words, if the scores that the raters are entering for each category are only "accurate" at the whole number level (and I imagine most of us would agree that they are probably not even truly accurate at that level given the subjectivity of the process), then the mean value calculated from all submitted scores is also really only accurate at the whole number level as well even though it is possible to calculate a number for the mean of all submitted scores that has an arbitrary number of digits to the right of the decimal place.  The "accuracy" of the numbers to the right of the decimal place is fictitious even though mathematically correct.   This is basic statistics. 

What this means in plain English is that if GD raters are using a scoring system basically similar the one I describe above, their ranking method is not statistically valid. 

The only number that is really statistically "accurate" for the score for each golf course in each category is the one to the left of the decimal place after rounding to eliminate the decimals.  If proper statistical principles are applied, the granularity of their ranking system falls apart.  The only statistically valid output of their process would be a list of courses that fall into groups with the same integer value summary scores rather than a list of courses rank ordered by numbers carried out to four decimal places of "accuracy". 

If someone has time to kill and wants to make the list consistent with statistical analysis, go through the data for the list of the top 100 courses on their web site and round each score for each category for each course to the nearest integer value.  Then apply the addition formula (which I believe gives 2x weighting to the "shot value" category.  You will end up with a whole number value for each course.  THIS is the number that should be used for the ranking.  It will not allow separation of courses into a simple rank order list.  Many courses will end up with the same integer value total score.  This is not a problem.  This is an accurate reflection of reality given the degree of accuracy of the rater's input values. 

One could easily take the criticism a level further and argue that the categories chosen are not true independent variables (and therefore some may be redundant) and even that the ten point scale that GD employs for each category is too granular and that it is not possible for the raters to accurately and reproducibly score courses using those criteria at that level of detail. 

IMHO, the only kinds of a rating systems that have any chance of being meaningful would be ones that don't try to get too granular and that stick to few and/or fairly narrow criteria for comparison. 

[John Kirk]

Excellent posts by Kevin and Kirk.

Ballyneal is another course with a lower conditioning score that has fabulous playing characteristics.  It's a bit different than the Bandon courses.  The fairway grass is a bit longer and stands up a bit, so the ball tends to have a cushion under it.  Bandon is very tight.

Sven's list of Resistance To Scoring has some issues.  For instance, Pacific Dunes is ranked higher than Sebonack.  Sebonack is very difficult.  If Resistance to Scoring is truly the difficulty for a scratch player from the back tees, shouldn't the courses be ordered by back tee course rating?

[J_ Crisham]

Excellent posts by Kevin and Kirk.

Ballyneal is another course with a lower conditioning score that has fabulous playing characteristics.  It's a bit different than the Bandon courses.  The fairway grass is a bit longer and stands up a bit, so the ball tends to have a cushion under it.  Bandon is very tight.

Sven's list of Resistance To Scoring has some issues.  For instance, Pacific Dunes is ranked higher than Sebonack.  Sebonack is very difficult.  If Resistance to Scoring is truly the difficulty for a scratch player from the back tees, shouldn't the courses be ordered by back tee course rating?

John,   When we played Sebonack  it was the day prior to Shinnecock. We played both courses at about 6900 yds. In our group was a 2, a 6, and a couple of 9 hndcpers. Our high score at Sebonack was maybe 82. At Shinny nobody broke 80- we all were in the 83-87 range. Conditions were pretty similar with regard to wind, temp etc. Our group did not feel beat up by Sebonack in fact we thought it was pretty user friendly ,much easier driving corridors than Shinny and the severity of the greens was certainly less at Sebonack. I will freely admit that Shinny is my 2nd favorite course in the US but as far as Sebonack being very difficult we did not view it that way. Also I must add that our home clubs are not pushover courses so it may be a matter of what we are used to. From the plates at Shinny versus Sebonack I have to believe it's Shinny if we are talking difficulty- no knock on Sebonack- it's one hell of a course and top 5 modern for me- I just thought it was pretty user friendly.

[Kevin Lynch]

[
Excellent work, Kevin.  Excellent!

Don't give me too much encouragement, or I may keep combing over these category ratings for more head-shakers. I've already taken Jim's spreadsheet and added multiple columns for analysis. 

But the conditioning is the one that bugs me the most.  Somewhere, Tom Doak expressed his frustration that GD seems to be pushing for firm, fast & less plush, but then courses are punished for actually implementing it (ie all of Bandon, Pinehurst #2).

[Mark Saltzman]

Anyone know if the old scores for the courses with major renovations/restorations were tossed out? eg P2, LACC

What about for smaller restorations? eg Quaker Ridge

[J_ Crisham]

Anyone know if the old scores for the courses with major renovations/restorations were tossed out? eg P2, LACC

What about for smaller restorations? eg Quaker Ridge

Mark,  With regard to Quaker Ridge the old rating by Golf Digest was in fact tossed out.

[John Kirk]

That's interesting, Jack.  When I played the two courses during my memorable week on Long Island, I was playing at the top of my game and shot a stress free 78 at Shinnecock, a score that could have been one or two lower if I had listened to my excellent caddie on every shot.  I played Sebonack twice and shot something like 80-85, though the 85 was a very good round in a 15-20 mph breeze.

My wife was also playing her best.  She birdied the Redan and parred #11 at Shinnecock, great memories.

I loved Shinnecock Hills.  One of my greatest experiences golfing.  It's so fun to play, though it would demoralize you if you weren't hitting it straight.  The best part was looking across the field on my way back to the clubhouse, either the 16th or 18th hole, and seeing a father and young son on the course, the father instructing his son on the finer points of the game.  Great moment, and a reminder that these great clubs are family affairs first and foremost.

I think everybody has their own interpretation of scoring difficulty.  The greens at Sebonack are clearly more undulating.  Also, at Sebonack, the fairway bunkers were often positioned where I rarely had the option to challenge them.  Thanks for the response.

[jkinney]

I find the Golf Digest rankings to be increasingly at variance over the years with my own rankings, which are much closer to those of Golf Magazine......I'm simply aghast at LACC North being ranked lower after Hanse's restoration than it was prior, and this is only one example. Enough said.

[Andy Troeger]

As mentioned, Quaker's old ratings were thrown out. As far as I know, LACC and #2 the old ratings were not thrown out.

And LACC did move up six spots; expecting more than that with six years of old ballots in there is probably not realistic.

[Tom_Doak]

And, should I cease my plans to travel to Bandon Resort?  I like firm & fast courses, but they rated 79th (PD), 81st (OM), 91st (BD) and 94th (BT) on the Conditioning Scale (supposedly a measure of F&F).   How over-watered is that place?    Perhaps I should start looking for nicely browned courses like Muirfield Village (6th) and Alotian (7th).

Kevin:  If you just read "Conditioning" as "Speed of Greens," the GOLF DIGEST panelists will begin to make more sense.  That's about all they are rating, as much as they pretend otherwise.  They just don't seem to have any idea what good fairway turf is about.

P.S.  I loved your comparison of Winged Foot East and Winged Foot West.

[Mark Saltzman]

As mentioned, Quaker's old ratings were thrown out. As far as I know, LACC and #2 the old ratings were not thrown out.

And LACC did move up six spots; expecting more than that with six years of old ballots in there is probably not realistic.

Thanks, Andy.  Exactly my point (though I'm surprised to hear the old ratings weren't thrown out).

[Andy Troeger]

Mark,
I think the courses can request that the old ballots be thrown out if they feel the changes are significant enough, but for that to make sense they have to be willing to accept a lot of panelist play to get back to 45 ballots quickly.