Golf Digest's 2013-14 Top 100 Rankings

[Tom_Doak]

Okay, I went back to the data link Jim provided for 2011-12, and here are the results.

Three of the courses Tommy picked as most strategic are also among the top ten for Resistance to Scoring:

Pine Valley               8.91 RS  -  9.03 SV  [the highest score in both categories]
Merion                    8.51 RS  -  8.56 SV
The Ocean Course     8.52 RS  -  8.04 SV


Baltusrol Upper, which Tommy picked, was not in the top 100 last time, so I don't have the data for it.

Tommy's other six strategic courses were:

Cypress Point (7.68 RS) =           8.38 SV
Sand Hills  (8.04 RS) =               8.21 SV
National Golf Links  (7.71 RS) =   8.17 SV
Crystal Downs  (8.15 RS) =         8.15 SV
San Francisco GC  (7.51 RS) =     7.80 SV
Ballyneal  (7.40 RS) =                7.69 SV


The rest of the top 10 in resistance to scoring were

Oakmont  (8.91 RS)             =  8.65 SV
Augusta National  (8.75 RS)  =  8.86  SV
Winged Ft West (8.74 RS)     =  8.40 SV
Shinnecock Hills  (8.67 RS)   =  8.63 SV
Bethpage Black  (8.56 RS)     = 8.07 SV
Oakland Hills  (8.44 RS)        = 8.09 SV
Whistling Straits  (8.41 RS)    = 8.15 SV


My observations -- and feel free to disagree:

1.  The highest Resistance-to-Scoring courses are generally getting higher marks for Shot Values than the strategic courses are.  In particular, Oakmont and Winged Foot West, which do not seem especially strategic courses to me, beat out any of the courses Tommy nominated as strategic, except for Pine Valley and Merion which are also very high in Resistance to Scoring.

2.  Generally speaking, the accepted top 20 courses get top-40 scores in every category across the board.

3.  The vague "Shot Values" category is sometimes used to boost courses that panelists think belong in the list, but which aren't getting high enough scores in other categories ... especially resistance to scoring.

[Andy Troeger]

Moving a little slow this morning, but here's my list. I'm missing about half of the top twenty courses and am modern-heavy on what I have seen, so keeping that in mind...

Pine Valley
Shinnecock Hills
Pete Dye GC
Pebble Beach
Friar's Head
The Golf Club
Cypress Point
Blackwolf Run River
TPC Sawgrass
Bandon Trails

I gave Rock Creek, Wolf Run, and Black Mesa scores equal to the above, but they don't qualify...

[Mark Bourgeois]

Jim's correlation figure indicates that raters largely are treating shot values and resistance to scoring as the same and therefore the rankings capture the same things three times. If someone were to regress all these criteria scores on outcomes to make sense of the power of criteria to ptedict outcome -- well, they couldn't, not legitimately. It would be plagued by autocorrelation, which is a fancy way of saying all these criteria are measuring a lot of the same thing.

To your point #3, it would be interesting to try to figure out what raters actually are measuring. As I stated previously, they may not know themselves.

One possible and surprising conclusion is the criteria have far less impact than one would think. Raters decide what they like then score accordingly.

[Terry Lavin]

One possible and surprising conclusion is the criteria have far less impact than one would think. Raters decide what they like then score accordingly.

This strikes me as undebatable and unsurprising.

[Jud_T]

One possible and surprising conclusion is the criteria have far less impact than one would think. Raters decide what they like then score accordingly.

This strikes me as undebatable and unsurprising.

There's no other reasonable explanation for the significant differences between Digest and the other rags.  Unless of course their panel is a bunch of masochistic bedpost notching wanks... 

[Tom_Doak]

Andy:  Thanks for playing!  I've listed the Shot Values ratings from 2011-12 for the courses not already listed above in my post to Tommy Williamsen ... along with a select few other "Shot Values" ratings for comparison.  Enjoy!

Moving a little slow this morning, but here's my list. I'm missing about half of the top twenty courses and am modern-heavy on what I have seen, so keeping that in mind...

Pine Valley
Shinnecock Hills
Pete Dye GC - 7.91 SV
Pebble Beach - 8.41 SV
Friar's Head - 7.92 SV
The Golf Club - 7.93 SV
Cypress Point - 8.38 SV
Blackwolf Run River - 7.59 SV
TPC Sawgrass - 7.97 SV
Bandon Trails - 7.63 SV

I gave Rock Creek, Wolf Run, and Black Mesa scores equal to the above, but they don't qualify...

Some selected others:

Butler National - 7.82 SV
Baltusrol Lower - 7.95 SV
Bethpage Black - 8.07 SV
Harbour Town - 7.74 SV
Medinah No. 3 - 8.17 SV
Olympic - 8.13 SV

So, of the courses you listed, only top-5 stalwarts Pebble Beach and Cypress Point outpointed Medinah and Olympic for Shot Values.

[BCrosby]

"One possible and surprising conclusion is the criteria have far less impact than one would think. Raters decide what they like then score accordingly."

No kidding. Though I'm not sure how "surprising" it is.

Bob

[Jim Franklin]

Sand Hills is perfect with a historical pedigree to boot.  It is underrated on this list.

John:  Sand Hills is a 10 in my book so I have no problem with the first assertion in your post, or perhaps even the third.

However, why is Sand Hills' historical pedigree of any significance?  Why would some courses that are less than 20 years old be considered to have greater pedigree than others?  [And, btw, I thought they finally took pedigree or "tradition" out of the voting because it was such a clear fudge factor.]


As for the grass types, it is bizarre that DIGEST just recently gave Bandon Dunes its environmental award and praised it as a model of the type of conditioning that they changed their rating criteria to favor ... yet the courses which follow that model continue to get lower "conditioning" numbers from the actual raters.

Tom _ This confounds me as well. GD wants firm and fast yet their panelists do not reward  (nor penalize) courses that are firm and fast (or green and lush). I know some panelists do, but the vast majority do not. I am one of a thousand so it is a tough battle to fight.

[Tom_Doak]

"One possible and surprising conclusion is the criteria have far less impact than one would think. Raters decide what they like then score accordingly."

No kidding. Though I'm not sure how "surprising" it is.

Bob

As Jud alludes, this is not entirely true.  The emphasis on difficulty is the best way to explain the outlier courses on the Digest list which do not appear on other lists.  Their panelists have been told over and over again that Resistance to Scoring and Shot Values (which some misunderstand to equate to difficulty) are important.  They are also the first two criteria listed on the form, which emphasizes their significance; and, of course, the Shot Values rating is doubled.

However, it's generally true that if a panelist thinks a course is a 9, he's going to give it all 8's and 9's and 10's across the board, whether or not it is really one of the most difficult courses or one of the most scenic.

[Terry Lavin]

One possible and surprising conclusion is the criteria have far less impact than one would think. Raters decide what they like then score accordingly.

This strikes me as undebatable and unsurprising.

I
There's no other reasonable explanation for the significant differences between Digest and the other rags.  Unless of course their panel is a bunch of masochistic bedpost notching wanks... 

They're not the "walk in the park" crowd.  It's groupthink and culture, but of a different kind. It also happens to be the predominant brand in the game. Right or wrong, GD swings the big stick in the ratings game, despite nine plus pages of well-meaning angst.

[Sean Leary]

One possible and surprising conclusion is the criteria have far less impact than one would think. Raters decide what they like then score accordingly.

This strikes me as undebatable and unsurprising.

There's no other reasonable explanation for the significant differences between Digest and the other rags.  Unless of course their panel is a bunch of masochistic bedpost notching wanks... 

I think it works the same across all of the rating panels, not just GD. It also wouldn't surprise me if other rating panels over and under-rate courses based upon some perceived "wrongs" on the GD list...

[Mac Plumart]

heteroskedasticity

[Mac Plumart]

A good friend on mine is a college professor at a presitgous university and teaches economics.  One of his main observations when he has his students develop excel spreadsheets to compute and/or predict economic situations is that they can whip out the spreadsheets in a much more rapid fashion than he can, but the forumulas they use within the spreadsheets are generally FUBAR'd.

Just because someone has the mathematical skills to make calculations, doesn't make the calculations valid or worthy.  A common flaw in high IQ people is that they believe the world can be described by formulas.  And, yes, there are specific instances where formulas can predict or explain things.  Generally these things are rooted in science...hard and fast science...with hard and fast rules...32 feet per second, f=m*V^2, etc.  BUT when trying to predict human behavior and/or human reaction to things, there are no proven scientific formulas.  However, there are some general guidelines and princples that can be used to understand WHY humans make the decisions they do.

I believe these rules and guidelines can be used to understand the golf rankings AND the information in the first paragraph I wrote might be able to be used to grasp why mathematics can not capture golf course greatness and/or preceived greatness.

[Jim Colton]

Here are some more correlations from the last go 'round:

Shot Values (SV) vs.:
- Resistance to Scoring (RS) 82.1%
- Design Variety (DV) 87.4%
- Memorability (Mem) 84.3%
- Aesthetics (Aes) 56.2%
- Conditioning (Con) 39.1%
- Ambience (Amb) 76.4%
- Overall Score (Tot) 94.1%

RS vs.:
DV 55.5%
Mem 51.9%
Aes 26.7%
Con 33.3%
Amb 44.2%
Tot 70.8%

DV vs.:
Mem 93.3%
Aes 72.4%
Con 34.0%
Amb 73.9%
Tot 91.4%

Mem vs.:
Aes 85.6%
Con 29.5%
Amb 77.4%
Tot 92.5%

Aes vs.:
Con 29.1%
Amb 65.5%
Tot 76.1%

Con vs.:
Amb 50.5%
Tot 53.7%

Amb vs Tot: 71.3%

[Mark Bourgeois]

Mac...

Heteroskedasticity in what way? Seems to me the issues potentially are autocorrelation (the criteria overlap considerably in what they're measuring) and, per the current discussion, specification error (the criteria don't capture what raters really are rating). I guess regardless of error we might end up in the same place because one fix for heteroskedasticity is to substitute the independent variables used.

Assuming we were doing any kind of regressions, I mean. The course's total score is just the sum of the individual criteria scores. Don't need a weather vane to know which way that wind blows. I think what we'd want to do is regress the independent variables on each other, for example regress shot values on resistance to scoring to see just how much shot values is explained by resistance to scoring. Although Jim's simple regression is good enough for me: the two are pretty much the same thing.

[Mark Bourgeois]

Shot Values (SV) vs.:
- Resistance to Scoring (RS) 82.1%
- Design Variety (DV) 87.4%
- Memorability (Mem) 84.3%

DV vs.:
Mem 93.3%

Mem vs.:
Aes 85.6%
Tot 92.5%

My my my, what a mess.

You could pretty much chunk everything but memorability. "The 100 Most Memorable Courses in America."

[Mac Plumart]

Mark...

I really and truly do not think these lists are as scientific as you are hoping they are.

Yes, you could do this work you are suggesting and if that is a fun exercise...go for it.  But Jim's correlation list sheds a lot of light on things.  BUT is this really a science experiment to uncover the secret formula of golf course greatness or more of an exercise in GD rater behavior?

For instance, is that fact that Shot Values are the most highly correlated variable to Overall score an indication of the value Shot Values are to course greatness or is it, as has been suggested, a Fudge Factor that raters use to make their Overall Score equal the score they WANT the course to achieve.



PS....heteroskedasticity in the same manner that angle of reflection equals angle of refraction.  

[Andy Troeger]

Or could shot values be the most correlated because they are worth twice as many points as any other category 

[Mac Plumart]

BOOM!


And if I was going to pick a "Fudge Factor" category, I want double points on that bad boy.  But that's just me.   

[Mark Bourgeois]

PS....heteroskedasticity in the same manner that angle of reflection equals angle of refraction.  

That's it, I am going to beat you like a rented mule in the 2013 Putting Championship. No quarter given.

And after I down the last drop from the magnum of Petrus you will serve at my dinner I am going after Andy for his obvious effort to troll me! 

But Andy -- no! don't fall for it! but I must! I must! -- yes, of course it's in lock step with the overall results. (Big boo-boo for any individual criterion to correlate strongly with other criteria, though.) My point really though is you could have everyone rate just "memorability," drop everything else, and the results probably wouldn't change much.

BOOM goes the dynamite!

[Mark Bourgeois]

Mark...

I really and truly do not think these lists are as scientific as you are hoping they are.

Mac...

I don't think you're getting my point. I am trying to understand Golf Digest's approach on their terms not mine. If I am fighting a battle it is against pseudo-science and the abuse of statistics. GD is but a bug on my windshield. 

For 100 Greatest:

To arrive at a course's final score, we first throw out "outlier" evaluations at the high and low end. (Statistician Dean Knuth, creator of the United States Golf Association's Slope and Course Rating system, does the math for us. He tosses any evaluation that's more than two standard deviations from a course's mean score.) We then total the course's average scores in the seven categories, double-weighting the Shot Values category. A course needs 45 evaluations over the past eight years to be eligible for America's 100 Greatest. The minimum ballots for 100 Greatest Public is 25, for Best in State 10.

Read More http://www.golfdigest.com/golf-courses/2011-05/100-greatest-golf-courses-methodology#ixzz2GeIKspGM

For 75 Toughest:

HOW WE DID IT
The last time Golf Digest ranked America's Toughest Courses, in 2007, a group of editors got together and compiled the list based on their experiences. This year we took a scientific approach. We sent a survey to the Golf Digest Panelists, the roughly 1,100 avid golfers who create our 100 Greatest Courses rankings, and asked them to list the 15 courses that wear them out the most. The survey ballot included all 50 courses from our 2007 list, the 100 courses with the highest combined USGA Course Rating and Slope Rating scores, and the 100 courses that panelists rated highest in Resistance to Scoring—one of the seven criteria they use when evaluating 100 Greatest candidates. We also encouraged panelists to include write-ins. Courses earned points for appearing on panelists' top-15 lists: five points for being first-, second- and third-toughest, four points for fourth-, fifth- and sixth-toughest, and so on. The courses appear in order of their point scores. —Peter Finch

Read More http://www.golfdigest.com/golf-courses/2012-01/americas-75-toughest-courses#ixzz2GeIn3hft

You know, it would be interesting to compare the 75 Toughest with the 100 Greatest as ordered by resistance to scoring. I take it those who make the former list but not the latter are "tough but unfair"?

[Andy Troeger]

Mark,
I think we need to analyze the Irish Niblick point totals versus the Aints compared to all other weeks of the fantasy football season with the same effort as all this stats stuff on the Digest list!  

[Tommy Williamsen]

One possible and surprising conclusion is the criteria have far less impact than one would think. Raters decide what they like then score accordingly.

For me that is just not always true.  I really dislike Bulle Rock in MD.  It just is no fun and I've seen most of the holes before. I don't ever want to go back.  Twice was more than enough. Yet, when I sat down to score it, it did rather well.  I was disappointed.  Rating golf courses is subjective to be sure.  Nonetheless, having to rate the course according to certain criteria helps objectivize it a bit.

[Mark Bourgeois]

Mark,
I think we need to analyze the Irish Niblick point totals versus the Aints compared to all other weeks of the fantasy football season with the same effort as all this stats stuff on the Digest list!  

It is going to take a long time to forget who dropped a 163-point bomb on me but meanwhile: Aints & Irish Niblick 4, River Monts 1.

[Tommy Williamsen]

Okay, I went back to the data link Jim provided for 2011-12, and here are the results.

Three of the courses Tommy picked as most strategic are also among the top ten for Resistance to Scoring:

Pine Valley               8.91 RS  -  9.03 SV  [the highest score in both categories]
Merion                    8.51 RS  -  8.56 SV
The Ocean Course     8.52 RS  -  8.04 SV


Baltusrol Upper, which Tommy picked, was not in the top 100 last time, so I don't have the data for it.

Tommy's other six strategic courses were:

Cypress Point (7.68 RS) =           8.38 SV
Sand Hills  (8.04 RS) =               8.21 SV
National Golf Links  (7.71 RS) =   8.17 SV
Crystal Downs  (8.15 RS) =         8.15 SV
San Francisco GC  (7.51 RS) =     7.80 SV
Ballyneal  (7.40 RS) =                7.69 SV


The rest of the top 10 in resistance to scoring were

Oakmont  (8.91 RS)             =  8.65 SV
Augusta National  (8.75 RS)  =  8.86  SV
Winged Ft West (8.74 RS)     =  8.40 SV
Shinnecock Hills  (8.67 RS)   =  8.63 SV
Bethpage Black  (8.56 RS)     = 8.07 SV
Oakland Hills  (8.44 RS)        = 8.09 SV
Whistling Straits  (8.41 RS)    = 8.15 SV


My observations -- and feel free to disagree:

1.  The highest Resistance-to-Scoring courses are generally getting higher marks for Shot Values than the strategic courses are.  In particular, Oakmont and Winged Foot West, which do not seem especially strategic courses to me, beat out any of the courses Tommy nominated as strategic, except for Pine Valley and Merion which are also very high in Resistance to Scoring.

2.  Generally speaking, the accepted top 20 courses get top-40 scores in every category across the board.

3.  The vague "Shot Values" category is sometimes used to boost courses that panelists think belong in the list, but which aren't getting high enough scores in other categories ... especially resistance to scoring.

I am not quite sure I understand what this means or even if it is a bad thing.  It makes sense that the top courses would have high scores across the board.  That is why they rank so highly.  I agree that many panelists have differing ideas about what constitutes shot values.  But we are closer to agreeing a definition than the Congress is on the fiscal cliff. It makes sense to me that shot values are doubled.  Resistance to scoring should also be part of the equation.  Nobody wants a pushover.  Where it may fall down is in courses like Shoreacres.  It is a brilliant design, it just isn't very difficult anymore.