Jim Colton's Adjustments to Golf Digest

[Mike Sweeney]

It got lost on another thread. Here is a cool spread sheet posted by Jim:


Lemonade from Lemons? A lot of discussion about the Golf Digest list and all of its flaws. I believe some of it is simply due to the crude nature of the fixed formula that defines "greatness". This tool allows you to set your own category weights and produce a custom Top 100/200 and observe which courses are most impacted.
https://docs.google.com/spreadsheets/d/1_aK5BPjQb-I09azD4Ax1PoFF8ahlC3Z8XN1Hvb6CVcw/copy

[Mike Sweeney]

And his original Blog:


The list below shows that a Shot Values only approach would bring Hazeltine National, Harbour Town, Pasatiempo, Streamsong Red, Chambers Bay, Galloway National, The Course at Yale (up from 177 to 90!), Colorado Golf Club, Cal Club and Shoal Creek into the top 100 at the expense of Monterey Peninsula Shore, Laurel Valley, Flint Hills National, Hudson National, The Course at Black Rock, The Preserve, Double Eagle, Mayacama, Diamond Creek and The Quarry at La Quinta (down from 83 to 144).It's not perfect, but it seems to be heading in the right direction. Perhaps Design Variety and Memorability or Ambience need to play a small role as well. But here's the Shot Value-only list as food for thought. More to come as we dig deeper into the numbers.


http://www.wegoblogger31.com/2017/01/golf-digest-2017-18-rankings-food-for.html

[Tom_Doak]

It's a neat tool, but it still assumes that the panelists are rating the courses right on each of the categories.  Playing around with it a bit shows that things don't change as much as you'd think when you eliminate one or two categories, because of the panelists' tendency to give higher or lower scores across all categories depending on how much they liked the course overall.  Pine Valley, Augusta, Shinnecock and Cypress Point all seem to be in the top ten for every category!

[David Wuthrich]

Thanks for sharing this!

[Ben Sims]

It's a neat tool, but it still assumes that the panelists are rating the courses right on each of the categories.  Playing around with it a bit shows that things don't change as much as you'd think when you eliminate one or two categories, because of the panelists' tendency to give higher or lower scores across all categories depending on how much they liked the course overall.  Pine Valley, Augusta, Shinnecock and Cypress Point all seem to be in the top ten for every category!

Doak magnum...splash one.


I wrote the pasted test below on a Facebook discussion. It's crude and not particularly well written. But it's as honest an opinion as I can offer. I am personally saddened and disgusted that Golf Digest's rankings, due to their popularity and sheer circulation, could one day be historically conflated as an arbiter of golf architecture. I used Jim's "new" list of the GD shot values only.


"Fun with numbers. I wanted to analyze the courses that gained or lost the most spots in Jim's new list. For round measurements, I used 12 as the standard deviation of the 97 courses that gained or lost spots. In total, 32 courses were off by 12 spots or more. Seven in the top 50, twenty five in the bottom 50. Of those 32, 14 were off by two standard devs. Eight courses were more than 24 spots different than their GD ranking and fell out of Jim's shot values top 100.


Here's some editorial opinions based on the numbers below. In the top fifty, gaining or losing a lot of spots was highly correlative with whether the course was modern or classic. In fact, it's 100% correlative depending on whether you consider a restoration as "modern." And plainly, hosting a major really seems to inflate shot values. The two courses in the top fifty that lose a ton of spots are both modern and very exclusive.


In the new bottom fifty, the differences are less marked. Until you get to the last few spots, the classic vs modern correlation seems to still exist, at least superficially. But then you see some modern courses that gained a bunch of spots. What do most of them have in common? You guessed it! Professional golf tournaments. Chiefly among those are Valhalla, Harbour Town, and Chambers Bay. An additional complication to the superficial analysis is the difference in gaining and losing spots between highly reputable classic courses like Yale and Pasatiempo (gainers) vs Maidstone and Kittansett (losers). I could offer some very opinionated reasons for those four courses gaining and losing as many spots as they did (whilst all being classic courses), but it would venture off path.


There's lots of other observations to be made from this list. It highlights--for me--the weakness of shot values as it relates to the quality of a golf course. From a mathematical perspective however, the fact that shot value accounts for a relatively small percentage of the total GD ranking as compared to how accurately it predicts the finals rankings is a very odd development. Only six courses were top 100 in shot values that gained two standard dev's from their original GD ranking. As you can see below, there were only 8 courses that did the opposite. That's 14 courses country-wide that the golf digest shot-values criteria completely misses on. Impressive indication of Golf Digest's rankings criteria? Or a indication that prior bias drives rankings? You judge and we'll debate.


Number of courses that change ranking: 97
Avg delta 11.96
Standard dev 12.78


Top 50
1 dev +: 5; Bethpage, medinah, oak tree, Inverness, Plainfield
1 dev -: 2; shadow creek, alotian


Bottom 50
1 dev + : 15; Olympia fields, aronimink, Valhalla, crooked stick, hazeltine, harbor town, pasatiempo, sahalee, streamsong red, chambers bay, Galloway, Yale, Colorado golf club, cal club, shoal creek
1 dev - : 10; Bandon dunes, castle pines, sebonack, kinloch, old Mac, canyata, Milwaukee, Maidstone, kittansett, Arcadia bluffs


2 standard dev + : 10; oak tree (24), Inverness (40), Plainfield (27), Olympia fields (25), hazeltine (33), harbour town (52!), pasatiempo (29), chambers bay (42), Galloway (28), Yale (87!!),
2 standard dev - : 4; Milwaukee (24), Maidstone (24), kittansett (30), Arcadia bluffs (32)


2 standard dev losers that fell outside top 100: 8; MPCC Shore (36), Hudson natl (27), Black Rock (29), Preserve (30), Double Eagle (40), Mayacama (31), Diamond Creek (47), Quarry @ La Quinta (57)

[Sean_A]

It's a neat tool, but it still assumes that the panelists are rating the courses right on each of the categories.  Playing around with it a bit shows that things don't change as much as you'd think when you eliminate one or two categories, because of the panelists' tendency to give higher or lower scores across all categories depending on how much they liked the course overall.  Pine Valley, Augusta, Shinnecock and Cypress Point all seem to be in the top ten for every category!

You mean panelists have a gut feeling toward courses then make the scores fit their gut?  I would never have thought this could be the case.


Ciao

[Mike Sweeney]

One more addition from Jim:


One last post on the latest Golf Digest top 100/200 rankings. In the past, I've looked at each category's contribution to a given course's total score as an psuedo-look at its DNA. With seven different categories, there are different ways to make the list. Some get by on Shot Values; others on Resistance to Scoring or Conditioning. Some are consistently strong across all categories (Colorado Golf Club's DNA is the closest to Golf Digest's formula).


http://www.wegoblogger31.com/2017/01/golf-digest-2017-18-rankings-inside.html

[Niall C]

Dear Saddened and Disgusted,


It's only a list. Not only that but it's a golf course rankings list, hardly threatens world peace. Suggest you calm down a bit.


Niall

[Mike Sweeney]

Dear Saddened and Disgusted,


It's only a list. Not only that but it's a golf course rankings list, hardly threatens world peace. Suggest you calm down a bit.


Niall

Niall,


There are a subset of data geeks here at GCA. We are just having fun with the data and it is a separate outlet from the GCA. Granted, the fact that  the artistic nature of Golf Architecture has intersected with data is weird, but it could actually lead to new ideas for the industry. Or maybe we are just data geeks


Either way, it is snowy, cold, and I have a bit of the flu in NYC. Tired of Trump-talk, can't I banter with some data geeks on hairy bunkers please!!??

[Paul Jones]

It's a neat tool, but it still assumes that the panelists are rating the courses right on each of the categories.  Playing around with it a bit shows that things don't change as much as you'd think when you eliminate one or two categories, because of the panelists' tendency to give higher or lower scores across all categories depending on how much they liked the course overall.  Pine Valley, Augusta, Shinnecock and Cypress Point all seem to be in the top ten for every category!

Tom,


Which category(s) would you remove Pine Valley, Augusta Natl, Shinnecock and/or Cypress Point from the Top 10?


Paul

[Mark_Fine]

Let me let you all in on a little secret, there are no flaws in GD's rating criteria.  And I mean this in all seriousness.  The only "flaws" are with the raters themselves.   

For example, if Tom Doak used the GD criteria to rate the Top 100 courses in the country/world, I believe he would still figure out a way to come up with the same list of courses that HE believes are the best.  He would make that happen with his collective ratings because the numbers are all relative anyway. 

These rankings are just subjective summaries of lots of skilled and unskilled opinions. 

[Jeff_Brauer]

Mark,


Yes, they all sort of fudge the numbers to justify their opinion, at least that would be human nature....not that there is anything wrong with that.....


In the end, anyone calling for smarter raters is really calling for people who think just like them, no? 


An average of ratings probably reveals the truth of how the masses think (you know, those deplorables who like carts, pretty beer cart girls, green turf, etc.) whether the creator of the rating agrees or not.  I understand the creator has probably given a lot more thought to a system, whatever it is, but whether based on deep thought or numerical system, in the end, emotion wins out.


I have seen it in my own design work. I do 4 or 5 routings and try to rank them.  Even if one comes out on top, I often pick another just because I intuitively like it.  That said,  it does make me think harder as to why I like one that isn't statistically "better."  While it is natural to bend stats to your initial opinion, I think changing your mind when the numbers show it in many cases isn't a bad response.


I have always said that sooner or later, math comes into it.  You want a fairway to hold shots, so you measure what cross slope holds them on various heights of cut.  You want the first putt to not run off the green, so you measure ones right on the edge, and try to keep slopes less than that, even if that number for a 13 stimp green is 2.4334%......etc.

[Mark_Fine]

Jeff,
The rankings are nothing more than opinions.   There is no "fudging" and GD doesn't force anything.   If they did I would have a real problem with it.  They don't say that if a course is 7000 yards long it has to have a difficulty factor of 8?  Or it is is 6500 yards it is 6. There is plenty of room in all the criteria to draw one's own opinion which goes back to my initial point.  It is all about the rater more than the criteria. 

[Joel_Stewart]

Let me let you all in on a little secret, there are no flaws in GD's rating criteria.  And I mean this in all seriousness.  The only "flaws" are with the raters themselves.   

Disagree. Here's the problem, the current method does not allow separation.


I recently played a Fazio course. Aesthetically and ambiance are fantastic, 8-9. Conditioning among the best in the country 9. Shot vales, very very good, 7-8. So as you can see, it will rank easily in the Top 100.  With that said, I would rather play PV, NGLA, or many other courses 10-0 over this Fazio.


Panelists go to Pebble Beach and play those courses and then venture to The Preserve. They give The Preserve comparable numbers even though it's not even close to the courses on the 17 mile drive.

[Mark_Fine]

Joel,
Why would you prefer PV and The National over this course?  If you truly feel that way it should be reflected some how in your numbers. 

There is no strict guideline on what "number" you have to give a particular category.  Your final tally should reflect how you feel about each golf course you look at.  There is plenty of opportunity for separation.

Reviewing golf courses is a lot like reviewing wines.  There might be a set "criteria" that most use to determine the quality of a wine but at the end of the day it comes down to the person doing the review and how knowledgable and experienced they are.

[Niall C]

Mike
 
My Saddened and Disgusted comment was aimed at Ben who described himself as such in his post, and I just thought that was way, way, over the top for what is a fairly innocuous ranking list. I was maybe thinking that my comment might help reduce his blood pressure somewhat but in hindsight I suspect it might do the opposite, red rag to a bull and all that, but hopefully not.


So have at it as I think they say. In that spirit, I always thought the inclusion of things like ambience and history etc were a bit bogus and just weighted the ranking against the architectural merits/demerits of the course.
 
Niall

[Tom_Doak]

Tom,

Which category(s) would you remove Pine Valley, Augusta Natl, Shinnecock and/or Cypress Point from the Top 10?

You mean I've got to go back and think through them all like a GOLF DIGEST panelist?  Ick.  OK:


SHOT VALUES:  I guess all four might make my top ten in America here.  We spent a long time on "shot values" last week and I'm still not sure how you really rate them across a whole course, but my emphasis would not be on difficulty, so Cypress Point might still get into the top ten.  I might decide that Pine Valley or Augusta are TOO severe though.


RESISTANCE TO SCORING:  Cypress Point probably isn't in America's top 200 in resistance to scoring, another reason I think this category should be abolished.  Pine Valley and Augusta almost certainly are.  Shinnecock is tough, but I'm sure there are a bunch of courses where it's harder to score.


DESIGN VARIETY:  I would not put Augusta in that conversation:  too many downhill and uphill holes, lack of short par-4's.  Shinnecock would probably not be in my top ten, though it's good.


MEMORABILITY:  Augusta has a few very memorable holes [6, 9, 10, 12, 13, 15], but it only succeeds in this category because you've seen all the holes for years before you even get there to play.  Cypress Point and Pine Valley are stars in this category; Shinnecock, again, very good but maybe not top 10.


AESTHETICS:  All depends on what you like.  If you're a sucker for golf along dramatic coastlines, only one of these should make it.  But I would think that very few people would actually have a value system that put BOTH Pine Valley and Augusta in the same elite category of aesthetics ... they seem like they would appeal to different groups of people.


AMBIENCE:  All depends on what you like here, too.  You could easily make the case that The Valley Club of Montecito is a much more pleasant place to play golf than any of these famous ones, and Cypress Point would be the only one of the four that could compete in that person's mind.  If you like history, Garden City and Myopia have their own charm and their own unique character with a vastly different sort of membership.