Golf Digest New US Ratings Now Available

[Jeff Schley]

Ira,

Its all good, i understand why many have a preference to not end on a par 3.  But the Good Dr thought it was OK, so that's certainly more than enough for me.

As for John, if he's deluded himself into thinking he has some higher understanding from having arrived at a so-called better player status, (despite his inability to articulate why), my only advice for him would be to go fetch his shinebox. 

Allow me......

[Kalen Braley]

Thanks for the assist Jeff!   


P.S. As for the list, try importing into a xls and shrinking it down and then posting.

[Jeff Schley]

Thanks for the assist Jeff!   


P.S. As for the list, try importing into a xls and shrinking it down and then posting.

How can I post something from my own files as I thought it had to be housed on the web somewhere and I use the BBcodehelper to tag an image.  Thansk Kalen as always.

[Kalen Braley]

Thanks for the assist Jeff!   


P.S. As for the list, try importing into a xls and shrinking it down and then posting.

How can I post something from my own files as I thought it had to be housed on the web somewhere and I use the BBcodehelper to tag an image.  Thansk Kalen as always.

Hey Jeff,

I used to use Photobucket, but now use Imgur.  You can upload, host, and post stuff for free!

Here's a pic i posted a year or so of the the future Buck Club using the site

[Jeff Schley]

Ok cool..... i'm try and education myself on it for I'd like to customize the images I post as it is annoying as heck to see some huge image that should be much smaller.  Thanks.

[Jeff Schley]

BTW the top 3 in each category: 
Shot Values - Pine Valley - 8.95, ANGC - 8.67, Shinnecock - 8.63
Resistance to Scoring - Pine Valley - 8.88, Oakmont - 8.84, ANGC - 8.64
Design Variety - Pine Valley - 9.14, Cypress Point 8.85, ANGC - 8.73 
Memorability - ANGC -9.29, Pine Valley - 9.24, Cypress Point - 9.23 
Aesthetics - Cypress Point - 9.27, AGNC - 9.20, Pebble Beach - 9.00
Conditioning - ANGC - 8.98, Pine Valley - 8.71, Oakmont - 8.70
Ambience - ANGC - 9.40, Pine Valley - 9.22, Cypress Point - 9.17

[Jeff Schley]

Thanks for the assist Jeff!   


P.S. As for the list, try importing into a xls and shrinking it down and then posting.

How can I post something from my own files as I thought it had to be housed on the web somewhere and I use the BBcodehelper to tag an image.  Thansk Kalen as always.

Hey Jeff,

I used to use Photobucket, but now use Imgur.  You can upload, host, and post stuff for free!

Here's a pic i posted a year or so of the the future Buck Club using the site

Cool..... can you shrink the size to post it to scale so it fits without the sliding bars to view the entire image?

[Tom_Doak]

Golf digest is trying to double the number or raters to almost 1900 by they year 2020 and I think they have about 1100 now.  They presently have to have 45 ratings from their raters in the last 5 years to qualify and want to raise that to 70 which for statistics is good because the more data points you have the more regression you will have towards the mean and a normal distribution to mitigate the outliers. 
.
.
.

Also you have to take a look at their "raters", the qualification criteria is basically have a 5.0 handicap index and be willing to pay $1000 up front and $250 bucks a year in dues. They must submit a minimum of 12 (one rater told me), but others have told me 24 a year.

Statistically upping the minimum to 70 is going to help accuracy of the field on what their criteria is, however the criteria itself is hotly debated on this site and others.  Increasing their pool of raters is good otherwise they won't get the 70 scores needed every 5 years for some courses to qualify.

I apologize for picking on your post but your first paragraph quoted above is the logical fallacy of the whole ratings game.


The notion that more raters = a better result, in statistics, is based on the premise that all the data is equally valid.


But when the data consists of people's opinions of golf courses [or restaurants, or art, or anything else], then the results depend most on WHO YOU PANELISTS ARE.  And that is certainly NOT a case of "more = better".  Indeed, many of their more experienced panelists feel the same way . . . their vote is insignificant in a tidal wave of garbage in, garbage out.




I've never entirely agreed with GOLF DIGEST's criteria of what a great course should be, either.  I really don't believe there is a mathematical formula to describe it, but to make the rankings meaningful, they have to pretend that people's opinions somehow have statistical significance.  But at this point I honestly don't know whether it's the system or the people that have more to do with my disagreement with some of their results.


And I'm not picking on GOLF DIGEST here . . . I will be happy to point out the flaws in the other magazines' systems, when it's their turn.   

[Tom_Doak]

BTW the top 3 in each category: 
Shot Values - Pine Valley - 8.95, ANGC - 8.67, Shinnecock - 8.63
Resistance to Scoring - Pine Valley - 8.88, Oakmont - 8.84, ANGC - 8.64
Design Variety - Pine Valley - 9.14, Cypress Point 8.85, ANGC - 8.73 
Memorability - ANGC -9.29, Pine Valley - 9.24, Cypress Point - 9.23 
Aesthetics - Cypress Point - 9.27, AGNC - 9.20, Pebble Beach - 9.00
Conditioning - ANGC - 8.98, Pine Valley - 8.71, Oakmont - 8.70
Ambience - ANGC - 9.40, Pine Valley - 9.22, Cypress Point - 9.17

Speaking of statistical significance, any statistician would tell you there's no way the numbers should work out as above, where not a single outlier to the top ten overall is in the top three in ANY of the categories they rate.


I guarantee you there are some less highly-ranked courses in America that are better conditioned than ANGC or Pine Valley, no matter how you define conditioning -- and/or courses with better aesthetics or ambience than the three listed, or better Design Variety or whatever.


The reason you see results like the above is because panelists are afraid to upset the status quo. 


For the top ten courses, they start with a 9, and their ratings in each category land between 8 and 10, with rare exceptions.  For the next rung, they rate the seven categories from 7 to 9, averaging out around 8. 


But they will rarely give a 10 in any category to a course that isn't totally respected, for fear of being dismissed from the panel as an "outlier" . . . and the converse is that they rarely give a highly-ranked course a 4 or 5 in one of the categories even if it is totally deserved [like, say, Design Variety for The Olympic Club or Firestone, or Conditioning for Yale].


If you're wondering why the top ten doesn't change very much, this is why.  Panelists know which lines they are not supposed to cross.

[Ira Fishman]

Tom,


I agree that all rating systems are flawed, particularly where they try to leave the false perception of statistical precision. I have thought that an Ordinal ranking by enough knowledgeable players would be most compelling, but the definition of knowledgeable is tricky and the panel would have to be large enough so that a sizable number of them would have played the pool of courses that might be worthy—perhaps a few hundred courses in the US alone.


Aside from the Doak Scale as implemented by a panel of four, how would you design the most reliable rating system? Which of the existing ones comes closest?


Ira

[Jeff Schley]

TD I certainly see your point and agree on the surface, however what was the criteria for the present GD raters to be selected? I know several and none have said they had to attend any course or agree to any rubric for their scoring, they simply met the handicap requirement and some were "connected" or claimed other raters were, which is why not all potential raters were selected.

The only differing characteristic is that the present raters simply have experience rating courses, whereas the new influx group don't. We could argue that since the first batch wasn't uniquely qualified to begin with, the characteristics of the expanded population are the same. I see this as potentially upsetting the status quo as we have seen it in the recent past, with the influx group not having as much bias and we will see some more variance.

So in summation, I would agree with you TD if the present raters had any higher qualifications or training.  However, I don't believe they have, they simply have experience which could be manifested in biased scoring, so the influx group could mitigate that bias by expanding the sample size. If anyone can point out some higher qualifications or training etc. the present raters have had (other than LSD at some point in their lives it appears), then I'm certainly open to hear that as it would prove useful here.

[Jeff Schley]

BTW the top 3 in each category: 
Shot Values - Pine Valley - 8.95, ANGC - 8.67, Shinnecock - 8.63
Resistance to Scoring - Pine Valley - 8.88, Oakmont - 8.84, ANGC - 8.64
Design Variety - Pine Valley - 9.14, Cypress Point 8.85, ANGC - 8.73 
Memorability - ANGC -9.29, Pine Valley - 9.24, Cypress Point - 9.23 
Aesthetics - Cypress Point - 9.27, AGNC - 9.20, Pebble Beach - 9.00
Conditioning - ANGC - 8.98, Pine Valley - 8.71, Oakmont - 8.70
Ambience - ANGC - 9.40, Pine Valley - 9.22, Cypress Point - 9.17

Speaking of statistical significance, any statistician would tell you there's no way the numbers should work out as above, where not a single outlier to the top ten overall is in the top three in ANY of the categories they rate.


I guarantee you there are some less highly-ranked courses in America that are better conditioned than ANGC or Pine Valley, no matter how you define conditioning -- and/or courses with better aesthetics or ambience than the three listed, or better Design Variety or whatever.


The reason you see results like the above is because panelists are afraid to upset the status quo. 


For the top ten courses, they start with a 9, and their ratings in each category land between 8 and 10, with rare exceptions.  For the next rung, they rate the seven categories from 7 to 9, averaging out around 8. 


But they will rarely give a 10 in any category to a course that isn't totally respected, for fear of being dismissed from the panel as an "outlier" . . . and the converse is that they rarely give a highly-ranked course a 4 or 5 in one of the categories even if it is totally deserved [like, say, Design Variety for The Olympic Club or Firestone, or Conditioning for Yale].


If you're wondering why the top ten doesn't change very much, this is why.  Panelists know which lines they are not supposed to cross.

I'd love for someone to import this data into a statistic software tool to tell us the standard deviation.  I'm guessing on the top end we will have the top 3 show some scoring more than 1 SD above the mean for sure in many categories, which is statistically significant. 

I'd also like to see the highest + SD in each category vs. the highest - SD.  There are some with big swings..... for example The Quarry at La Quinta it appears is #99 in resistance to scoring, but is #6 in conditioning that is certainly quite a swing and I would think that would be one of the biggest. Also notice some outliers in conditioning on the low end. Maidstone is 99th at 7.42 and Yeaman's Hall is 100th at 7.35, which have the distinction of the 2 worst conditioned courses on the Top 100, however I think this could be misleading as given the band of error I think it would get them back within 1 SD of the mean, so not statistically significant.


BTW if anyone gets this into excel PLEASE send it to me!

I want to add another column while we are geeking out for a minute and that is the Doak Scale for each.  You could have TD's score vs. the average of the 4 or whatever.

[Tom_Doak]

Aside from the Doak Scale as implemented by a panel of four, how would you design the most reliable rating system? Which of the existing ones comes closest?

(e) None of the above.  Every one of the rating systems has grown continually further away from sticking to a smallish panel of well-respected people, as a function of trying to make more $$ off the process and/or pretending that it makes the results more statistically significant.

[Jeff Schley]

Aside from the Doak Scale as implemented by a panel of four, how would you design the most reliable rating system? Which of the existing ones comes closest?

(e) None of the above.  Every one of the rating systems has grown continually further away from sticking to a smallish panel of well-respected people, as a function of trying to make more $$ off the process and/or pretending that it makes the results more statistically significant.

Richard Pryor in Brewster's Millions.

[Tom_Doak]

TD I certainly see your point and agree on the surface, however what was the criteria for the present GD raters to be selected? I know several and none have said they had to attend any course or agree to any rubric for their scoring, they simply met the handicap requirement and some were "connected" or claimed other raters were, which is why not all potential raters were selected.

The only differing characteristic is that the present raters simply have experience rating courses, whereas the new influx group don't. We could argue that since the first batch wasn't uniquely qualified to begin with, the characteristics of the expanded population are the same. I see this as potentially upsetting the status quo as we have seen it in the recent past, with the influx group not having as much bias and we will see some more variance.

So in summation, I would agree with you TD if the present raters had any higher qualifications or training.  However, I don't believe they have, they simply have experience which could be manifested in biased scoring, so the influx group could mitigate that bias by expanding the sample size. If anyone can point out some higher qualifications or training etc. the present raters have had (other than LSD at some point in their lives it appears), then I'm certainly open to hear that as it would prove useful here.

Jeff:


There is no good answer to your questions.  Who would do the training?  Years ago I spoke at a GOLF DIGEST Panelist Summit and listened to Ron Whitten try to explain Shot Values to a group of their panelists, and I couldn't follow his logic; I don't think most of them did, either.  But that's because they are trying to pretend these things are not subjective, when they certainly ARE.


The only way to rate golf courses is to embrace the subjectivity of it all, which then forces you to explain WHY you have your opinions, in English [or Japanese?] and not by spouting off a sequence of numbers.  It's a book, or an essay, or a debate . . . but it's not a spreadsheet.


But if you're going to select raters, you should start over and have the editors pick the 100 or 200 people whose opinions they respect most, instead of just taking everyone who fits a certain profile and will pay for the privilege.  It's just like politics or law enforcement . . . the people who self-select for it are often the last people you want in those positions   

[Peter Pallotta]

Maybe it's not the system that's flawed, maybe it's the concept. And if so, you can't fix the concept by improving the system. Tweak or abandon the criteria, replace 1000 panelists with 50 architects or 100 writers/critics, rank the courses in sets of 10 (with no distinctions within any one set), the concept still remains the same, ie that many opinions are more accurate than one opinion; that expert assessments are less biased and clear-eyed than those of amateurs and neophytes; and that the marginalizing (through number-crunching) of 'outlier' courses is a positive function of the concept, instead of its greatest flaw/weakness. If I were an architect aiming to scale the sunlit uplands of golf course architecture and to strive for/achieve the transcendent, I sure wouldn't want to have my work experienced & judged by a 'concept' that so preferenced pre-existing thinking over the new, or that seemed purposely designed to dismiss the outlier as a mistake. Indeed, I suppose such a concept almost guarantees that the transcendent would never be recognized or appreciated, or that it would even ever see the light of day. (And if that were the case, would I even try as an architect, or even think to try, for that new sunlit upland?) As I suggested once in a long ago and much disliked thread, I think that 'we're living in conservative times'.
 

[Jeff Loh]

If Golf Digest bites the dust do the raters get their 1k back?

[John Kavanaugh]

99% of all golfers would be better off just listening to Matt Ginella.

[James Reader]

Taking Tom’s points above about the lack of variation in the top 3 across each of the categories, it’s interesting (maybe just to me!) to see that this feature doesn’t exist to the same extent in Golf World’s new ranking of the Top 100 courses in England.  Perhaps this is because the rankings are based on a panel of just 30, selected on the basis of some (unspecified - other than that they have to have played at least 70 on the list) assessment of their “passion and knowledge”.


The rankings by category, with their overall position in brackets:


Design - 1. Lytham (; 2= Sunningdale Old (1); 2= Woking (17)
Setting - 1 Perranporth (67); 2= St Enodoc (6); 2= Sunningdale Old (1)
Playability - 1= Ferndown (36); 1= New Zealand (41); 3= The Belfry(!) (59); 3= Sunningdale Old (1); 3= Alwoodley (16)
Memorability - 1 St Enodoc (6); 2 Birkdale (2); 3 Royal St George’s (3)
Presentation - 1 Lytham (; 2 Birkdale (2); 3 Hoylake (15)
Consistency - 1 Birkdale (2); 2 Lytham (; 3 Royal St George’s (3)


Does this mean it’s “better”?  No idea, but it does perhaps suggest there is a bit less of the “group-think” apparent in the Golf Digest list.  Personally I take all these things with a pinch of salt, but if a course appearing anywhere on one of these lists is one of the factors that will encourage me to try to play there at some point (along with other things like the Confidential Guide and Sean’s tours on this site).


With that in mind, one that is now very definitely on my list is Formby Ladies.  It hadn’t been on my radar until Sean’s tour last year but it’s the highest new entry on this list at number 62; perhaps another point in these rankings’ favour that such an unheralded course can appear in such a high position.  Not too far away for me so I’ll try to get there at some point this year.

[Jeff Schley]

OK I used part of my day to make a spreadsheet and filled in just the shot values data.  I want to preface this by saying I'm not a data scientist, Ph.D., or uniquely qualified, just a former MBA who had a few classes over my schooling and appreciate the data if the topic interests me. So I'm attempting to apply my limited knowledge.

This is what I came up with:

Number (N)=   100 (this represents the 100 courses in our data set)
Mean=   7.8304 (this is the average)
Standard Deviation=   0.2985

Here are the  SD below and above the mean going to 3 SD's.

-3 = 6.9350
-2 = 7.2335
-1 = 7.5319
mean = 7.8304
1 = 8.1289
2 = 8.4273
3 = 8.7258

SD's capture a % of all the data, so as you get larger you get a larger % of all possible scores. 
1 SD = 68% of data
2 SD = 95% of data
3 SD = 99.7% of data

So what does all this SHIIIIIIIT mean? Heavily screwed towards the top ranked courses as Pine Valley for instance would be MORE than 3 SD's above the mean as the expected score and probably 4 or so which accounts for 99.9% of all scores.  By contrast the lowest score is Diamond Creek at 7.4187 which is just over 1 SD below the mean.  In general 2 SD's are considered statistically significant, although it is much more detailed than that (using hypothesis testing and p/z scores).


Thus Pine Valley is a HUGE outlier when it comes to shot values, which remember is 25% of the overall total for it is doubled from the original 12.5%. Here is the data set sorted highest to lowest and you can see not many lower ranked courses jump those in front and vice versa for this category.

Rank   Course   Shot Values
1   Pine Valley   8.9455
2   Augusta National   8.6669
4   Shinnecock Hills   8.6307
5   Oakmont   8.6228
6   Merion G.C. (East)   8.4923
3   Cypress Point Club   8.4596
8   National G. Links of America   8.3341
11   Winged Foot (West)   8.2819
9   Sand Hills   8.2561
7   Pebble Beach    8.255
14   Chicago   8.1367
16   Muirfield Village   8.1359
13   Crystal Downs   8.1353
12   Seminole   8.1067
10   Fishers Island Club   8.0976
23   Riviera   8.092
20   Oakland Hills   8.0822
22   Oak Hill    8.0748
27   Prairie Dunes   8.0732
37   Bethpage Black   8.0622
19   Los Angeles CC   8.0591
18   The Country Club   8.0536
15   Friar's Head   8.0453
29   Pinehurst No. 2   8.0362
24   Kiawah Island (Ocean Course)   8.0254
34   Southern Hills   8.008
21   Whistling Straits   7.9975
33   The Olympic Club   7.9885
17   Pacific Dunes   7.986
25   Wade Hampton   7.949
28   The Honors Course   7.9359
30   Peachtree   7.8812
53   Medinah no. 3   7.8539
38   The Golf Club (Ohio)   7.8416
46   Ballyneal   7.8413
45   Spanish Bay   7.8369
26   Shadow Creek   7.8322
42   Erin Hills    7.8253
47   Butler National   7.8213
41   Baltursol (Lower)   7.821
60   Oak Tree National   7.8208
32   Gozzer Ranch Golf & Lake Club   7.8176
35   San Francisco   7.8172
43   Victoria National   7.8111
49   TPC Sawgrass   7.8093
64   Pete Dye   7.7931
40   Pikewood National   7.7724
44   Camargo Club   7.7706
88   Inverness Club   7.7614
31   The Alotian Club   7.7526
75   Olympia Fields (North)   7.7437
39   Sebonack   7.7386
36   Bandon Dunes   7.7362
51   Garden City   7.7344
82   Plainfield   7.7338
52   Winged Foot (East)   7.7279
59   Old Sandwich   7.7222
54   Whispering Pines   7.7172
48   Castle Pines   7.716
57   Scioto   7.7074
58   Interlacken   7.6927
55   Shoreacres   7.6899
61   Baltursol (Upper)   7.6888
84   Valhalla   7.6871
85   Aronimink   7.6853
65   Dallas National   7.6784
71   Quaker Ridge   7.6699
80   Congressional    7.6581
98   Old Town Club   7.6572
63   Somerset Hills   7.6547
50   Old Macdonald   7.6544
87   Boston GC   7.6325
69   Bandon Trails   7.6321
66   Kinlock   7.6276
99   Crooked Stick   7.6273
73   Essex County Club   7.6215
72   Cherry Hills   7.6206
95   Yeamans Hall Club   7.6129
62   Canyata   7.605
83   Calusa Pines   7.5781
76   Myopia Hunt Club   7.5744
89   Sleep Hollow   7.572
67   Maidstone Club   7.5678
97   Blackwolf Run (River)   7.5657
56   Monterey Peninsula (Shore)   7.5635
68   Arcadia Bluffs   7.5497
70   Estancia   7.5481
79   Monterey Peninsula (Dunes)   7.5451
74   Milwakee CC   7.535
93   Spring Hill   7.5282
81   Hudson National    7.5217
86   The Valely Club of Montecito   7.5077
94   Rich Harvest Links   7.5049
91   The Golf Club at Black Rock   7.4924
78   The Kittansett Club   7.4899
100   Mayacama   7.4699
92   Double Eagle Club   7.4535
96   The Preserve GC   7.4374
77   The Quarry at La Quinta   7.4358
90   Diamond Creek   7.4187

[James Reader]

Taking Tom’s points above about the lack of variation in the top 3 across each of the categories, it’s interesting (maybe just to me!) to see that this feature doesn’t exist to the same extent in Golf World’s new ranking of the Top 100 courses in England.  Perhaps this is because the rankings are based on a panel of just 30, selected on the basis of some (unspecified - other than that they have to have played at least 70 on the list) assessment of their “passion and knowledge”.


The rankings by category, with their overall position in brackets:


Design - 1. Lytham ( ; 2= Sunningdale Old (1); 2= Woking (17)
Setting - 1 Perranporth (67); 2= St Enodoc (6); 2= Sunningdale Old (1)
Playability - 1= Ferndown (36); 1= New Zealand (41); 3= The Belfry(!) (59); 3= Sunningdale Old (1); 3= Alwoodley (16)
Memorability - 1 St Enodoc (6); 2 Birkdale (2); 3 Royal St George’s (3)
Presentation - 1 Lytham ( ; 2 Birkdale (2); 3 Hoylake (15)
Consistency - 1 Birkdale (2); 2 Lytham ( ; 3 Royal St George’s (3)


Does this mean it’s “better”?  No idea, but it does perhaps suggest there is a bit less of the “group-think” apparent in the Golf Digest list.  Personally I take all these things with a pinch of salt, but if a course appearing anywhere on one of these lists is one of the factors that will encourage me to try to play there at some point (along with other things like the Confidential Guide and Sean’s tours on this site).


With that in mind, one that is now very definitely on my list is Formby Ladies.  It hadn’t been on my radar until Sean’s tour last year but it’s the highest new entry on this list at number 62; perhaps another point in these rankings’ favour that such an unheralded course can appear in such a high position.  Not too far away for me so I’ll try to get there at some point this year.

In spite of the sunglasses emoji Lytham is no.8 on the list!

[Tom_Doak]

Taking Tom’s points above about the lack of variation in the top 3 across each of the categories, it’s interesting (maybe just to me!) to see that this feature doesn’t exist to the same extent in Golf World’s new ranking of the Top 100 courses in England.  Perhaps this is because the rankings are based on a panel of just 30, selected on the basis of some (unspecified - other than that they have to have played at least 70 on the list) assessment of their “passion and knowledge”.

James:


I think there are two reasons for the wider variation there:


1.  Smaller sample size, and
2.  They haven't broken it out that way a lot of times before, so the panelists don't know what the "approved" answer should be.


P.S.  They are giving points for "consistency" ?  Ugh.

[corey miller]

One of the advantages of this list is that it makes my off grounds invites to the assistant pros from my club appear much more valuable than they were. 

Not much I see in regards to routing or green to tee walks and then when you add in the variability of such between member tees and back tees at many places and you have a mess.  IMO.  Does a club get penalized if the member tees are close and the back tees are 100 yard walk backwards on every hole?  [size=78%]

[/size]Ambiance I guess?[size=78%]

[/size]Perhaps we could have the GD rater people on this site put up their top 100 lists? It would be really instructive perhaps if they did it with the GD criteria and then with criteria all their own.

[Tom_Doak]

Perhaps we could have the GD rater people on this site put up their top 100 lists? It would be really instructive perhaps if they did it with the GD criteria and then with criteria all their own.

Do you really, truly want to see some panelist's spreadsheet with 700 numbers [and that's just for the courses that are IN the list, not the challengers], and try to make any sense of it?

[corey miller]

Tom


I think the individual lists may be  a lot more instructive than all the group think.


Imagine making a big ticket purchase (home, car) using such metrics and percentages?




 Seems to me that most just decide how the place "moves" them and it should be the same in golf.