Re-analysis of GCA's Top 100

[Tom_Doak]

Jim: I think you have it right.  For every possible pair of courses, I looked only at the raters who had played both.  Then I allocated 1 point between the two courses just as you described.

There were 177 courses that were in the top 200 and had more than 10 ratings.  Therefore, for each course they could earn as many as 176 points (which Pine Valley did) or as little as 0.  To say that Pine Valley beat every other course in a head-to-head match-up is much more meaningful than saying that they had the highest average rating among a list a raters who all played a different subset of courses.

Anthony:

Now I'm confused again as to how you did this.

I presumed what you had done was to compare individuals' votes on the top 200.  For example, on my own ballot, Pine Valley would have beaten nearly all of the courses, but tied with a few others [including Cypress Point] ... hence it would not have a "perfect" score.  And are you really saying that NO ONE rated any course higher than Pine Valley?  That's a level of group-think that I have never seen in any ranking process.

[Anthony Fowler]

Hi Tom,
I'm sorry about the confusion.  It's not the case that every rater prefers Pine Valley to every other course.  Rather, if we took a vote between Pine Valley and Course X for all raters who had played both Pine Valley and X, Pine Valley would win for any of the 176 other courses in my analysis.  We're counting a vote for X over Y as giving X a higher score on the Doak Scale than Y, and we're calling it a tie (abstention vote due to indifference) if a rater gave them the same score.  I don't think this is a problem, because I don't think raters should worry about splitting hairs between two 10s.

Jim: Your point is correct that raters did not cast head-to-head votes.  I would be really interested to know whether the results would be different had this been the case.  I pushed for this when Ian first proposed his ranking project, but the idea was not adopted.  Nonetheless, it would be rather strange if someone would give Pinehurst a 9, Wannamoisett an 8, but then say that they prefer Wannamoisett over Pinehurst head-to-head.  It's certainly possible, but I wouldn't put much stock in that persons ratings in this scenario.

[Anthony Fowler]

Anthony, can you post some of the actual scores.  e.g. how did the PV/CPC "matchup" turn out?   

Of course.  The Pine Valley/Cypress Point matchup turns out to be fairly uninteresting since only 28 people rated both courses and none of them gave either course less than a 9.  Remember that you should not read seriously into the difference between any two courses that are close to each other on the list.  23 raters gave them the same score, 4, preferred PV, and 1 preferred CP.

Here are some of the popular public matchups

Pacific Dunes vs. Pebble Beach: 22 indifferent, 32 for Pacific, 14 for Pebble

Pacific Dunes vs. St. Andrews: 30 indifferent, 19 for St. Andrews, 8 for Pacific

Pacific Dunes vs. Pinehurst No. 2: 20 indifferent, 33 for Pacific, 5 for Pinehurst

Pebble Beach vs. St. Andrews: 20 indifferent, 40 for St. Andrews, 7 for Pebble

Pebble Beach vs. Pinehurst: 30 indifferent, 14 for Pinehurst, 22 for Pebble

St. Andrews vs. Pinehurst: 10 indifferent, 13 for St. Andrews, 4 for Pinehurst


What about some regional matchups?

Pacific Dunes vs. Bandon Dunes: 10 indifferent, 77 for Pacific, 1 for Bandon

Ballyneal vs. Sand Hills: 10 indifferent, 4 for Ballyneal, 19 for Sand Hills

Whistling Straits vs. Blackwolf Run: 11 indifferent, 6 for Blackwolf, 19 for Whistling

Shinnecock vs. NGLA: 23 indifferent, 9 for NGLA, 8 for Shinny

Cypress vs. Pebble: 13 indifferent, 2 for Pebble, 39 for Cypress

Pebble vs. Spyglass: 23 indifferent, 15 for Spyglass, 37 for Pebble

Crystal Downs vs. Kingsley Club: 9 indifferent, 7 for Kingsley, 14 for Crystal Downs

Riviera vs. LACC: 10 indifferent, 3 for LACC, 14 for Riviera

Olympic vs. SFGC: 11 indifferent, 2 for Olympic, 25 for SFGC

[Jim Nugent]

Tom Doak, I think PV wins over CPC because more people gave it higher Doak scores.  Majority rules, in other words.

Or "sort-of" majority.  One problem with the match-ups is the small numbers.  e.g. PV/CPC came down to five people.  While I'm not a statistician, my reaction is that was more a tie than a clear-cut win for PV.  Almost everyone who played both rated them a 10.  Same with Shinnie/NGLA.  IMO we should count that as a tie, not a win for NGLA. 

Anthony, the main reason I see head-to-head matches might yield different results is that the Doak scale is not too precise.  Ten = ten, even if we prefer one course.  But say we could give scores of 9.9, 9.8, and so on.  Lots of ties would get unlocked.  A true head-to-head matchup might overcome this problem and show people's real preferences. 

Btw, very cool idea and analysis, with lots of interesting results.  I'm still not sure it's more accurate than simply averaging the raw scores, though.  Want to give this a bit more thought. 

IMO the statistically-massaged version is far worse than either yours or the raw arithmetic mean. 



 

[Jud_T]

I love the head to head idea-the problem is that you need both a statistically significant sample size of people who've played both courses, at least 20 and more likely 30, and a meaningful win by of say 2 or 3 guys (i.e. not 15-14 etc...).  The problem with rating courses on specific criteria (i.e. the walk in the park factor, resistance to scoring etc.) is nobody even here can agree on exactly what the criteria should be...

[Jim Colton]

Anthony,
 
  I ran my approach looking at all of the individual head-to-head match-ups (there were around 800,000 of them).  My results are generally the same as yours, although I have Cypress Point #1 and Pine Valley #2.  I also ran one that looked at the score differential between the two and the results were largely the same (Merion jumped to #4 in that one).

  I used 5 rankings as my minimum cut-off.  Morfortaine (36) and Hamilton (52) come in with only 6 and 8 ratings, respectively, but 3 of those were from panelists with over 275 courses rated, so you are relying on their expert judgment for the most part.

Anthony's head-to-head rating in ()

1   Cypress Point Club (2)
2   Pine Valley Golf Club (1)
3   Royal Melbourne (West) (4)
4   Sand Hills Golf Club (T5)
5   Shinnecock Hills Golf Club (T5)
6   National Golf Links of America (3)
7   Royal County Down (T7)
8   Merion Golf Club (East) (T7)
9   St. Andrews (Old) (T7)
10   Royal Dornoch (10)
11   Oakmont Country Club (12)
12   Pacific Dunes (11)
13   Crystal Downs Country Club (14)
14   Royal Portrush (13)
15   Muirfield (15)
16   Kingston Heath (22)
17   Ballybunion (16)
18   Augusta National Golf Club (T17)
19   Riviera Country Club (T17)
20   Seminole Golf Club (23)
21   Prairie Dunes Country Club (T19)
22   Pebble Beach Golf Links (T19)
23   San Francisco Golf Club (21)
24   Fishers Island Golf Club (28)
25   Winged Foot Golf Club (West) (24)
26   Ballyneal (27)
27   Royal St. George's (T25)
28   Friar’s Head  (T29)
29   Pinehurst No. 2 (T25)
30   Barnbougle Dunes (T29)
31   Sunningdale (Old) (31)
32   Highlands Links (35)
33   Cape Kidnappers, N.Z. (34)
34   Rye (32)
35   Turnberry (Ailsa) (33)
36   Morfontaine (NR)
37   New South Wales GC (39)
38   Chicago Golf Club (T36)
39   The Country Club (Composite) (38)
40   Sebonack Golf Club (44)
41   St. George's Hill (51)
42   Swinley Forest (T41)
43   Los Angeles Country Club (North) (40)
44   Royal Birkdale (50)
45   Lahinch (T46)
46   Casa de Campo (Teeth of the Dog) (T36)
47   St. George's (Canada) (T54)
48   Bethpage State Park (Black) (43)
49   Garden City Golf Club (45)
50   St. Enodoc (T54)
51   Woodhall Spa (Hotchkin) (T41)
52   Hamilton (NR)
53   North Berwick (T46)
54   The Golf Club (48)
55   TPC at Sawgrass (Players Stadium) (49)
56   Ocean Course at Kiawah Island (53)
57   Yale University Golf Course (52)
58   Cruden Bay (58)
59   Holston Hills Country Club (64)
60   Carnoustie (Championship) (T54)
61   Myopia Hunt Club (60)
62   Valley Club of Montecito (T72)
63   Portmarnock (Old) (63)
64   Rock Creek Cattle Company (T69)
65   Mid Ocean (T72)
66   Olympic Club (Lake) (57)
67   Ganton (62)
68   Wannamoisett Country Club (T74)
69   Royal Melbourne (East) (T69)
70   Pete Dye Golf Club (76)
71   Pasatiempo Golf Club (68)
72   Royal Lytham & St. Annes (61)
73   Prestwick (59)
74   Macrihanish (T69)
75   Plainfield Country Club (80)
76   Royal Liverpool (65)
77   Sunningdale (New) (66)
78   Boston Golf Club (NR)
79   Bandon Trails  (T77)
80   Walton Heath (Old) (67)
81   Royal Porthcawl (T78)
82   Kingsley Club (79)
83   Bandon Dunes (T82)
84   Paraparaumu Beach (92)
85   Monterey Peninsula Country Club (Shore) (87)
86   Shoreacres Golf Club (85)
87   Oakland Hills Country Club (South) (T74)
88   Southern Hills Country Club (90)
89   Camargo Club (T82)
90   Royal Troon (81)
91   Royal West Norfolk (T81)
92   Maidstone Club (T81)
93   Yeamans Hall Club (86)
94   Quaker Ridge Golf Club (84)
95   Western Gailes (T93)
96   Pronghorn (Fazio) (NR)
97   Whistling Straits (Straits) (99)
98   Pennard (T97)
99   Banff Springs (91)
100   Somerset Hills Country Club (T95)

[Anthony Fowler]

Jim: Thank you for all the work.  Check your messages, and we'll see if we can resolve some of the discrepancies.

[Kevin Lynch]

I love the head to head idea-the problem is that you need both a statistically significant sample size of people who've played both courses, at least 20 and more likely 30, and a meaningful win by of say 2 or 3 guys (i.e. not 15-14 etc...).  The problem with rating courses on specific criteria (i.e. the walk in the park factor, resistance to scoring etc.) is nobody even here can agree on exactly what the criteria should be...

Jud,

I agree with the risks of having a statistically significant sample size.  Yes, you may need more raters to get the number of head-to-heads you want (vs. the traditional "aggregate score" methods), but I think it is worth the effort because of a superior methodology.

In "aggregate ranking" listings, the risk is whether ALL the raters can maintain a constancy relative to the proscribed scale (which is practically impossible - my 8 may be your 6).  However, what I really like about the "head-to-head" methodology is that it makes the individual rater the Constant, which is going to get a more honest result for comparative ranking.

I really like the thought behind this methodology - it's not a perfect system, but I think it's an advancement vs. traditional methods.  Great Job Anthony!!


Ultimately, all quantitative rankings have to be taken with an acknowledgement that it's difficult to quantify something that is inherently qualitative.  Numerical rankings will never replace descriptive critiques like The Confidential Guide.  But even in that book, you almost have to put the numbers as an afterthought.  There are times when I've read two different reviews by Doak and been shocked to see that they have the same "number," at which point I value the comments more.