G.C.A. UNOFFICIAL Top 100 Released

[Mark Pavy]

The existing formula looks fine.

I'd also be pushing for allowing voting after having viewed the entire course in 3D. This would increase the amount of courses people can "rate" dramatically.

Regarding the list produced 5 yrs ago, it wouldn't surprise me if 10-20 courses make way.

[George Freeman]

Bump.  I consider this the best list of courses I have seen to date.  As others have mentioned, it would be fun to re-do the exercise every 5 years or so to bring new courses into the fold.  Also, I think expanding the list would be helpful as well.

It's not so much the number at which the courses are listed, but simply their inclusion that makes the list very interesting and helpful.

[Jeremy Rivando]

Fantastic list guys, at first glance it appears to be the most honest and accurate ranking of the Top 100.  Thanks for all the work by those involved in such a lengthy process.

[Jud_T]

I still think we should do the Unofficial Nuzzo Rankings-  i.e. 3 separate lists for Fun, Pretty and Challenging that people could look at separately and/or re-combine with whatever weightings they wanted.  How an absolute ranking of Kingsley, Fishers and Bethpage Black tells you anything other than garbage in equals garbage out is beyond me.

[Jim Hoak]

Is it time for a redo of this?  It's been 6 years, many new courses, and many new people on GCA.  I'm not volunteering; I wouldn't even know where to start.  But I'd love to see a final product.  Just hoping, I guess, that someone will pick it up.

[George Freeman]

I know we all hate (love) course ranking lists around here, but I thought this was a fun exercise and I still think this is the best list I have seen (who doesn't love groupthink?).

It's been seven years since we did this exercise.  I'm sure there would be more than a few new additions to this list.  Is it time for an update?  Ian?  Jonathan C?

Hi All,

After a very long delay, the Unofficial GCA Top 100 is finally out!  I would like to say a quick thank you to:

-Ran for letting me conduct this experiment (I would like to stress again that this is not sponsored by GCA nor is it an official GCA project)
-Tom Doak for coming up with a comprehensive list of courses
-Jonathan Cummings for his amazing statistics work
-And to everyone who nominated courses, voted on them, and gave suggestions for the format of the experiment.

A quick review of the process:
We began with a list of nearly 400 courses for Top 100 consideration.  From there, GCA members nominated courses to be added to the ballot (two votes were required for a course to make the ballot). From there we arrived at a list of 412 courses that could be considered for the top 100 in the world.  Members then voted on each course they had played using the Doak Scale.

-There were 177 valid voters
-There were 8403 valid votes
-Raters had played an average of 47 courses on the list
-Each course was rated on average 20 times
-The average rating was 6.9

The lists:

Two Top 100 lists have been created, so you may choose the one that suits your opinion.  The first contains "raw" rankings.  Courses are ranked simply by the average vote.  However, only courses with 10 or more votes are included.

Raw rankings (courses with 10 or more votes included):



The second list was adjusted by Jonathan to give experienced voters more weight, guard against raters who consistently rated too low or too high, and discard outliers (Jonathan can explain this all much better than I can). Here is Jonathan's summary of the process:

"Matrix ranked by avg (highest-to-lowest)               
Courses with less than 10 votes deleted               
Only rater votes within +/-2SD of the overall course average included               
Bias Adjustments: Delete Raters with less than 30 votes and |SD|>1; Adjust Raters with more than 30 votes and |SD|>1 by rater/overall average differences               
Courses with less than 10 votes again deleted"

Adjusted rankings:


Feel free to discuss which method you prefer, as well as any comments you have on the lists.  Thanks again for your participation.

**EDIT**

You can download a spreadsheet with all the basic stats here:
http://www.mediafire.com/?ojoy0mn1yzj

Here is a snapshot of the full raw rankings (all eligible courses):
http://img.photobucket.com/albums/v603/ian8389/GCA/Raw_All.jpg

And the full adjusted rankings:
http://img.photobucket.com/albums/v603/ian8389/GCA/Raw_All.jpg

Note: see important message on reply #86 (page 3) regarding these "full" rankings.

[Tom_Doak]

I bet if we attempt to repeat this it will turn out more differently than you expect.


The magazines usually remind panelists how they voted two years earlier, so they won't just change their votes back and forth without having a reason to do so.  Without the reminder, and with a fairly large turnover in voters from seven years ago, it will change ... even though this time it should come out more as expected, because everyone has seen the results from last time, and will inevitably be influenced by them.

[John Kavanaugh]

I hope St. Peter is a golfer and anyone associated with this list has to explain some of these 10's.

[Scott Warren]

We are now a decade removed from this exercise. Not only do we have a good few new courses that have been built since then (Lost Farm, Tara Iti, Castle Stuart, Old Mac, Streamsong x 3, Sand Valley x ?) those of us who participated the first time will have seen a lot more of the candidate courses since 2009, several notable courses have undergone significant renos... Can we made it happen?

[Tom_Doak]

It's a lot of work to put these things together.  I don't see anyone volunteering to do the work.


Honestly, there are too many rankings already.

[Sean_A]

It's a lot of work to put these things together.  I don't see anyone volunteering to do the work.

Honestly, there are too many rankings already.

I am not volunteering to run the show, but I wouldn't mind seeing an update.  Why?  It is the best ranking I have ever seen. If there is no fancy math used, it could be a fairly simple (but time consuming) data gathering project.  To come up with a course list you start with this original list, add in the courses that would have made it, but didn't have enough votes.  Then maybe add any course on a major list which makes the top 50.  Then add odd courses suggested by GCAers with at least two other seconders...none of these courses can be on the master list used for the original list.

Ciao

[Jim Franklin]

I hope St. Peter is a golfer and anyone associated with this list has to explain some of these 10's.

That's the fact Jack.

[Kalen Braley]

If someone wants to spearhead the effort, i can certainly assist on the back end with the spreadsheet work and putting together various graphs or statistical analysis...

[Jim Nugent]

Sean, I agree.  Also do not statistically massage the data.  Give it to us in raw form.  Last time the statistical model cast out any scores of less than 9 for Pine Valley. 

It's gotta be a lot work. 

As for so many courses getting 10s... rating is subjective.  I'm glad PV got at least one 7...P2 and Cruden Bay got at least one 4.  Shows we're not entirely consumed by group think. 
 

[Tom_Doak]

Where is Ian Linford when you need him?

I do think it would be instructive to this group to see how much this ranking will have changed over ten years.  I suspect it will have changed more due to turnover on the Discussion Group, and changes in the groupthink, than as the result of new courses having been built -- though there may be 8-10 new courses that will challenge for the top 100.

I suspect a couple of courses will go from around 50th to out of the list entirely - which should not happen if such an exercise was more than subjective, but of course, it's not.

I agree with Jim N. that all voting should be presented in raw form.  If a course only gets two votes and they are both 9's, put it high on the list, and we can judge for ourselves whether to take the word of those two panelists.  "Data analysis" is what the mainstream media use to downplay Bernie Sanders and ignore other topics they don't want you to hear about.  If there are questions about the worthiness of some course with only a handful of votes . . . that's exactly the most interesting kind of debate to have about rankings.  We already have a pretty good idea of what the mainstream view is, via plenty of other rankings.

Other ideas:

1.  All ballots should be "on the record".  If you're going to be the guy who gives Pine Valley a 7 or another top-100 course a 4, or some outlier course a 10, you should be ready to defend that, instead of just voting extremely to put your thumb on the scale.  By the same token, your vote should not be squelched in the results just because you don't agree with the consensus.

2.  I'm not sure what criteria were used last time for what is an 8 or a 9 or a 10.  There ought to be some scale for this:  a 10 is for a top 10 course, a 9 for top 25 or top 50, etc. 

3.  And then panelists should be asked not to give more 10's and 9's and 8's than the number of courses they have seen that made each of those hurdles in the previous list.  If some less-traveled panelists are handing out 10's to their personal top ten, having seen only two of the actual top ten, that skews the rankings in favor of the courses they happen to have seen.  If you haven't seen any of the world top ten, then 9 or 8 should be the top of your scale until you have.

[Niall C]

3.  And then panelists should be asked not to give more 10's and 9's and 8's than the number of courses they have seen that made each of those hurdles in the previous list.  If some less-traveled panelists are handing out 10's to their personal top ten, having seen only two of the actual top ten, that skews the rankings in favor of the courses they happen to have seen.  If you haven't seen any of the world top ten, then 9 or 8 should be the top of your scale until you have.

Tom


I understand your thoughts on breadth of experience but what about depth of experience ? Your idea skews the scoring towards the hit and runners. If you take a notional top ten, and assuming you don't know the person scoring, then I think I would rather get the view of someone who has played 3 or 4 of the top courses half a dozen times each than someone who played all 10 only once each.


Niall

[corey miller]

Even better would be a compilation of everyone's rankings.  I really don't care if another person tells me Ballyneal and Friar's Head are both 9's or 10's but do care the debate between the two and why it looks like (from the 2009 list) Friars Head has surpassed Ballyneal. 


Of course, for many, without existing ratings it would be difficult to produce new ones.

[Tim Gallant]

Dare I volunteer? I am happy to compile the list! Leave it with me for the weekend and I will try to reach out to Ran, Tom D, Ian L, Kalen, and anyone else who might want to help, including a few smart fellas I know (paging Ian G!).


I will try to create a new thread and what the process will be early next week. 

[John Kavanaugh]

If we are going to have transparency in these ratings can we also get personal information about the raters?


I recently had a buddy that I trust tell me how great the Trump course in Palm Beach is to play and be a member. The fact that he is a 70 year old 14 handicap megamillionare republican plays a role in how I value his opinion. As I'm sure it would yours.


Would age, club affiliations and handicap really be too much to ask?

[Tom_Doak]

3.  And then panelists should be asked not to give more 10's and 9's and 8's than the number of courses they have seen that made each of those hurdles in the previous list.  If some less-traveled panelists are handing out 10's to their personal top ten, having seen only two of the actual top ten, that skews the rankings in favor of the courses they happen to have seen.  If you haven't seen any of the world top ten, then 9 or 8 should be the top of your scale until you have.

Tom


I understand your thoughts on breadth of experience but what about depth of experience ? Your idea skews the scoring towards the hit and runners. If you take a notional top ten, and assuming you don't know the person scoring, then I think I would rather get the view of someone who has played 3 or 4 of the top courses half a dozen times each than someone who played all 10 only once each.

Niall:

The breadth of experience part is just meant to be a control.  Playing a good but not top-twenty course 100 times does not give you any more perspective on what a "10" is.

I agree with you on hit-and-run panelists, but my experience is that people's votes seldom change very much on subsequent visits.  When we started the GOLF Magazine balloting, I only counted a ballot at half weight if the panelist had only seen the course and not played it.  It was a lot of extra work to keep track of, that had a negligible effect on the results, so we dropped that after a while.

I suppose if you wanted, you could give double weight to the vote of someone who had played a course 5 times or more, but all that will do is boost the status quo.  For example, I have played all but four of the top 25 courses five times or more, but there are not many lesser-known contender courses that I would have played 5x or more.  I'm only going to play the lower-ranked courses five times if I have some personal connection to them.  In this case, that would add weight to the few courses like Ballyneal and Kingsley where multiple posters to the DG are also members, and those places are probably already getting more weight in our poll than they should.


I don't doubt that experience counts for a lot, but you either trust everyone here to vote on level terms, or you don't.

[Tom_Doak]

Dare I volunteer? I am happy to compile the list! Leave it with me for the weekend and I will try to reach out to Ran, Tom D, Ian L, Kalen, and anyone else who might want to help, including a few smart fellas I know (paging Ian G!).


I will try to create a new thread and what the process will be early next week.

Sent you a pm with the best advice I could give.  God bless you for dealing with all the hassle of this.


I started to list courses which deserve to be on the ballot, but realized there are a lot, so I will let the DG nominate these.  My nominations that maybe no one else has seen:  Victoria GC in Sri Lanka, La Moye in Jersey (Channel Islands), Whistling Rock and South Cape and Pine Beach in South Korea, Arrowtown in N.Z., Himalayan GC in Nepal, St. Andrews Beach in Australia, and maybe a few in Vietnam (but unfortunately none of the ones I saw).

[Kalen Braley]

Tom,

I could build a formula to weight raters based on what they've played.

For example one weight class based on number of top 25s played, another for 26-100, and perhaps even another for 101-200.  And then average them out for a total rater weight to be applied to thier course rankings.

[Tom_Doak]

Tom,

I could build a formula to weight raters based on what they've played.

For example one weight class based on number of top 25s played, another for 26-100, and perhaps even another for 101-200.  And then average them out for a total rater weight to be applied to thier course rankings.

I personally would not go that way at all.  If this is a GCA rating, you should treat all GCA participants equally.


I just think you need to keep raters realistic on what their own experience is.  List the top 100 courses from last time first, and make everyone count how many of those they have seen . . . and that is how many courses they should be casting a top-100 vote for, with possibly a little leeway.  [A guy who has seen 60 of the courses could arguably vote for 65, but a guy who has seen 20 should not vote for 65.  And for that matter, a guy who has seen 65 should not vote for 20 and cast bad votes on everything else.]  If the voter won't be realistic, then throw out his ballot.


This does NOT favor the voters who have seen more courses.  It just keeps everyone realistic in how many high votes they should be awarding, because there is not a single person out there who's seen EVERY course that is possibly deserving.  Even the folks who have played all of the current top 100 should keep an open mind that there is someplace they haven't been, that would top one of the ones they've seen.

[John Kirk]

I thought Ian (and Jonathan) incorporated adjustments to account for the issues that Tom brings up.  Here are a few words from the opening post:

The second list was adjusted by Jonathan to give experienced voters more weight, guard against raters who consistently rated too low or too high, and discard outliers (Jonathan can explain this all much better than I can). Here is Jonathan's summary of the process:

"Matrix ranked by avg (highest-to-lowest)               
Courses with less than 10 votes deleted               
Only rater votes within +/-2SD of the overall course average included               
Bias Adjustments: Delete Raters with less than 30 votes and |SD|>1; Adjust Raters with more than 30 votes and |SD|>1 by rater/overall average differences               
Courses with less than 10 votes again deleted"

I personally would not go that way at all.  If this is a GCA rating, you should treat all GCA participants equally.

I just think you need to keep raters realistic on what their own experience is.  List the top 100 courses from last time first, and make everyone count how many of those they have seen . . . and that is how many courses they should be casting a top-100 vote for, with possibly a little leeway.  [A guy who has seen 60 of the courses could arguably vote for 65, but a guy who has seen 20 should not vote for 65.  And for that matter, a guy who has seen 65 should not vote for 20 and cast bad votes on everything else.]  If the voter won't be realistic, then throw out his ballot.

I agree.

[Tom_Doak]

I thought Ian (and Jonathan) incorporated adjustments to account for the issues that Tom brings up.  Here are a few words from the opening post:

The second list was adjusted by Jonathan to give experienced voters more weight, guard against raters who consistently rated too low or too high, and discard outliers (Jonathan can explain this all much better than I can). Here is Jonathan's summary of the process:

"Matrix ranked by avg (highest-to-lowest)               
Courses with less than 10 votes deleted              
Only rater votes within +/-2SD of the overall course average included              
Bias Adjustments: Delete Raters with less than 30 votes and |SD|>1; Adjust Raters with more than 30 votes and |SD|>1 by rater/overall average differences              
Courses with less than 10 votes again deleted"

John:


I guess that's what they thought they were doing, but I disagree with all three of the methods highlighted in red.


1)  if a course has only five votes, don't delete it, show the results it got.  This is an unofficial ranking anyway.  Don't squelch what the voters have said just because there weren't enough of them weighing in on Morfontaine, De Pan, etc.


2)  The two-standard-deviations thing just throws out any low votes for the top courses as being non-conforming.  That shows no respect for the voter.  If someone wants to give Pine Valley a 7, they should be willing to defend their reasoning, but their vote should count.  I guess that opens things up to guys deliberately trying to shaft a particular course, so there should be some "floor" . . . instead of 1-10 voting maybe 5-10 or 6-10, as any course on the ballot should be deemed at least somewhat worthy.


3)  It doesn't matter if you have seen less than thirty courses, as long as you're not voting more of them into the top 100 than you should based on your experience.  Again, the standard deviation thing is just a math-based excuse to disenfranchise certain voters.  Save that for American politics, where it belongs!