[John Kirk]

3)  It doesn't matter if you have seen less than thirty courses, as long as you're not voting more of them into the top 100 than you should based on your experience.  Again, the standard deviation thing is just a math-based excuse to disenfranchise certain voters.  Save that for American politics, where it belongs!   

Good one.      By the way, what is it with the line spacing and font controls on the website these last couple years?  It's much more difficult to format things the way you want.

OK, let's take me as an example.  I've played 34 of the raw top 100 courses, most of which are in the top 50.  So I would be expected to rate 30-40 courses as top 100 courses.  What is considered a minimum top 100 course rating?  The 100th best course has a raw score of 7.4, so I could justify rating 30-40 scores at 8 or higher.  I guess you have to trust the raters to be honest and unselfish.  The last list was generally comparable to the three major published lists.

Maybe you solve that by making a "7" a top 100 course.  I know Stone Eagle well, having played it a number of times, and rate it a 7, but enough people are put off by the severity of the terrain to rate it 5 or 6, which knocks it out of the top 100.

But that makes the distinction between a 7 and 8 very important in this exercise.  For a reminder, here are the definitions for a 6, 7 and 8 golf course:

6 — A very good course, definitely worth a game if you’re in town, but not necessarily worth a special trip to see. It shouldn’t disappoint you.
7 — An excellent course, worth checking out if you get anywhere within 100 miles. You can expect to find soundly designed, interesting holes, good course conditioning, and a pretty setting, if not necessarily anything unique to the world of golf.
8 — One of the very best courses in its region (although there are more 8’s in some places, and none in others), and worth a special trip to see. Could have some drawbacks, but these will clearly be spelled out, and it will make up for them with something really special in addition to the generally excellent layout.

[Scott Warren]

Dare I volunteer? I am happy to compile the list! Leave it with me for the weekend and I will try to reach out to Ran, Tom D, Ian L, Kalen, and anyone else who might want to help, including a few smart fellas I know (paging Ian G!).


I will try to create a new thread and what the process will be early next week.

I'm happy to assist -- I'll PM you.

[Tom_Doak]

OK, let's take me as an example.  I've played 34 of the raw top 100 courses, most of which are in the top 50.  So I would be expected to rate 30-40 courses as top 100 courses.  What is considered a minimum top 100 course rating?  The 100th best course has a raw score of 7.4, so I could justify rating 30-40 scores at 8 or higher.  I guess you have to trust the raters to be honest and unselfish.  The last list was generally comparable to the three major published lists.

Maybe you solve that by making a "7" a top 100 course.  I know Stone Eagle well, having played it a number of times, and rate it a 7, but enough people are put off by the severity of the terrain to rate it 5 or 6, which knocks it out of the top 100.

But that makes the distinction between a 7 and 8 very important in this exercise.  For a reminder, here are the definitions for a 6, 7 and 8 golf course:

6 — A very good course, definitely worth a game if you’re in town, but not necessarily worth a special trip to see. It shouldn’t disappoint you.
7 — An excellent course, worth checking out if you get anywhere within 100 miles. You can expect to find soundly designed, interesting holes, good course conditioning, and a pretty setting, if not necessarily anything unique to the world of golf.
8 — One of the very best courses in its region (although there are more 8’s in some places, and none in others), and worth a special trip to see. Could have some drawbacks, but these will clearly be spelled out, and it will make up for them with something really special in addition to the generally excellent layout.

I don't think you can use the Doak Scale, directly, for this exercise.  If you're trying to rate the top 100 courses, you have to tie the numbers to top 100 or not.  8-9-10 means you think it should be in; 7-6-5 means you think it should be out.  And let's all agree that there are no 4's on the ballot, or if there are, they're not going to finish anywhere close to the top 100, so who cares?

[Jim Nugent]

If we are going to have transparency in these ratings can we also get personal information about the raters?

Would age, club affiliations and handicap really be too much to ask?

Unless we show how each individual rater scores every single course, how would that data help much?

[John Kavanaugh]

I thought we were going to show each raters individual scores.


This site would benefit greatly if a poster like me would give a rating for every course I have played and then willingly debate those choices. Or we could just toss my rankings because I don't think some beloved course doesn't sniff the top 20 or top 200.

[Kalen Braley]

Great points all.

I do agree however with John Kirk, you have to address outliers because just one bad score by someone with an axe to grind can throw it off.  Looking back a number posts ago (#130) you have Cruden Bay and Oly Lake with lows of 4s and highs of 10s?  Seems a bit too wide of a gap to rationally explain...

Perhaps 3 SDs? 

[Jim Nugent]

Kalen, don't drop any scores.  How do you know what the rating was based on? 

Doak gave Castle a 0.  Doesn't count? 

Count all scores.  Show the mean, low and high.  Let us see everything and make up our own minds. 

[Tom_Doak]

Kalen, don't drop any scores.  How do you know what the rating was based on? 

Doak gave Castle a 0.  Doesn't count? 

Count all scores.  Show the mean, low and high.  Let us see everything and make up our own minds.

The lowest score in this poll should be a 5.  A 5 doesn't really deserve to be in the discussion, same as a zero, but it stays high enough that you don't have to throw out the vote. 


The Castle Course shouldn't be on the ballot anyway . . . nobody thinks it's a top 100 course, do they? 
I do hope The New Course is on the ballot this time; it has supporters, and even though I disagree, I'm curious to see where it would wind up.

[Ian_L]

Hi all,


I was surprised to open up GCA and see my name there after so many years. While I don't think I have the time to spearhead this effort as I did last time, I'd be honored to help Tim out and try to answer questions about the previous iteration as best I remember it. I might even have the raw data somewhere from the last poll which I am sure a few would enjoy playing with...


To address a few questions about the previous process off the bat:


- We decided to collect names of entrants to confirm that they were GCA members but not publicize them. This was done to encourage honest assessments and also I believe to address Tom's concern that he and others in the industry might be put in an awkward position with a client if they rated their own course lower than hoped (given the publication of a new Confidential Guide I am guessing Tom has come to terms with this!)


- the Doak scale was used.


- There were absolutely cases of attempted manipulation through giving absurdly low scores to certain top courses.


- I will try to find the raw data and make it available, without names since we promised anonymity.


Given that this was nearly a decade ago I could be dead wrong in my recollections, and I apologize in advance!

[Tim Gallant]

Ian L,


Thank you for offering to help. I'll drop you a PM now. It would be great to see the raw data, as this will likely form the foundation for the new list, in which we can drop certain courses that scored too low from last time, and also allow the group to vote in new courses.

[John Kavanaugh]

Will you be deleting the votes of people who are no longer members of the site? This thread has made me think of some classics like Shultzie, Matt Ward and Colton.

[Garland Bayley]

Great points all.

I do agree however with John Kirk, you have to address outliers because just one bad score by someone with an axe to grind can throw it off.  Looking back a number posts ago (#130) you have Cruden Bay and Oly Lake with lows of 4s and highs of 10s?  Seems a bit too wide of a gap to rationally explain...

Perhaps 3 SDs? 

If it runs from 4 to 10, then it clearly doesn't suit a wide selection of golfers, and an average of 7 is just.

I disagree with Tom. There will probably be courses that deserve a 4 for being too one dimensional.  E.g., Olympic Lake.

[Kalen Braley]

Garland,

According to Ian, the last time we did this, the DS was used to generate the 0-10 values.

The disconnect I am seeing is how two people could see the same course and one think its only (4) modestly interesting with only a few distinctive holes and barely above an average course...and the next guy (10) that its basically perfect and even skipping one hole would be missing out and belongs among the elite courses of the world.

P.S Your 4,10 - 7 average is basically my point.  Two data points with an average that doesn't come close to describing either data point.  This is the essence of why statistical analysis exists and is so valuable to just about everything we try to categorize...

[Peter Pallotta]

I don't quite understand the Tom D - John Kirk exchange (probably because I dropped my stats course in university).
If a rater gives a Top 10 course a "10", but it's the only Top 10 course he's ever played, is it still a "10"?
Will his "10" be worth & weighted the same in the GCA compendium/list as the "10" granted to that same Top 10 course but by a rater who has played all the Top 10 courses and most of the Top 100?
Should my "10" for Crystal Downs be treated as equal to Tom D's "10" for Crystal Downs, for final score-keeping purposes?
 
 

[Jim Nugent]

What are the biggest point spreads in the new CG, and for what courses?

It's sad to learn from Ian that some of us gamed the last rankings.  Even so, I note that didn't change much.  The raw and adjusted lists are nearly identical.  But the raw rankings contain lots more info.  And perhaps some unfair low ratings were offset by some unfair high ratings, i.e. some raters gave their  favorite courses higher scores than they knew they deserve?

I don't see why we can't accept scores lower than 5.  Crystal Downs -- a 10 on the Doak Scale -- got at least one 6.  If the 6 is acceptable -- IMO it is -- why couldn't one of the 8s then get a 4? 

By giving the raw scores, the means and SDs, we can easily see how the majority ranked each course.  Let us decide if we want to discount the 7(s?) PV received, or the 4 P2 got. 

[Kalen Braley]

Peter,

That's what I was trying to address in post #146, to implement a weighted rater value, which would be a multiplier and result in your 10 carrying less weight than Tom D's 10.  Exactly how much "weight" would be applied is up for debate, but it makes sense.  Are you going to value a rating the same for one that's played a wide swath of courses, or one thats only played a handful?  Its a bit like looking on Google for a plumber.  If one has 10 - 5 star ratings and another 500, I'm probably going to value the one with more data points....

Jim

6 to 10 seems to be at least in a plausible range, but that data set from 10 years ago has a few at 4-10 and many others at 4-9 or 3-8, which seems to fail a basic smell test.  So i absolutely believe Ian when he says the ratings had issues with homerism and axe grinding...

[David Jones]

Ian L,


Thank you for offering to help. I'll drop you a PM now. It would be great to see the raw data, as this will likely form the foundation for the new list, in which we can drop certain courses that scored too low from last time, and also allow the group to vote in new courses.

If there is ever to be a thread that is a ‘can or worms’ then this is it!


Bravo to Tim for taking this on.  I can’t wait to see the list and how it compares to my attempt here to average out what I think are the only three credible lists left in the world of golf rankings.


https://www.ukgolfguy.com/the-ultimate-top-100-golf-courses


I had to drop the Golf Course Architecture list as it was a little out of date but will gladly include the new GCA list.


Tim, you risk being horribly bogged down in the world of coefficients and all kinds of nonsense. But good luck!


Now, Friar’s Head must be top 10......

[Kalen Braley]

On a side note, I do think there is good value in applying some analytical exercises in this venture.

I've been following Nate Silver's research and ongoing work for years and if you haven't had a chance to read his book, it is excellent, even if you're not terribly interested in the technical aspects!

https://en.wikipedia.org/wiki/The_Signal_and_the_Noise

[Carl Nichols]

Great points all.

I do agree however with John Kirk, you have to address outliers because just one bad score by someone with an axe to grind can throw it off.  Looking back a number posts ago (#130) you have Cruden Bay and Oly Lake with lows of 4s and highs of 10s?  Seems a bit too wide of a gap to rationally explain...

Perhaps 3 SDs? 

If it runs from 4 to 10, then it clearly doesn't suit a wide selection of golfers, and an average of 7 is just.

I disagree with Tom. There will probably be courses that deserve a 4 for being too one dimensional.  E.g., Olympic Lake.

Have you played Olympic Lake?

[Tim Gallant]

Peter,

That's what I was trying to address in post #146, to implement a weighted rater value, which would be a multiplier and result in your 10 carrying less weight than Tom D's 10.  Exactly how much "weight" would be applied is up for debate, but it makes sense.  Are you going to value a rating the same for one that's played a wide swath of courses, or one thats only played a handful?  Its a bit like looking on Google for a plumber.  If one has 10 - 5 star ratings and another 500, I'm probably going to value the one with more data points....

Jim

6 to 10 seems to be at least in a plausible range, but that data set from 10 years ago has a few at 4-10 and many others at 4-9 or 3-8, which seems to fail a basic smell test.  So i absolutely believe Ian when he says the ratings had issues with homerism and axe grinding...

Interesting Kalen! Let's pick-up and discuss off the thread, and maybe we can find a solution.

[Tim Gallant]

Ian L,


Thank you for offering to help. I'll drop you a PM now. It would be great to see the raw data, as this will likely form the foundation for the new list, in which we can drop certain courses that scored too low from last time, and also allow the group to vote in new courses.

If there is ever to be a thread that is a ‘can or worms’ then this is it!


Bravo to Tim for taking this on.  I can’t wait to see the list and how it compares to my attempt here to average out what I think are the only three credible lists left in the world of golf rankings.


https://www.ukgolfguy.com/the-ultimate-top-100-golf-courses


I had to drop the Golf Course Architecture list as it was a little out of date but will gladly include the new GCA list.


Tim, you risk being horribly bogged down in the world of coefficients and all kinds of nonsense. But good luck!


Now, Friar’s Head must be top 10......

Should be fun! I'm looking forward to it, and have a bit of time in August to devote to this. Let's see how it plays out and hopefully it will create good talking points!

[John Kavanaugh]

Let's use some common sense here. You can't let a guy like Kalen get his hands on these numbers anymore than a guy like me. The final list is only as good as those who compile it.

[Ira Fishman]

Tim,


The perfect is the enemy of the good.  I am confident that any methodology you settle upon will be extremely sound.  Plus this should be great fun so many thanks for taking it on.  The only negative is that my cheapskate self needs to break down and finally buy the CG so that I have any idea of how the Doak Scale actually works.


Ira

[John Kavanaugh]

- There were absolutely cases of attempted manipulation through giving absurdly low scores to certain top courses.

Can we stop pretending that absurdly high scores are not also submitted. Who gets to decide that a course like Sweetens Cove receives "certain top course" status and is beyond being given a 3?

[Kalen Braley]

Let's use some common sense here. You can't let a guy like Kalen get his hands on these numbers anymore than a guy like me. The final list is only as good as those who compile it.

Ouch John.

I've spent my entire career doing Quality Assurance and Standards with plenty of spreadsheet time.  Anything I work on for GCA.com will be just like my job, full disclosure with access to everything. We may have our differences, but I'd still hire you anytime to build a road for me. 

[Tags:]

Tags:

[Tags:]

Tags:

Call to undefined function theme_linktree()