GCA Unofficial Rankings

[Andy Troeger]

Tim,
Fair point and I think I should have separated Pebble and Muirfield Village. Although, come to think of it I'm not sure how anyone rated Muirfield Village as a 5 either. Even if its not one's cup of tea its easily worth taking a day to visit.

[Matt_Ward]

What's really silly with all this "consensus" driven assessments is what Tom D mentioned -- you will have people who will vote way high or low for certain courses in order to spike the final outcome.

Tim B -- you are 1000% correct -- it is inane for anyone to see PB as a five (5). It's really an attempt to bump the course down a peg or two by applying a number that might sway the overall assessment.

The "sabotage" factor, as Doak described it -- lives when you have people who can vote without being specifically IDed.

Candidly, the whole system fails because even if people actually applied low or high numbers because of true feelings -- the people doing the compiling might still believe something is amiss because of their OWN viewpoints on such courses.

Once you get into the guessing game about what people are really saying it's akin to the Bradley effect that many people cited prior to the Obama election. What people say and what people really think can be different.

[Ian_L]

Here is a new list with more policing than the previous version.  By cutting down on outliers and looking a bit more at suspicious voting patterns I've decreased the standard deviation on some of the courses.  Pebble did not increase much mostly because there are a lot of 8's and 9's. 



Honestly, I don't think it's worth putting much more effort into preventing cheaters.  I think this list in general is fairly accurate, with some exceptions.  I really don't care to play policeman, since it will probably result in my own bias showing in the rankings. The simple truth is, the list will never be perfect, and there is always some idiot trying to tamper with the results.

 If you would like a more complete list to rate as Tom and others suggested earlier, please post here.  If there is a lot of interest, I'd be willing to look into that possibility (again, with Ran's permission), and we could discuss the best way to proceed.  If not, I hope you enjoyed the activity, and thanks for participating.

[Sean_A]

Ian

It would be interesting to see the "next" ten (or less) from folks who think their courses can challenge an average of 6.72 (7) with a minimum number of votes (don't know what a good number would be).  I wonder how many "write ins" would get into the top 100.  Matthew's list suggest there may be quite a few. 

Ciao

[Jim Nugent]

Based on what people have said about various courses here over the years, the list looks pretty accurate to me.  i.e. it probably gives us a pretty good average of the tastes of GCA.com posters.

e.g. lots of people on this forum feel Pebble has many ordinary (or less) holes.  ANGC gets attacked often, because the club has changed the course so radically from the way Mack designed it.  No surprise that these courses got booted down a lot in the ratings.

On the other hand, most people in this forum love Doak's courses.  He could probably get more in the top 100.  e.g. I have heard raves for Rock Creek.  Seems like almost everyone who played it would give it an 8 or 9.  That might put it in the top 30 or even 20.  And I will make a wild prediction -- really go out on a limb -- and suggest that Old Mac will also be very popular with the GCA.com posters, when it opens.  In a few years, Doak could easily have 6 or more courses in the top 30. 

If I counted right, 7 of Golf Mag's top 10 are also top 10 in this list.  16 of the top 20.  23 of the top 30 and 32 of the top 40.  So the differences are not all that great.   

[Adam Clayman]

This is why every time the suggestion has been made to create a gca.com list it is not embraced by the powers that be.

[Tom_Doak]

Adam:

What is?

[Charlie Goerges]

At the very least, this type of exercise is worthwhile just to show people how hard it is to create a decent ranking (not that this is isn't one). Sorry if this point has been made, I didn't read every post.

At any rate, the subject is exactly the sort of thing that boards like this exist to discuss. If it's possible to like one course more than another, then it's possible to rank them. If it's possible, then it will be done.

Charlie

[Adam Clayman]

Sorry Tom. I was posting from my phone and the lag time was greater than I thought.
I was mainly referring to the part about the integrity of the information gathering.

 Vindictive voting, after being asked nicely to follow the rules of the survey, illustrates just how ugly and petty certain people can be. What a pity!

They have a lot to learn about understanding the spirit of this sport, and obviously, their fields of play.

One great aspect about gca.com, it mimics the sport in identifying character.

[Tom_Doak]

Ian:

I would encourage you to go ahead and expand this.  If you do, I'm betting the bottom half of your list will include a dozen courses which are full of character but overlooked by the magazine panelists for various reasons ... just as North Berwick used to fall short there but is really high on this list.

You would just have to figure out how to cross-reference the voters' names against the list of registered users on GCA.com, and then keep certain people from voting 82 times!

[Jonathan Cummings]

You know - Ian's list, while informal, has all the ingredients to be the best top 100 world list of them all.  Ian (with Ran's blessing) please consider adding another 100 or so courses to your ballot and try this exercise again.  With a very little amount of policing (which you seem to be doing) your generated rankings may have more integrity than any of the magazine lists.

JC

[Matt_Ward]

Not to bust anyone's bubble -- but the exclusion of Plainfield while Baltusrol / Lower and Somerset Hills, to name just two that are listed, is rather glaring error.

Ditto the fact that Winged Foot / West is behind the likes of area courses like Fisher's Island and Garden City GC, is another error, in my mind.

Another item -- Shadow Creek ?

Please in the "invented" golf course category one can easily include the likes of Wolf Creek in NV and Bayonne in NJ as better overall examples.

Final item for now -- Congressional / Blue and East Lake are fine layouts but not top 100 in the world.

[Tom_Doak]

Matt:

Ian only took votes on the top 100 as listed by GOLF Magazine, and re-ordered them based on those rankings.  Which explains all of your re-hashed complaints about the GOLF Magazine list.

[Rob Rigg]

Ian,

If you make GCA.com users create their own name/pw to vote then you can bust people for derelict behavior or at least ask them what the deal is if their rating is several standard deviations away from the norm.

[Ian_L]

Hi all,

Ran has approved continuing this activity, provided we make it clear that it is NOT endorsed by GolfClubAtlas.com .  Now, continue this process, there are several points that need to be addressed.  I would like to get your opinions on these and any other issues to discuss.

Voting Security

There are three options I see to try to ensure legal voting.

1. Continue as we did in the first experiment.
Pros: anonymous (voter does not feel social pressure), user-friendly.
Cons: easy to vote multiple times, no accountability for blatant misuse.

2, Require the participant to enter GCA user name in the survey.
Pros: increased accountability, each user can only vote once.
Cons: participants could easily enter another's user name to vote multiple times (voter fraud).

3. Participant sends me a private message (assuming I run the survey) containing his email address, and I send him an e-mail invite to participate in the survey.
Pros: voter fraud nearly impossible, participant can only vote once, survey closed off to outsiders, increased accountability.
Cons: privacy issue (some people might not want to give me their e-mail), much more involved for the participant (might turn off voters), I would need to switch to SurveyMonkey.com , meaning only bare-bones statistics would be available (average, number of votes).  My editing powers would also be slightly reduced.
 
4. Your suggestion.

Forming the List

1. How many courses? Suggestions have so far varied between 200 and 400.  Or should there be a cutoff at all?

2.  What will the nomination process be?  I could make a form for write-ins, or simply create a new thread in the discussion forum.

3. Should the current Top 100 be automatically included?


Please feel free to bring up any other issues/problems you see.

[Carl Nichols]

Ian:
FWIW, I would vote for 2.  The possibility of people using others' user names seems unlikely to me, and if you get a situation where someone tries to vote for the first time but can't, you can just void the original vote.  It also seems like a lot less of a hassle for you.

[Matthew Hunt]

What about having to fill in a reason for giving a course for five or less. To make it more accurate allow haolf votes because one could be stuck betweeen giving Sand Hills aa 9 or 10 or Bandon 7 or 8.

[Tom_Doak]

Ian:

I agree with Carl's suggestion as to limiting voter fraud.  Seems unlikely that there are too many a-holes out there trying to sabotage your list ... the bigger worry is just certain posters trying to stack the deck.

As for Matthew's suggestion, limit the voting scale to between 5 and 10.  The average course on the ballot will be a 7 or an 8, so that gives people just as much chance to push a course down as they can build it up -- but no more.

I think your ballot should be about 400 courses.  For simplicity, you could just start with GOLFWEEK's two top 100 lists for U.S. courses (of course, you'll miss Rock Creek), and the GOLF WORLD list of the top 100 in the UK and Ireland (or maybe the top 50-75 off that list).  I'd be glad to contribute the list of overseas courses on which GOLF Magazine votes, if you want.

The one complication of adding to the ballot is that you will find some voters are tougher than others.  In a large sample, you'll find that some panelists give out a lot of 9's [thus increasing the stature of every course they have gone to see], while others in trying to discriminate will give out too many 6's [thus pushing down all the courses they don't favor].  In the end, you'll just have to decide whether to disqualify those voters or not.

[Matt_Ward]

Gents:

The problem in doing such a grand exercise like this is that you then have to be some sort of seer and try to FILTER out elements based on solely one's gut opinion on the motivations of countless voters. Where does that line get drawn -- who draws it -- etc, etc, etc.

I mean you have to decide whether to intervene with one person and not with another.

All of this simply complicates the overall final result.

Doak hit the nail squarely on the head -- in any consensus driven formula you will have certain "tough" voters and certain "more accepting voters."

Once you start to play the role of whether a vote is a "hanging chat" or whether it's accurate you run the risk in having a listing that will either be politically correct (as so many are) or you might run the risk in having a listing that is indeed so "out there" that few will embrace it.

In sum -- it's a great exercise for those so inclined but it really has its clear limitations.

So be it ...

[Jonathan Cummings]

Matt - true, but you know I have always contended that there is some sound numerical policing that can and should be done to any ranking database. 

Ian - let me have the flat sheet when you are finished (raters by courses), blank out both rater names and course names and substitute 1-300 vs. a-xx.  I will be able to calculate three pieces of useful information that you may choose to apply to "correct" your raw database: rater bias (low or high trends to the average), rater dispersion (spread in rater voting - suggesting degree of random voting) and confidence limits (set a 95% confidence and I will be able to tell you how many votes should be a minimum for each course before it is included in the list). 

There are sound statistical corrections you can perform on this or any scientific set of cause and effect data.  In the measurement world (which I've been employed for 29 years) you'd be laughed out of any technical conference if you hadn't employed statistical analysis to your data.  The rest (corrected average rankings) is subjective which is the interested and fun of Ian's exercise. 

JC

[Ian_L]

Looks like option number 2 is the preferred way to go in the voting security category.  Each participant will be required to insert his/her user name.  I think the names themselves should still remain as private as possible.

Tom, I would be interested in seeing the Golf Magazine list you mentioned.  If you would like to e-mail it to me my address is ian8389 at gmail.com .  Posting it in this thread works fine too.  If I understand correctly, your suggestion is to combine these lists, and then have a GCA nomination process for other potential Top 100 courses?

Matthew, while I think your suggestion is a good idea, I bet you could come up with reasons to give any course a 4.  After all, it's just your personal opinion.  I think it might be easier to use statistical methods to determine these outliers, and perhaps make the minimum rating 5 as Tom suggested.  However, your idea is certainly something to be considered.

Unfortunately, the form won't let me give options of half-increments.  But isn't part of the fun trying to make that decision when you're torn between an 8 and a 9?

Jonathan, I would be interested in more specific information about how your statistical methods work.  Unfortunately, I don't actually know much about stats, so I don't know how much you'll be able to explain to me... I've e-mailed you some raw data from the previous exercise.

Matt, I agree.

[Matt_Ward]

Jonathan:

The flaws are quite simple -- you can filter only so much. When you have a consensus driven formula you have people who have played course "A" but then a number of them have not played course "B" and then some will have played "A" but not "C" or "D".

The issue as Doak mentioned is quite correct -- you get people who throw forward a nine (9) score when reality would say such a course is no more than a seven (7). Some opt the other way with lower numbers.

All the math in the world doesn't change the fact that unless you really screen who the people who are participating you have so much possibility for mischief of one type or the other. At the end of the day -- ratings don't lend themselves to a consensus driven formula -- that's just me saying that. I'd much rather see a listing from an individual because at least I know where that person stands and don't have to play this endless game of guessing if the people who participated did so because of one reason or the other.

Please don't misunderstand me to think that what Ian is attempting isn't fun to do. I just don't see the end result being meaningful.

[Rob Rigg]

Jonathan has a really good idea - by building confidence intervals it can be determined whether or not the votes are statistically significant or not.

We may run into minimum sample size issues though since some courses will have few votes scattered across several rankings.

I hope that makes sense, my stats memory is a bit rusty.

[Matt_Ward]

Rob:

You hit on one of my previous points - there are people who think that a course with lesser number of voters is somehow not "eligible" for consideration. Too many people erroneously believe that if you have a course that is played more it therefore should be rated higher.

Candidly, plenty of courses from the mountain time zone have really made an impression with me and they often get little notice or votes because of their remote locations.

The ratings in so many ways are nothing more than a reaffirmation of the same predictable layouts time after time. No doubt some of them are worthy of such a continued high position - there are more than few that should have been tossed some time ago.

[Rob Rigg]

Matt,
 
Agreed, of course this applies to any broad ranking that is tabulated.

eg) Restaurants - if you are located in NYC, SF, London, LA, etc. it is a layup to get on the radar of magazines, national papers, etc. but if you are off the beaten track it can take years to get any love and sometimes you are just lucky that the right person is passing through town.

It is likely that courses like RCCC and Ballyneal will continue to rate very well with the knowledgable and well traveled members of GCA but continue to lurk on the fringes of magazine rankings because enough people do not get out and play them.

This is another one of the reasons that a broad ranking based on the input from GCA members would be quite interesting. Of course, there a couple of biases on the site that will skew the rankings in a different way . . .

Nothing is perfect.