The Formula for Ranking Great Golf Courses

[Jonathan Cummings]

Hey Tom - this wasn't what "I" was trying to do, this was the averaged results of a poll taken at Essex.  There were 30 responses (small sampling I know) and what I listed was how the number fell.  I had no design or interest in changing the outcome.  

I've always been curious if there is a more scientific way to select and weigh categories.  I would like to do this on a larger scale.  Maybe this is suggesting something scary - courses with housing on them are at a disadvantage in the rating game against courses with no houses.  You have long known where my feeling lie in this subject.

JC

[Joel_Stewart]

To me, if one person thinks a course is extra-special [or super-overrated], that says something more important than whether they gave it a 6 or a 7 in one category, so why would you throw that out?

I might not understand your question but you would throw it out because its an aberration and not consistent.  Every panel has "homers" as you call them and the key is to eliminate those odd votes.

Look I am terribly confused with yesterdays results.  You see moves up like NGLA, Seminole, Sand Hills (the classics) yet modern courses like Mayacama and The Preserve coming from oblivion to Top 100 on the same list?

I have always wrestled with the fact that many GD panelists are impressed with Rees or Fazio and like modern designs but yesterdays results showed either a real weakness in the voting system or the panelists.

[Mike Nuzzo]

Using only the Resistance to scoring, Variety, 20% of aesthics and conditioning, and 50% of the walking:

PineValleyG.C.
ShinnecockHillsG.C.
AugustaNationalG.C.
OakmontC.C.
PebbleBeachG.Links
WingedFootG.C.(West)
MerionG.C.(East)
CypressPointClub
OaklandHillsC.C.(South)
PinehurstResort&C.C.(No.2)
BethpageSt.ParkG.Cse.(Black)
VictoriaNationalG.C.
CrystalDownsC.C.
MedinahC.C.(No.3)
MuirfieldVillageG.C.
WadeHamptonG.C.
SouthernHillsC.C.
TheCountryClub(Clyde/Squirrel)
TheOlympicClub(Lake)
SandHillsG.C.
SeminoleG.C.
QuakerRidgeG.C.
WhistlingStraits(Straits)
OakHillC.C.(East)
NationalGolfLinksofAmerica
PrairieDunesC.C.
TheOceanCourse
ButlerNationalG.C.
OlympiaFieldsC.C.(North)
PacificDunes
WingedFootG.C.(East)
TheHonorsCourse
BandonDunes
LosAngelesC.C.(North)
ShadowCreek
SpyglassHillG.Cse.
FishersIslandClub
KinlochG.C.
BaltusrolG.C.(Lower)
SanFranciscoG.C.
RivieraC.C.
RichHarvestLinks
InvernessClub
FlintHillsNationalG.C.
DallasNationalG.C.
TheGolfClub
LongCoveClub
BlackwolfRun(River)
ShoalCreek
CrookedStickG.C.
ForestHighlandsG.C.(Canyon)
ChicagoG.C.
SandRidgeG.C.
BlackDiamondRanch(Quarry)
HazeltineNationalG.C.
SomersetHillsC.C.
GardenCityG.C.
PlainfieldC.C.
CastlePinesG.C.
InterlachenC.C.
MayacamaG.C.
TPCatSawgrass(Stadium)
ThePrinceCse.
TrumpInternationalG.C.
SciotoC.C.
ArcadiaBluffsG.C.
OceanForestG.C.
CanterburyG.C.
GallowayNationalG.C.
ValhallaG.C.
CongressionalC.C.(Blue)
TheQuarryatLaQuinta
MilwaukeeC.C.
PeachtreeG.C.
CamargoClub
CherryHillsC.C.
HawksRidgeG.C.
AtlanticG.C.
DoubleEagleClub
AroniminkG.C.
TheHomestead(Cascades)
SahaleeC.C.(South/North)
HarbourTownG.Links
EugeneC.C.
HudsonNationalG.C.
PeteDyeG.C.
ColonialC.C.
LaurelValleyG.C.
MaidstoneClub
GrandfatherG.&C.C.
KittansettClub
EastLakeG.C.
ThePreserveG.C.
TheEstanciaClub
TheValleyClubofMontecito
OldWaverlyG.C.
TheG.C.atBriar’sCreek
Sanctuary
SageValleyG.C.
Shoreacres

All of those catagories aren't needed, and it would allow more refinement of the raters and those 2 important catagories.

?

[Tom_Doak]

Joel:

In an ideal world, the "homers" would recuse themselves from voting [just as architects do on their own courses], but you wouldn't throw out a high vote or low vote if a trusted panelist saw more [or less] in a course than his peers did.

The confusion you feel is simply an unveiling of what the "tradition" points used to do ... keep the rest of the definition from showing its true colors.  Courses which are clearly over the "ridiculously good" line like NGLA and Sand Hills and Fishers Island are not affected, but otherwise, the definition of greatness you are left with CLEARLY favors the modern, long, expensively manicured courses over the rest of the classics.

Unless there is someone out there who really thinks Rich Harvest Links and Flint Hills National are ridiculously good [besides their architect, of course].

[Tom_Doak]

Mike N:  With your new formula you've got Butler National above SF, Riviera and Pacific Dunes, and only a couple of spots below NGLA.  And Victoria National is above NGLA and Crystal Downs.  Are you sure that's what you want?

[Mike Nuzzo]

No.
I am exploring simplifying the catagories and wound up with near the same results.
I would guess it to be easier to manage only 2 main cats, rather then double one, plus 4 others.
Let the humans be subjective - don't try and take it away with all of those cats.
And get rid of the homers, hopefully they'll be easier to identify with only 2 cats.

?

[Mike Nuzzo]

Using only the variey catagory:

PineValleyG.C.
ShinnecockHillsG.C.
CypressPointClub
AugustaNationalG.C.
PebbleBeachG.Links
MerionG.C.(East)
OakmontC.C.
NationalGolfLinksofAmerica
CrystalDownsC.C.
FishersIslandClub
WadeHamptonG.C.
SandHillsG.C.
TheCountryClub(Clyde/Squirrel)
PacificDunes
ShadowCreek
VictoriaNationalG.C.
PinehurstResort&C.C.(No.2)
WingedFootG.C.(West)
MuirfieldVillageG.C.
SouthernHillsC.C.
PrairieDunesC.C.
SeminoleG.C.
BethpageSt.ParkG.Cse.(Black)
TheOlympicClub(Lake)
OaklandHillsC.C.(South)
SanFranciscoG.C.
QuakerRidgeG.C.
WingedFootG.C.(East)
KinlochG.C.
BandonDunes
LosAngelesC.C.(North)
OlympiaFieldsC.C.(North)
WhistlingStraits(Straits)
SomersetHillsC.C.
TheGolfClub
RivieraC.C.
TheHonorsCourse
OakHillC.C.(East)
ForestHighlandsG.C.(Canyon)
DallasNationalG.C.
GardenCityG.C.
ThePrinceCse.
BlackDiamondRanch(Quarry)
ChicagoG.C.
MedinahC.C.(No.3)
BlackwolfRun(River)
CamargoClub
FlintHillsNationalG.C.
LongCoveClub
TheOceanCourse
SandRidgeG.C.
MaidstoneClub
CanterburyG.C.
InterlachenC.C.
RichHarvestLinks
SpyglassHillG.Cse.
TheValleyClubofMontecito
MilwaukeeC.C.
TheQuarryatLaQuinta
ArcadiaBluffsG.C.
ButlerNationalG.C.
TrumpInternationalG.C.
PlainfieldC.C.
SciotoC.C.
ShoalCreek
CherryHillsC.C.
Sanctuary
TheHomestead(Cascades)
ValhallaG.C.
TheEstanciaClub
Shoreacres
HawksRidgeG.C.
MayacamaG.C.
PeteDyeG.C.
CastlePinesG.C.
DoubleEagleClub
GallowayNationalG.C.
TPCatSawgrass(Stadium)
CrookedStickG.C.
HarbourTownG.Links
BaltusrolG.C.(Lower)
KittansettClub
GrandfatherG.&C.C.
InvernessClub
PeachtreeG.C.
EugeneC.C.
TheG.C.atBriar’sCreek
LaurelValleyG.C.
OldWaverlyG.C.
ThePreserveG.C.
ColonialC.C.
CongressionalC.C.(Blue)
AtlanticG.C.
HudsonNationalG.C.
OceanForestG.C.
SageValleyG.C.
SahaleeC.C.(South/North)
HazeltineNationalG.C.
EastLakeG.C.
AroniminkG.C.

[Mike Nuzzo]

Only the Resistance to scoring:

ShinnecockHillsG.C.
OakmontC.C.
PineValleyG.C.
WingedFootG.C.(West)
BethpageSt.ParkG.Cse.(Black)
MedinahC.C.(No.3)
OaklandHillsC.C.(South)
PinehurstResort&C.C.(No.2)
AugustaNationalG.C.
TheOceanCourse
TheOlympicClub(Lake)
ButlerNationalG.C.
VictoriaNationalG.C.
SpyglassHillG.Cse.
PebbleBeachG.Links
MerionG.C.(East)
WhistlingStraits(Straits)
OakHillC.C.(East)
ThePrinceCse.
SouthernHillsC.C.
BaltusrolG.C.(Lower)
MuirfieldVillageG.C.
QuakerRidgeG.C.
HazeltineNationalG.C.
CrystalDownsC.C.
TheCountryClub(Clyde/Squirrel)
OlympiaFieldsC.C.(North)
InvernessClub
TPCatSawgrass(Stadium)
TheHonorsCourse
PrairieDunesC.C.
WadeHamptonG.C.
LosAngelesC.C.(North)
WingedFootG.C.(East)
CrookedStickG.C.
SeminoleG.C.
BandonDunes
SandHillsG.C.
RivieraC.C.
AroniminkG.C.
BlackwolfRun(River)
LongCoveClub
PlainfieldC.C.
CongressionalC.C.(Blue)
DallasNationalG.C.
CypressPointClub
CastlePinesG.C.
PacificDunes
OceanForestG.C.
RichHarvestLinks
PeachtreeG.C.
GallowayNationalG.C.
FlintHillsNationalG.C.
MayacamaG.C.
TheGolfClub
ArcadiaBluffsG.C.
TrumpInternationalG.C.
ShoalCreek
ValhallaG.C.
ForestHighlandsG.C.(Canyon)
HarbourTownG.Links
AtlanticG.C.
SahaleeC.C.(South/North)
ColonialC.C.
ShadowCreek
EastLakeG.C.
KinlochG.C.
HudsonNationalG.C.
BlackDiamondRanch(Quarry)
SanFranciscoG.C.
SandRidgeG.C.
SciotoC.C.
HawksRidgeG.C.
NationalGolfLinksofAmerica
EugeneC.C.
CanterburyG.C.
PeteDyeG.C.
TheHomestead(Cascades)
GardenCityG.C.
ChicagoG.C.
InterlachenC.C.
CherryHillsC.C.
ThePreserveG.C.
TheQuarryatLaQuinta
FishersIslandClub
LaurelValleyG.C.
GrandfatherG.&C.C.
Sanctuary
DoubleEagleClub
SomersetHillsC.C.
TheG.C.atBriar’sCreek
OldWaverlyG.C.
MilwaukeeC.C.
CamargoClub
TheEstanciaClub
MaidstoneClub
KittansettClub
SageValleyG.C.
TheValleyClubofMontecito
Shoreacres


Where are the Homers?

[George Pazin]

Keep going, Mike, there's got to be some iteration that will result in Oakmont #1....

Tom -

I think I understand what you're trying to say, I'm simply pointing out that the raters involved will likely skew the scores in various categories, no matter what the category is, to achieve the end result they desire. In other words, I don't think there is a magical formula to overcome the human element, and thus we're stuck with a list that favors what the raters favor, long manicured modern courses.

[Patrick_Mucci]

Here's CB:  from GOLF ILLUSTRATED, February 1, 1907:

Nature of the soil - 18 points
Perfection in undulation and hillocks - 15 points
Putting greens - quality of turf - 10 points
Putting greens - nature of undulation - 5 points
Putting greens - variety - 3 points
Nature, size and variety of bunkers and other hazards - 7 points
Proper placing of bunkers and other hazards - 11 points
Best length of holes - 12 points
Variety and arrangement of length - 6 points
Quality of turf of fair green - 8 points
Width of fair green of the course, 45 to 60 yards - 3 points
Nature of teeing grounds and proximity to putting greens - 2 points

Tom Doak,

If you were to use CBM's categories, what weighting, or points would you assign to each category ?

I was somewhat surprised to see more weight credited to undulations in the fairways then undulations in the putting greens.[/color]

[Mike Nuzzo]

Pat,
I would say that fairways are a wrongly undervalued element in today's rankings and game.  They should be weighted more.

[THuckaby2]

I find no disgrace and in fact some honor in saying I agree with Matt also.  The best way to do this would be to have a smallish group of frequently travelling golfers, all of whom would be very expert and give a national context to their assessments.

Where I likely differ from Matt is that I see this as having as much chance of happening as the San Jose Giants winning the world series this year (they're a single A team).

Where will these people come from?  How will they be compensated for the vast amounts of time this will require?  Who will do the compensating?

This is one of those ideas that's great in theory, but has little relevance in the real world.  It ain't gonna happen.

Thus it is much more effective to take Tom Doak's question at face value, and attempt to ascertain the proper criteria and worth of each, to make a valuable rating methodology.  THIS has a chance of being effected... each magazine (but particularly GD, I think) does seem willing to tweak its methodology in attempt to get this right.  It's not outside the realm of possibility that if a group like this came up with a cogent revised system, it might be adopted in whole or in part by one of the magazines.

So to that end, I think Jonathan Cummings is really on the right track... we just need to refine that a bit.

So put me down for somehow allowing for esthetics, as well as ambiance and tradition or feel or whatever you want to call it.  I believe that does matter.  It would be weighted quite lightly, for sure, relative to the other more important criteria.  But it ought to count for something.  The rest of it can go just how JC's compilation came out....

TH

[Mike Nuzzo]

You guys are out of control with all this formula stuff.  

Sean,
Who are the "guys"?
Is there anyone else besides me out of control?  
They aren't formulas, I've separated the elements.
Which list is more represenative of your tastes:
The Variety list or the Resistance to Scoring list?

[Andy Levett]

Here's CB:  from GOLF ILLUSTRATED, February 1, 1907:

In discussing and comparing the merits of various courses, one is struck immediately with the futility of argument unless some basis of excellence is agreed upon -- premises on which to anchor.  In view of this, I have tried to enumerate all the essential features of a perfect golf course in accordance with the enlightened criticism of to-day, and to give each of these essential characteristics a value, the sum total of which would be 100, or perfection.  Following is the result:

Nature of the soil - 18 points
Perfection in undulation and hillocks - 15 points
Putting greens - quality of turf - 10 points
Putting greens - nature of undulation - 5 points
Putting greens - variety - 3 points
Nature, size and variety of bunkers and other hazards - 7 points
Proper placing of bunkers and other hazards - 11 points
Best length of holes - 12 points
Variety and arrangement of length - 6 points
Quality of turf of fair green - 8 points
Width of fair green of the course, 45 to 60 yards - 3 points
Nature of teeing grounds and proximity to putting greens - 2 points

That's a pretty interesting start, and it doesn't look anything at all like the two ranking systems at the magazines.

Interesting people have jumped on CBM for weighting ‘nature of soil’ so heavily. I tend to think he got it about right.

 However great the architect, however large the budget, few suggest any of the clay courses in the UK can hold a candle to links and heathland. The only clay courses in the GB & I Top 100 lists are a handful of  recent openings like Loch Lomond and Mount Juliet.

 Colt and MacKenzie  did great, largely unsung, architecture on the heavy inland soil that characterises most of  northern England, where I live. But their art counts for little against the natural advantages of the land enjoyed on the coast, where men of less ability and application achieved finer results thanks to the quality of the materials they were working with.

On a less parochial note, it puzzles me that most of the great GB & I links are ranked lower than clay courses like ANGC, Oakmont and Winged Foot on world lists. Never played the US courses, probably never will, but am curious for those who have done so to make a case for them ahead of the likes of Portrush and Dornoch.

[Jim_Kennedy]

"Resistance to Usurpers" - A course in the top 25 can not be knocked out of it's position by any other course unless the balloteers have played them both.

[Patrick_Mucci]

Pat,
I would say that fairways are a wrongly undervalued element in today's rankings and game.  They should be weighted more.

Mike,

More then greens ?

What would Donald Ross say about that ?[/color]

Jim Kennedy,

Agreed ![/color]


Tom Doak,

Don't you think that THE critical factor in any system that requires judgement is the quality of the analyst ?[/color]

[Mike Nuzzo]

Pat,
My intent wasn't more than greens, but more than they appear to be weighted.
What would NGLA be like with flat fairways?
What would NGLA be like with flat greens?
Which would you rather play?

I've never met the Donald...I couldn't say.  

[Matt_Ward]

Huck:

There are people doing what I suggested already. The key people at the magazine know who they are. It's not a CIA secret. I have met plenty of people in my travels who visit as many, if not more courses, than I.

They don't need to be compensated. I know these people are deeply passionate about the game and clearly independent of their choices from what I prefer.

No matter what the criteria is -- the formula rests upon the application of the rater. Cure the rater issue and you nip in the bud plenty of the oddities that exist today. You will never remove all the oddities because ratings will always be subjective.

But one thing you don't need is the yellow pages additions of raters. That simply nullifies the viewpoints of those who are fully cognizant of what needs to be done and are removed from the blatant "homer" assessments that often come with such involvement.

[Paul_Turner]

Shouldn't courses be given points for offering something unique?  Uniquely interesting but not gimmickery.

[Robert Thompson]

Matt: I guess I don't understand how having fewer raters would make the situation better. And how are you going to determine who these raters are? The ones who have your aesthetic viewpoint on golf? Mr. Whitten's?
As I see it, having a large panel should help remove the occasions when one person's take has too much influence on the outcome.
Maybe, just maybe, Golf Digest gets exactly what it wants out of its ratings. Perhaps the magazine's conclusions are different than what those on this posting board would approve of. But GCA is only a small group of individuals -- none of which can apparently even determine what criteria they would use to rate courses.
All of that said, when I input my ratings for GD, I give it a lot of thought, take some notes, carefully consider the various facets of the course. Despite all of this work, a course like The Rock in Canada wins best new in GD, which seemed unreasonable to me. I concluded that there must be a lot of raters out there that simply saw the course entirely differently. Or, it is possible they simply ignored the ratings guidelines put forth by GD.

I have a hard time reconciling all of this -- but I continue to try and enjoy participating in Golf Digest's panel. It gives me access to some courses I would never have seen so I can continue to build my knowledge base when it comes to courses and architecture.

[Doug Siebert]

I don't think you can quantify this stuff down to formulas and point systems.  I wonder if a lot of the reason so many of us like the Doak Scale ratings was precisely because Tom Doak didn't go assigning percentages to various categories and calculating the scores.  A nice simple system that in my mind boils down to "how much hassle is it worth for the opportunity to see this course?"

If he'd come up with a complex system like some describe, I think that instead of arguing about individual ratings that some disagreed with, perhaps a '5' he assigned that someone feels should be a '7' or a '3', they'd be more likely to try to poke holes into the system he used to come up with the numbers.  Either the categories are wrong, or the percentages are wrong, or the application was wrong.  So many ways to criticize.  As it is, all anyone can say is that they think a given course is more or less worthy than he says, but it just comes down to a simple matter of opinion.

And since you have humans making the ratings and they all have their biases and baggage, why not just let them assign a number from 0 to 10, based loosely on some guidelines to how much hassle you recommend someone should go through to play this course?  If you believe they should drop everything and fly halfway across the world at last minute airfares for a once in a lifetime chance to play a course, as some might do for say a Pine Valley or ANGC, its a 10.  If you don't think they should play it even if it is next door to their house, they can play for free, and the next closest course is 30 miles away and costs $200, its a 0.  Yes, you'll get people who will believe a course is worth travelling across the country to play because it is just so green and has excellent flowerbeds filled with many colorful flowers in full bloom that spell the hole number by each teebox and has really nifty waterfalls, but aren't those people going to screw with the numbers and give it a high score for categories they shouldn't anyway, like "shot values"?  I could see that because they'll interpret that phrase to mean that you get so more value for your money per shot when you get to listen to the waterfall burble while you putt!

[Tom_Doak]

Doug:

You have just described the GOLF Magazine voting system, although they use letter grades instead of 0-10, because 0-10 leaves more latitude for curmudgeons to put most courses at 4-7 and optimists to put most courses from 6-9.  [Think of the difference between Michelin's three stars where most places get zero, and GOLF DIGEST's Places to Play where the majority of courses get three stars out of five.]

Paul T:

Uniqueness [or originality] gets high marks in the Doak scale, and on my ballot for GOLF.  I'm amazed more people don't agree, but I've had several arguments with the esteemed Mr. Morrissett on this point, because he couldn't rate his beloved Yeamans Hall [among others] so highly if originality counted for much.

Indeed uniqueness is the fundamental break between GOLF DIGEST's idea of greatness and mine.  When you have a formula for greatness, then once you have found something which gets high marks you could theoretically just keep repeating it and continue to get those high marks.  This is how some architects believe many of their courses should be ranked highly ... they think their "formula" is superior to all the older courses which came before.  I could not disagree more strongly.

[Patrick_Mucci]

Paul,

Shouldn't courses be given points for offering something unique?  Uniquely interesting but not gimmickery.

Who would define and draw the distinction ?[/color]

[jeffwarne]

If the GD rankings were universally accepted and agreed upon  by GCA,wouldn't the treehouse get a bit crowded and thus cease to have a reason to exist.

[THuckaby2]

Huck:

There are people doing what I suggested already. The key people at the magazine know who they are. It's not a CIA secret. I have met plenty of people in my travels who visit as many, if not more courses, than I.

They don't need to be compensated. I know these people are deeply passionate about the game and clearly independent of their choices from what I prefer.

No matter what the criteria is -- the formula rests upon the application of the rater. Cure the rater issue and you nip in the bud plenty of the oddities that exist today. You will never remove all the oddities because ratings will always be subjective.

But one thing you don't need is the yellow pages additions of raters. That simply nullifies the viewpoints of those who are fully cognizant of what needs to be done and are removed from the blatant "homer" assessments that often come with such involvement.

Matt:

I'm sure there are others who travel as much as you.  And I'm sure there are several who wouldn't need to be compensated to do this.  BUT... could you ever get enough such that you overcome the "groupspeak" and "views of a few" massive problems inherent in having too few people do this?

I'd agree that one can get to a status of having too many raters, as you say GD has.  But to me that's far better than too few.

Because as much as I respect your opinion, well... if all ratings are based on just what YOU think, that's not going to mean much more beyond "Matt Ward's Top 100."  A system that takes in many different viewpoints eliminates possibilities of bias and personal preferences, and that has to be VERY important.

So you do need a significant number to make this work... You seem to have suggested 100 before.... do you really think there are 100 people, nationwide, who would do this full-time, at their own expense?  Because that is what it would take, to see all the courses that needed to be seen....

I can't believe that's possible.

And even if it is, well then the issue of "rich guy's viewpoint" rises it's head.  Obviously only those of independent means, so to speak, could do this.  Do we really want ONLY their viewpoint for assessment of golf courses?  Man it seems to me very important to have some non-wealthy people involved also....

Hopefully you can see the issues here.  To me, they overwhelm the reality of making this happen... or in any case, overwhelm the possibility of this being an improvement on the current way if it's just 100 rich guys....

Thus to me it would seem much more practical and realistic and beneficial to tweak the methodology.  That can be done.  Oh, as part of this it couldn't hurt to reign in some raters, or cull the troops... But the main thing is fix the methodology and all questions get answered... far better than coming up with a small group of super-wealthy super-raters.

TH