Now, I do not really believe there is a precise formula that can be used.  What stands out about the greatest of courses is how different they are from one another ... how they take some aspect of golf architecture and raise it to a new level not contemplated by other courses.

But it's clear that GOLF DIGEST is bound to stay with its system, and its inherent flaw is to assume that all its categories should be weighted equally [except Shot Values, which are doubly important, whatever they are].

What I would like to see is for everyone here to try and define what the criteria should be, and to assign weights to the relative importance of each.  You can start with the current GOLF DIGEST or GOLFWEEK criteria if you want, or you can make up your own ... though try to make it a short list, please!

While you get started, I'm going to go look up Charles Blair Macdonald's criteria and post them.

Tom: It is an interesting point, but I think many of Macdonald's points are as arbitrary as the ones GD uses. I mean, "Nature of Soil?" Come on. Seems to me he was also judging conditioning ("quality of turf"), which is one of those factors that gets the folks on this site all worked up. But "width of fair green?" Maybe the best comparision these days would be "over treed."
That said, I think "Proper placing of bunkers and other hazards" is interesting, as is "Variety and arrangement of length."

But doesn't Macdonald's criteria simply prove there is no proper way to determine the factors that are used to judge a great golf course? I used to think Golf magazine's lack of criteria hurt it, but now I'm not so sure....

This is a mix of MacDonald's and Mackenzie's "IDEALS"  Not sure exactly how to score the following, but these are preliminary.  This should get things going.

I ask "Does the course"

1. Appeal to different levels of golfer?
2. Have variety in Greens, size, undulation, width, length, etc?
3. Character in its Bunkers?
4. Have a Good Routing, minimize distances between holes, flow with the land, use the best features the land presents?
5. Have PROPER PLACEMENT of above features and various other hazards?
6. Present a "Walk in the Park?" From Golfweek

An interesting question to ask as well, "How does the course presently compare to the architects original design.  Has the course been modified for the better or for worse.  Does it have that TIMELESS quality?

Adam - the problem with employing weights for revisits is that it is a great idea but simply impractical  All panels establish their lists on probably 80% one-time visits.  I know that of the 820 courses I've seen, probably 700 I've seen and played only once.  I'll but the same is true with Brad, Tom, Bobby Fagan, you, etc.

Tom - here is a list of categories that we proposed in a poll at the Essex gathering.  Each participant was asked to give a 1-to-10 on how meaninging and measureable these categories to their evaluation of a golf course.  I averaged the results and ranked them.  I computed weightings (based on the product of the meaningful and measureable averages) to the ten highest scoring categories and used them to perform a sample rankings.

I think I am going to open this to the GCA folks and see if there is any interest in expanding the poll.    

(sorry for the poor pagination - Word tables don't translate well to quasi-html scripting used by YaBB SE.

JC

Rank   Proposed Category   Meaningful   Measurable   Sum
            
1   Interest of Greens & Chipping   8.8   8.1   16.9
2   Memorability   8.6   7.9   16.5
3   Variety/Mem of Par 5s   8.4   7.6   16.0
4   Invasion Factor   7.6   8.3   15.9
5   Design Balance   8.4   7.4   15.8
6   Variety/Mem of Par 4s   8.2   7.5   15.8
7   Variety/Mem of Par 3s   8.3   7.5   15.8
8   Fun   8.6   7.0   15.5
9   Richness/Variety of Problems   7.8   7.6   15.4
10   Lack of Distractions   7.5   7.6   15.1
11   Beauty   7.5   7.3   14.8
12   Shot Value   7.9   6.8   14.7
13   Options (1[few] - 10[many])   7.5   7.0   14.5
14   Playability   7.6   7.0   14.5
15   Integration with Surrounding   7.5   7.0   14.5
16   Conditioning   7.0   7.4   14.5
17   Natural Setting/Land Plan   7.5   6.6   14.1
18   Quality of Shaping   7.1   7.0   14.1
19   Esthetics   7.5   6.5   14.0
20   Resistance to Scoring   6.4   7.5   13.9
21   Walking   5.5   8.2   13.7
22   Integrity of Original Design   7.3   6.2   13.4
23   Ease & Intimacy of Routing   7.0   6.5   13.4
24   Landscape/Tree Management   6.4   6.7   13.1
25   Variations in Stance   6.0   7.0   13.0
26   Sense of Theme   6.7   5.9   12.6
27   Lack of Blindness   4.7   7.9   12.6
28   Fairness   5.9   6.3   12.2
29   Tradition   5.9   6.0   11.9
30   History   5.8   5.9   11.8
31   Ball Striking   4.2   7.0   11.2
32   Walk in the Park   5.2   5.3   10.5

Tom -- I've read Macdonald's book and understand the historical period in which he was writing.
That said, he comments on fairway width would have clearly hurt the likes of TOC, which he used so liberally in his own designs, wouldn't it?
Tom, I'm not trying to rip Macdonald -- just to point out that some of his concepts were more applicable to 1910 than they are to current conditions. Courses changed a lot since then and it seems to me the criteria to judge them should as well.

Jonathan: What a list of categories. I find that fillling out the Golf Digest categories takes some time, if you really give it some thought. Apparently this one would take hours....

Tom D:

Hate to say this but you missed the point on what is really needed ... it's not the criteria because in all likelihood most people would come together on what are the main criteria for discussion.

The issue is what kind of person is capable in doing what you seek -- that is -- to identify the best courses in the various states in the USA.

Once a publication goes down the sorry road in having a gigantic ratings panel you inevitably have issues ...

Does the rater really travel THAT often to even be considered a NATIONAL rater? Frankly, few do. Most should be no more than at the regional or state level.

Tom, I've been doing ratings for quite a few number of years. A number of fellow raters I have met on my many forays throughout the countryside -- whether they be from Digest, GW or Golf Mag -- are quite competent to make such judgements. But more diligence needs to take place in terms of who makes the call on courses -- adding more and more people only produces a "yellow pages" directory of information and flies in the face of what is really noteworthy for inclusion at the highest of levels.

The issue you seem to dismiss is that Digest is the one that needs to be approached to change not only the methodology but the nature of what constitutues a "panelist" for their publication. Digest needs to understand that simply putting out a bi-ennial listing that has little credibility only serves to undercut their self-proclaimed position as the golf publication of RECORD.

What's really funny is that until the publication admits its own shortcomings the nature of what comes forward will never really be improved IMHO.

JC- With results that have the volatility of a Toronto Stock Exchange penny stock, impractical is passing off the list as something a bank would actually finance, or re-finance a project on.

While everyone values their own opinion, why would anyone have a problem having their vote only count for 1/4, until your next visit? And if you never visit again, doesn't that state volumes?

Matt:

Start another thread about who ought to be on the panel, or not, if you want.  You'll find there is not much agreement on that subject either ... everybody likes their own opinion best.  GOLF Magazine used to have the best-balanced panel, with a mixture of everyone in the golf business from Tour pros to photographers to people like Ran, but now even I don't know who half their panel is [though that still probably gives them more stature than GOLF DIGEST or GOLFWEEK].

My criteria would be - "What is the best course."  

Scoring should be done on the Doak scale.

Raters should be required to play the course three times at different times of year and to play a sufficient number of courses to have a valid basis for comparison.  Thus, ratings should be made over a three year period.

George:

I have to agree with you [glad that I can say it's you and not Matt].  Rankings are inherently subjective so the most important thing is whom do you trust?  This is how the GOLF Magazine system was conceived and I still think it's the most sensible way, although I trusted the judgment of the people on their panel ten years ago a lot more than the people who have replaced them.

But GOLF DIGEST is not going to change their system ... they insist on a formula and a points system, and when you put that in place the formula you use pretty much pre-determines the results.  That's the subject of this thread.

To Joel's point, anytime you take a composite score based on different criteria, you will be summarizing the results.

Perhaps showing the detail scoring for each criteria will provide additional fodder for discussion and how some of these rankings were achieved ...

Jon:  That's scary.  It looks like you are trying to weed out "esthetics," which is something the architect does have partial control over, but then between "invasion factor" and "lack of distractions" you are basically giving courses without housing a 19-point head start on courses without.  I'm sure there are lots of people who would agree with that, but that makes the work of many golf architects entirely moot.

And being a good test of golf is worth zero?

I nominate the "Ridiculously Good Test!"

I'm borrowing the phrase from someone on this Board that used it once in my presence, but I would say after all the technical tests which are important (variety of angles for tee shots and approach shots, course compatibility with environs, conditioning, routing, shot making requirements or hitting every club in the bag, mix of routing with prevailing wind so you're not always into or down wind, etc.) it separates the great courses from the rest.

It's sort of a Walk in the Park test meets ambiance test.  

It's basically how many times I look at a playing partner and say, or mutter to myself, "This place is ridiculously good!"

One word of caution, the results of play must be kept in mind so as not to confuse appreciation for successful shots or lack thereof with appreciation of ridiculously good architecture.