I think Adrian Stiff's effort to come up with definitive rules for evaluating the relative merits of golf courses proves it's not possible. That's because you cannot go from a statement of fact to a statement of judgment. The whole realm of aesthetics is, by its very nature, an experiential, emotional, personal one. But that doesn't mean that anything goes. The responsibility of a good rater, or of the person who runs a rating team, is to develop criteria, to explain, to educate, and to encourage others to deepen their understanding of the art and to learn to appreciate what they mean when they say that some courses are better than others, that they have more appeal, they are more interesting, more compelling. That's certainly not a numbers game -- it has absolutely nothing to do with length, width or scorecards. it has to do with feel, locality, a sense of uniqueness, of the nature of beauty.
The idea that ratings panels teat all raters as equals is garbage. We select, train, educate, ask them to attend events, encourage them to play as many different courses in as many different locales as possible, present them with lectures, encourage them to read. Some raters who do a lousy job get thrown off -- which by definition means we don't think their effort is sincere, or they are hopelessly biased, or lazy, or a liability to the panel. That surely doesn't mean that every rater is equally qualified or equally skilled or adept. But my job as a critic is always to educate, to explain, the give reasons for why I think certain ways and to commit to that process publicly in print, both in words and in numbers.
There's certainly nothing objective about it. No more than rescanning an out-of-focus original photograph will ever get it more tightly focused. But the ratings entail an exercise that helps people learn about the craft, helps them appreciate the game more, probably makes them better players because they become a little smarter about the ground they are contesting, and awakes others as to the loose consensus about which courses are widely deemed to be better than others. In this sense it's less an absolutist declaration of good/bad than a family resemblance (a la Wittgenstein) regarding recognizable similarities.
On an additional note, I find it hysterically funny that one of the fiercest critics of the whole process immodestly (as is his wont) declares himself an informal adviser to some raters. This sounds to me much like Machiavelli's preface to "The Prince," where from the lowly vantage point of exile on his farm (to which he has been banished for his suspect political affiliations), he begs Lorenzo de Medici to be allowed to counsel him on the secrets of gaining and securing power.