The Unfortunate Side of Course Rankings

[Brian Potash]

As some of you may know, I have been the Green Chairman at Hudson National Golf Club in Westchester, NY since last fall.

On the Tom Fazio Best Courses thread earlier this week, Tom Doak said "I honestly can't believe that Hudson National is in a top-100 list."

Of course I wasn't thrilled to hear those comments, but they got me thinking a bit about the whole rankings system, especially as it applies to a club that falls at the tail end of the current GD top 100 (I think Hudson has been between 83 and 94 for the last 4 ranking cycles).



We now have a new Super, Ryan Oliver, who is off to a tremendous start.  The members are very happy and the volume of play is up significantly.  Ryan is a tireless worker, who is doing everything in his power to present our course in the best light possible.  He has been a pleasure to work with, and as I had hoped and expected, has made my job as Green Chair both easy and fun.

**As a quick aside, and I am sure this is very prevelant at many courses in America - If the course is rated 90 in the GD rankings, most members do not think that "A panalist of GD raters feel that we are the 90th best course in the country.  They feel that "Hudson National is the 90th best course in the country".  In other words, for better of worse the GD ranking number is a fact and not an opinion.  I'd suspect that 80% of club members do not know that Golf Magazine rates courses, and that 98% of club members have never heard heard of Golfweek.**

When members see a course rated 90, instead of feeling lucky that they get to play a course that is rated in the top 1% of courses in the country, many want to know what needs to be done and how long will it take to move up to the top 50.   Most times I suspect that there is absoulutely nothing to be done once the bones of the course are in place.

What concerns me is that many members will ultimately view Ryan's success or failure in December 14' when the next rankings come out.  Regardless of where the course ends up, I think this is tremendously unfortunate, and a disservice to a man who is doing a terrific job.

Wondering what others think.......

Thanks,

Brian

[Sam Morrow]

Brian,

 No offense but if I were you I wouldn't care, do what you can do to make it a better club. If you do this and the course falls in the rankings don't worry.

[Bryan Icenhower]

How many of the factors that go into the overall score (and hence ranking) does he have control over?

[Nigel Islam]

I'm in the same situation you are Brian. I belong to a club ranked at the bottom of the top 100, but realistically I realize it probably should be out when you consider the dozen courses such as Camargo that don't even count in the top 100 because of lack of ballots. Honestly the courses that are ranked from around 40ish to 200 (and maybe even more courses than that) on GD are very, very close. A difference of 7 spots is negligible. It is a shame that rating can influence people's jobs, but at the same time ratings help motivate me to learn more about certain courses, so I personally am interested in looking at them.

[Ted Sturges]

Brian,

Interesting topic.  

My view is that you should spend your energy attempting to educate members that these rankings aren't worth the paper they are written on.  I live about a mile north of Crooked Stick.  Crooked Stick has hung around the bottom 15 in the GD top 100 list (in and out of that list) for the last decade.  Mr. Dye has continued to "tinker" with the course, which has generally resulted in the course being ranked lower rather than higher.  Yes, the members like to brag that they make those sort of lists, but until more golf knowledgable people begin to realize that the opinions of the GD panelists are much less valuable than the rank and file golfers believe, the "unfortunate side" of the rankings phenomenon, as you describe it, will exist.

I remember playing Fishers Island with one of the long time members there a few years ago.  He told me that his club was pleased with the fact that not enough GD panelists had gained access to the course for it to get a higher ranking than it had at the time.  I thought that was awesome.  At that time, that club couldn't care less what some magazine thought of their course (they knew they had one of the best courses in the world).  Last summer Tom Doak told me that the leadership at Crystal Downs paid close attention to where they were on those lists, and I was shocked that their club didn't have an attitude toward the rankings more like FI.  

Educating your members that (a) the GD list has had lots of "misrankings" and embarrasing course ommissions over the years, and (b) courses can't really do anything to move "up" (just ask Pete Dye about his work at CS).  Setting a different expectation for those members at your club will help them gain more in their golf architecture education and not put so much (or any) weight into the rankings in their club decision making process.

TS

[David_Tepper]

Brian P. -

I sympathize with your situation. Whenever the subject of rankings comes up with your members, I would ask them, "which is more important, whether you enjoy being a member of this club and playing this course 365 days of the year or worrying about where the name of our club shows up on a list one day a year?" Hopefully, they will know the right answer.

DT 

[Tom_Doak]

Brian:

A course in the 90's on any of the lists is just an eyelash from disappointment.  There is so little separating the courses that one person's vote is the difference between the course being in or out of the top 100 ... and of course it is completely a matter of opinion to start with.

If a club views its ranking as a referendum on its greenkeeper, that's the height of idiocy.  It's sick enough that green committees think "we are falling behind ... we ought to make a change to the course."  If you are one of the top 100 courses, or even close to it, you probably are better off leaving well enough alone.

P.S.  I suspect you won't want to sell copies of the updated Confidential Guide in the pro shop there.

[Mark Bourgeois]

Brian,

It looks like a bunch of members are fooled by randomness.

I just went back and looked at the scores for the 2011 rankings showing Hudson National ranked 83rd, with a previous ranking not of 90th but of 94th. HN's 2011 score was 61.43; in that same ranking / cycle, for HN to have been ranked 94th it would have had to score 61.07. (Actually that would have given it 93rd due to ties.)

The difference between 61.07 and 61.43 is 0.6 percent (six-tenths of 1 percent). To put it another way, to raise the score 0.36 points (61.43 less 61.07) using 2011 criteria means an improvement of the average score in Shot Values of 0.09 points and in each other category of 0.045 points.

I would be interested to hear someone explain what it means for one course to be superior by 0.045 points to another in, say, Design Variety. For example, Course X has 0.045 points better (more?) Design Variety than Course Y for the following reasons....

What does an improvement of 0.045 points mean at the level of individual ratings? Well, there are 1,000 panelists; suppose 100 scores are used (and all count -- none are thrown out as outliers). To keep this relatively simple, suppose one half give it a score corresponding to the 93rd-ranked course. How much higher do the other half need to score the course to get it to 83rd? 0.09 points higher in each category except for the double-weighted Shot Values, which require 0.18 points higher.

I invite anyone to explain a 0.09 point difference -- nine-hundredths of one point -- in, say, Memorability.

[Brian Potash]

I think Nigel brings up an interesting point in that what if I don't think we are a top 50/75/100 course?  The more I learn about GCA, and the more courses I play, the less spots available on my own personal list.  My job is to continue to do everything possible to empower Ryan to do everything he can to make our course as good as it can be.  At that point I am satisfied regardless of an arbitrary number.  I hope the majority of our members will feel the same.

[Richard Choi]

P.S.  I suspect you won't want to sell copies of the updated Confidential Guide in the pro shop there.

I believe this belongs in the GCA Hall of Fame as one of the best quotes ever.

My apologies Brian....

[Dan Kelly]

I suggest that you change the thread title to "Yet Another Unfortunate Side of Course Rankings."

It's absolutely ridiculous, and just plain vanity, that anyone anywhere should care if the course he calls Home is "rated" 17th or 54th or 90th or 150th or (like my home course, which I love) not rated at all, so long as the course delivers a satisfying experience.

I just tell people my course is tremendous fun -- an invisible hidden gem, so far as all of the rankings are concerned.

I'm wondering: What are the *fortunate* sides of course rankings?

[C. Squier]

If your current super really is as good as you say, then getting fired for a drop in rankings may be the best thing that could ever happen to him.....b/c he'll then be hired (quickly) by a club that will appreciate him for the work he does.  I would hate to have my job depend on the opinions of non-members and not the tangible evidence on the course.

[Mike Nuzzo]

How do you know if any other super in the world would have lost those few greens, not saved them like he did?
Did your budgets change from your previous super to your current one?
Do you say yes to your new super more?
I do not know anything about either of your supers, but I am with Tom D. regarding the club viewing the ranking as a referendum on them.
Maybe if you made home made cheese puffs and great milk shakes your ranking would go up 12 spots.
Cheers
Mike

[Brian Potash]

If your current super really is as good as you say, then getting fired for a drop in rankings may be the best thing that could ever happen to him.....b/c he'll then be hired (quickly) by a club that will appreciate him for the work he does.  I would hate to have my job depend on the opinions of non-members and not the tangible evidence on the course.

I didn't mean to suggest that our current Super would be fired for a drop in the rankings.  That would be insane.  But there is truth that a rise is the rankings was a relevant factor in retaining an old super.  I think that is slightly less insane.

My overall point was just how unfortunate it would be to judge the performance of any super using the rankings as a significant factor.

[Michael George]

Brian:

Honestly, I would try and identify the influential members that value rankings so much and take them to courses that are not within Golf Digest's top 100, but are true gems in the area (Myopia Hunt, Essex and Fairfield certainly come to mind).  Really no other way to handle it other than to change opinion within the club.....and no way to change opinion without having people see it with their own eyes.   Unfortunately, guys like us can talk until we are blue in the face, but too many club members "know everything" and need to see it to believe it.

Now, if they get a copy of the Updated Confidential Guide and are upset with Tom's opinion of the course, you can always just show them the list of Doak courses that Hudson National is still ranked over (Tom - I honestly love your work, but you had that one teed up pretty nicely).  

But as I have regularly said on this site, the golf course is only part of a club...the most important part.... but the make-up of the membership, the food, the service, the mission and other factors certainly make a great club and no magazine ranks the club as a whole.
 

[Paul_Turner]

Mark B

The whole two decimal point accuracy is dumb because the rankings have no idea of the error in the score.

Spot check a rater and ask his/her scores for a course.  Unless he/she rated the course last week,  or can refer to notes, then I reckon they give a different score each time.

[Peter Pallotta]

Brian - thanks much for your good and interesting and honest post. We've discussed (and I've posted on) countless 'ranking' threads around here, but I don't remember ever seeing such a specific and pointed example of exactly how the rubber meets the road re those rankings. It's a world (private clubs) and a role (Green Chairman) that I know next to nothing about, and so it is good to learn about the potential impact on real-life people and real-life courses of our idiotic but seemingly endless fascination with 'lists'.  (Quick - name me the top 10 Par 3s you've played that come after short Par 4s of under 290 yards but immediately before long Par 5s of over 580 yards on golf courses that lie east of the Mississippi but north of the Mason-Dixon line designed between 1930 and 1958 by one-armed architects with drinking problems and degrees in microbiology).

Peter

[Joel_Stewart]

P.S.  I suspect you won't want to sell copies of the updated Confidential Guide in the pro shop there.

I believe this belongs in the GCA Hall of Fame as one of the best quotes ever.

+1.  I laughed out loud when I read that.

Regarding your super, there is really only 1 category that he should be concerned with and that is conditioning.  Hudson Nationals score was 7.8628 which is middle of the road.  ANGC was ranked first at 8.9.

It's all fodder though.  I know a course that has dropped almost every year for the last 25 years and the superintendent was recently promoted.  Another course in Northern California had a great superintendent who did some amazing things but he was rough around the edges and rubbed people the wrong way.  He was fired and found a new job very quickly.

[Joey Chase]

Brian,

Your club is truly lucky to have Ryan as it's Superintendent.  He is an amazingly hard worker.  Right when he was brought in at Metedeconk, he made positive changes and never stopped during his tenure there.  Congratulations and good luck!

[Bill Brightly]

Brian,

While Tom does not say exactly why he feels HN should not be on a top 100 list, I bet it has nothing to do with the conditioning of the course. It is a beautiful course on a pretty dramatice site, but it is a HARD golf course. For me, it is the razor cur bunker edges that create an occasional unplayable situation that I find to be a bit harsh. But that has nothing to do with your super.

[Mark Bourgeois]

Good morning Joel and thanks for playing. To Paul's point, let me ask two questions. Suppose the green committee chairman came to you and said the committee decided they were going to strive for a precise conditioning score of 8.1735. Not 8.1736 or 8.1734.  8.1735.  Now the questions:

1. What is the difference between their current score of 7.8628 and their target score of 8.1735?
2. What specifically do they need to do to gain an improvement of 0.3107?

I ask about precision because GD has decided that a precise interval of 0.0001 is meaningful (otherwise why do it) and therefore explainable.

[Mark Saltzman]

Mark, given what you know about the GD rating system and for us non-statisticians, what number of decimal places is supportable (or is the question what number are statistically significant)?

If any assumptions have to be made, feel free to make them.

Don't forget, show your work!

[Mark Bourgeois]

Mark, I would ask one question and make one point.

Question: for those out there who rate courses, what is the minimum "defensible" (explainable) difference in scores? For example, if you were rating the conditioning of a course could you provide a coherent and systemic explanation -- an explanation that would provide the information necessary for someone to change the score by the exact interval and no more -- of how one category score differs from the next interval score? For example, can you explain what exactly accounts for a score of 5 vs 6, such that a change to the course could be made that would lead to an exact improvement (or degradation) by an interval of 1? Of 5.1 vs 5.2? Of 6.73 vs 6.74? Etc.

Point: seems to me the point of carrying out scores to one-tenthousanth of a point is not scoring but a contrived method for creating spacing. In other words, if courses were scored to a level of precision that actually reflected meaningful differences, then there would be huge numbers of ties. You might see for example 30 courses tied at one score; I think that would more accurately reflect what many say on here, namely that the difference between a course ranked, I dunno, 63rd, and one ranked 89th is not meaningful.

But that wouldn't sell magazines. Also, using precision enables Golf Digest (and doesn't Golfweek do something similar?) to pretend like there's some sort of intellectual honesty to what they're doing. It's an old trick, using statistics that way.

[Andy Troeger]

Mark,
In the 2011 list they used two decimals and had quite a few ties, including for #100. I'd be surprised if the rationale for going to four decimals was anything other than breaking those ties. GolfWeek only uses two decimals, but given that they don't list ties even for courses with the same score, I would guess that they use the next decimal to break the tie.

Even on my personal list, I refuse to have ties. They bug me, for some odd reason. So I can understand why the magazines break them, even if the differences are not statistically relevant. I think anyone who has studied the methodologies realizes that the difference between #76 and #77 is negligible.

[Brad Isaacs]

It has always bothered me when the use of the word contrived is used in the same discussion as statistical significance. It eventually gets back to the same kind of discussion as pornography," you know it when you see it". 
As to rankings affecting people's jobs, human nature is not always kind. I don't think this is necessarily a solvable issue nor should it be. What would be more interesting to me is, what are the factors that cause a drift in course rankings over time? Is it Fazio drifting to Doke to now perhaps Hanse, also know as flavor of the month? Is it an effect of location? What is the impact of the economic conditions on the ratings and or Raters?