Dr. Klein's article on Magazine Rankings

[John_Conley]

Dr. Klein was a lightning rod on this board a few months ago when Golfweek released their "America's Best" lists.  Many hinted at corruption, flaws in the process, and tainted ballots.  Scuttlebutt died quickly as the truth was learned about the statistical methods used to present the best rankings possible; could it be that the truth isn't as interesting as allegations of conspiracy?

The most recent issue of Superintendent's News includes a great article from Brad on the can of worms each ranking opens.  Did anyone else see it?  If so, what are your thoughts?  Others: you may wish to look for it on the Golfweek website as it addresses many of the questions asked here on this board about the ranking.

Golf Digest will release their Top 100 soon and the same backlash may follow.  Before anyone sends those e-mails or calls that magazine, they ought to read Brad's article to get some perspective on the other side.

[THuckaby2]

Would love to read that article - any way to post a link to it?  Does it exist on-line?

TH

ps - sending you instant message re other matters...

[Paul Richards]

Tom and John:

I just checked the Golfweek website and it appears that this article hasn't been posted there yet.

[THuckaby2]

Darn.  Thanks, Paul.  Superintendent News isn't a magazine I see very often, unfortunately. I would love to see this article some way... the whole ratings this is so overblown, any help explaining it and downplaying it would be a good thing.

TH

[Matt_Ward]

Tom H:

There is no way to "downplay" ratings. The whole purpose of ratings is to generate "buzz" and magazines are in the business of making sure they have "buzz." The question becomes how the info is gathered who does the assessment and by what criteria. I would submit that some ratings are better than others. Which ones? I guess for some people where they stand depends upon where they sit.

[THuckaby2]

It's just wishful thinking, Matt.  In my perfect world these rankings would just be for conversation only and wouldn't make a whit of difference in the real world.  Unfortunately, they do, both for courses and for magazines.

As for which is better, well... we all have our preferences and given they all do measure different things really, it's kinda apples and oranges so I'd just say each is fine in its own way.

TH

[Paul Richards]

Tom:

Sort of like Gary Player's quote, "the best course of its kind that I have seen"?

[John_Conley]

The question becomes how the info is gathered who does the assessment and by what criteria. I would submit that some ratings are better than others.

Matt:

Unfortunately, not enough people are getting the chance to see the article.  The gist of it is that "you can't please many of the people" when you do something like publish a ranking.  To him, it seemed like everyone felt slighted.

Your comments about process are valid.  Anyone with a suggestion for a way to improve any of the rankings will be heard by me.  (Not like I can change anything, but I'm always curious as to the "best way" to do something like this.)  Too often, such a "suggestion" is no more than "make your list look like my opinion and it will be better"-type talk.

I felt that a number of things could be improved with the Golfweek "America's Best" lists as I learned the process.  To the credit of Brad, Jonathan, and others, continual gradual refinements have improved the statistical quality.

With the stock market, money managers separate "systematic" and "unsystematic" risk - the kind you can get rid of and the kind you can never escape.  In my eyes, the "America's Best" list has taken into account what can be taken into account and is subject to the nothing more than the same whims such a process will always be subject to.

[THuckaby2]

Paul - well, Gary seems to be insincere in that, doesn't he?  I honestly mean what I say.  When one looks at the criteria and methods used by each magazine doing ratings and rankings, they are each distinct and different and to me, measure different things.  Each does a fine job of achieving its own ends.

We can argue till the cows come home about which does the best job of ranking the "best" golf courses, because that can be defined in so many different ways....

So it's all just happy talk to me, except for what to me is sorta sad:  the fact that people make or lose money, jobs, prestige, whatever based on what these whimsical magazine rankings say.

TH

[Matt_Ward]

John C:

Fair point. Let me dwell on a few particulars and I'll post ASAP.

My only concern regarding Brad was his intertwining the aspect of Rustic Canyon's clubhouse and driving range when discussing the OVERALL IMPACT of the golf course in his review. I would hope that someone of such distinguished expertise would brush aside anything that is a sideshow and concentrate on the main act -- to wit -- the golf course!

P.S. I'm also perplexed that Rustic Canyon didn't even crack the top 100 modern as well as a few others that I mentioned on another thread dealing with the results from GW.

John, I coordinate the bi-ennial ratings for Jersey Golfer and trust me there are no happy campers sending us letters. Why? Only those that got pushed out or down have a beef. The rest say little or nothing. Kudos to Brad and the GW panel and I think the invitation to make suggestions is a good one and I'll forward a few thoughts ASAP.

I look forward to reading Brad's piece in Superintendent's News.

[Mark_Fine]

The main thing that bothers me on any of these rankings is when a course makes a "big" move, up or down.  There needs to be some really good reason for that change.  

[Brad Klein]

I'll arrange to have the column posted today on Golfweek.com site and will manage (somehow) to link it to GCA.

[THuckaby2]

That would be great, Brad - many thanks for the effort.

TH

[John_Conley]

The main thing that bothers me on any of these rankings is when a course makes a "big" move, up or down.  There needs to be some really good reason for that change.  

Mark:

I don't know of any big moves.  Can you give an example?

[Mark_Fine]

John,
Didn't someone say Hollywood fall off the list from #57 or something.  I don't have the list in front of me.  

You have to admit, that is drastic, especially on the Classic List which in theory should not have that much movement (at least not backwards).  If a course is discovered or if one has had a lot of restoration work done to it, you could see one moving up and/or on to the list.  But how does one go backwards so fast?  

This is serious stuff as you know.  Heck I'll go play a course that dropped from #78 to #79 or something on GD's list and I'll sometimes get comments that imply we just completely wrote them off as any good.  

Mark

[JohnV]

Mark, for those who where there at the ratings outing last week where Bradley showed the chart mapping the numbers, it is fairly obvious how courses, even on the classical list can drop from the bottom half off the list.  There is very little difference in the numbers between 50 and 100.  It is sort of like the World Rankings for golfers.  Tiger is further ahead of Ernie than he is of the 140th player on the list.  A player can make a huge move by winning one event.  Similarly, a course could make a pretty big move by having a bunch of raters visit it and either love it or not love it.

Take Golf Digest's rankings as an example since they print the points.  The difference between 1 and 50 is 12 points while the difference between 51 and 100 is only 3.  It doesn't take a huge surge of votes to move a course a fairly long way.  Since the Golfweek scale is 1 to 10 in .5 increments, the change can happen fairly quickly to a course that barely had enough raters to qualify and then got a bunch more who felt differently about it.   I am frankly surprised that there is less movement in the ratings.

[John_Conley]

Mark:

I don't consider a move of 44 spots from #57 Classic to #101 a big move at all.  Drastic?  That's a word with a pretty strong connotation.  Surely there must be something diabolical at work to move a course so far!

Hollywood is not as renowned as Winged Foot or Baltusrol, so it is likely that fewer have seen it.  John V does a great job of answering the question - Hollywood doesn't even make the Golf Digest Top 100 (where C&M are lumped together), so you can't even see it drop because it never gets reported.

Any list is hardly gospel; it is a ranking of people's opinions at a certain point in time.  Alabama was #1 this year in basketball and failed to make the NCAA Tournament.  Being #1 doesn't mean you'd beat the teams below you every time just like being #57 doesn't mean every person would agree it is better than #105.  (#1 vs. #45 is a different story - it is unlikely anyone would prefer course #45 over the #1 ranked layout.)

The biggest movement in terms of number of places happens with courses that have a smaller sample size.  Any list runs into this problem.  As a result, GOLF Magazine is probably a little slower to uncover new entrants (fewer panelists, plus they are less likely to run around to see new courses), Golf Digest makes a conscious decision to wait 2 years, and Golfweek requires 10 (I think it was 5) visits before being eligible.

A move of 44 spots from #3 to #47 would be titanic.

I have seen significant movement on the Modern list over time as a course may gradually fall as new courses are built.  Much of the new construction has been high budget and targeted at the top end, so what was a Top 100 course 6 years ago may only be a Top 160 course today - assuming 10 courses per year open that are worthy of such an accolade.

The pre-1960 list should have a high degree of statistical integrity since the universe hasn't changed for 40 years.  Once a critical number of panelists have seen the courses, subsequent visits won't impact things much.  Of course, you could always have the next Hollywood that has only had the minimum number of visits because it is only now being appreciated.

Matt Ward:

I think Brad has only one vote, the same as me or anyone else on the panel.  Read any Whitten review and you'll see the same type of commentary.  Champions Gate and Mystic Dunes are two that come to mind.

If you haven't already, read Brad's article when it becomes available.  My intent was to find out why no one comes back to say, "That makes sense," when a response came from Brad, yet he was a whipping boy for many when the lists came out.  Turns out no one has seen the article.  Read it when you can and you'll see the other side of the equation.

[Brad Klein]

It's now posted on Golfweek.com, as mentioned in another, newly opened thread. URL is as follows:

http://www.golfweek.com/articles/2003/superintendent/columns/32271.asp

[Scott_Burroughs]

Brad,

The guy representing George Wright Muni would be more disappointed with the state public list if Taconic were included as it should be.  Taconic is not private as it is indicated on the Classic list.

[Mark_Fine]

Brad,
Nice article (I just read it).  Every thing you state is very true!  I've always said, these lists (right or wrong) are taken very seriously, too seriously.  At the same time, that has to be understood because that is the way it will be.

John/John,
Thanks for the comments.  Believe me I know the sensitivities of these lists and have been living it for many years with GD.  I always tell courses on the lists, just be happy you made it (regardless of your number).  But as you know, it's still not that easy.  It's even tougher when a course has fallen off and you want to go back there and re-evaluate it.  

I do, however, disagree with part of what you said in that I feel it should be harder for a course to go backwards than forwards.  It should be much harder for a course ranked #57 to fall off the list than for a new one to show up in that position.  Most of the big moves on GW's list had a reason for moving such as Fenway, Aronimink, etc.  Those courses did work to polish up their property.  I don't recall reading one explaination that a course jumped up 30 spots because more panelists came to see it.  Did you?  

But when a course goes more than 43 spots backwards (remember no one knows if Hollywood is now at #101 or #300), that is much harder to explain.  I do realize that a course with only a few votes can move significantly, but a change like Hollywood tells me there is wide variation in the opinions of the panelists voting on it.  

I really don't want to drag this out and I'm really not picking on GW as all the lists have to deal with this.  Furthermore, I have never played Hollywood so I couldn't tell you where it belongs.  I'm just pointing out (as Brad did in his article) that these lists are taken very seriously and I would have trouble explaining to the members in a case like that, what happened.  

Mark

[Brad Klein]

re: Taconic. I called them six years running and asked very clearly each time if they are daily-fee or private membership and how they wanted to be listed, with the implications for each. They kept telling me "private" and I have never received a complaint or concern from them for the listing as we have it.

[A_Clay_Man]

Dr. Brad- As long as you aren't the guy who's gonna get hurt, you must think these serious calls are more than a little humerous? I find it funny that they are asking you, why.

Is it just me, or is there a golf lesson when natural chaos from controversy causes volatilty.
 
Keep up the good work

[Brad Klein]

I feel bad for these people, and I tried to convey that in the column. Darn, this is a game, a very fine game, one we all love. But some of these people (by no means not all or even most of them) are like high school or college football coaches desparate to motivate themselves and their troops and to improve. I try to help, I think we all do, and the game benefits marginally from this effort. But when there are so many truly important things out there - a war, misery, starvation, disease - it seems so weird that this is what gets our focus.

[Jim_Kennedy]

Scott,
Taconic is semi-private and therefore not strictly public. I don't think a Taconic member can enter a Public Links championship.

John Conley,
I think ratings could be improved in how they are presented. Just give us the numbers and forget about totaling them up and telling who is #1. If course A gets more "tradition" points than course B but fewer "conditioning" points than course B and course C gets gets more "shot value" points then either A or B just show us that!  Let the public debate who's number whatever on the list of 100 courses provided to them. This way no one gets their noses out of joint and there would be 10 times more locker room conversations which would add to the buzz and probably add to the popularity of rankings, especially among people like myself who don't really care one way or the other if ratings are published.    

[Mark_Fine]

Jim,
I could be wrong but I'm not sure GD would publish all their numbers if they could do it all over again.  All they did by trying to be open about their process was set themselves up for complaints.  
Mark