The rating game, a personal story...

[Jim Thompson]

I jut got my copy of Golfweek and noticed the Top 50 new listing.  We didn't qualify due to the number visits we had from Golfweek raters at the time the list was compiled, as such we are not listed.  Brad was kind enough to let me know that we have now had enough visits to have a "real" number.  That number being a rating of 6.0.  In the GolfDigest rating game we were number six new affordable, behind: 1) Bully Pulpit, 2) Arrowhead, 3) Eagle Ridge, 4) The Shoals, 5) Highland Meadows.  Bully Pulpit has a Golfweek rating of 5.85 and none of the others are even listed in the Golfweek ratings.  We would have come in at #34.

Since so many raters come here, what is soooo different between the standards / criteria to cause this? What are the major means by which raters are contacted to see certain courses over others?  Did I miss something?  Was I supposed to pay the Golfweek rater pimp to get traffic? (That’s more of a joke, by the way, but we were approached by many people saying thy would “get us traffic.”)

I'm really just trying to understand how this stuff works.

Thanks!

JT

[Adam Clayman]

Jim, I don't believe I can help you figure this out.

I would however recommend that if anyone of the magazines raters abuses their privledge, or boast of their sphere of influence, you should consider it a public service by informing the magazine of their actions.

The only other thing you can do is to confirm that someone is in fact on the current roster of raters for that magazine.
This implies that they act accordingly and give you ample notice of their arrival, so that you can have time to check them out. Email Brad for Golfweek and he can quickly confirm. Or in the least, give you another email addy to direct the question to.

Sorry, it isn't directly related to your query, but having a good soup might help too.

[Ulrich Mayring]

I cannot take ratings seriously, where the rater announces before-hand that he will come. Any serious rating outfit (like the Guide Michelin) goes to considerable lengths to ensure and protect the anonymity of their raters.

Ulrich

[John Kavanaugh]

Jim,

Do you believe your course is one of the top 100 courses built since 1960.  I think that you will find with the Golfweek crowd that it is very dangerous to increase expectations by naming holes after classical features when you did not use an architect well known on this site.  Digest guys don't intelectualize things to that point and also are not handcuffed by the possibillity of accidently pushing a modern course into the top 100 by giving it too high a rating.  I know you read this site everyday and would be surprised if you thought Angel's Crossing was getting the buzz of the likes of a Rustic Canyon which at one point may have been number 110 and then eventually broke into the ranks.  Remember that everytime a Golfweek guy gives a course a 7.0 he runs the risk of pushing off a course like Rustic...or probably soon to be Stone Eagle or The Rawls.

I think a 6.0 shows you and your course alot of respect....you should be proud but hardly surprised given the amout of time you spend on this site.  Would you have hired a different architect if you found GCA before you did and one of your primary goals was to reach top 100 status....That is really the key...not how you treat people or who sees your course.

[Mike_Cirba]

John,

Do you really believe your own sh*t?

[John Kavanaugh]

I nailed this one baby doll...If you don't believe me go read the My Home Course write up on Angels Crossing and tell me you are surprised it is a 6.0.

[Mike_Young]

Since so many raters come here, what is soooo different between the standards / criteria to cause this? What are the major means by which raters are contacted to see certain courses over others?  Did I miss something?  Was I supposed to pay the Golfweek rater pimp to get traffic? (That’s more of a joke, by the way, but we were approached by many people saying thy would “get us traffic.”)

I'm really just trying to understand how this stuff works.

Thanks!

JT

Jim,
Fact is there are several guys that make a living promoting courses for the rankings and bringing in writers and raters.....
and there are architects that can sell a job by promising a rating......
I don't like it but it is the way the world turns.....you need a developer that will pay the fees for the promoters to promote the course and bring in the people....they are very good at it....I can give you some names if you IM me.
Mike

[Brad Klein]

Jim, you send me a private email and I share the details with you and you air it? Nice going.

[John Kavanaugh]

Jim, you send me a private email and I share the details with you and you air it? Nice going.

Brad,

In the last few months I have really come to respect how difficult this part of your job must be.  I wish I had a solution that could make more people happy without creating more lists.

[Jim Thompson]

Wow, I guess this is a magic button topic.  First of let me say that I AM NOT UPSET WITH THE PROCESS OR THE PEOPLE.  I AM NOT WHINING ABOUT NOT HAVING SOMETHING TO ADD TO MY MARKETING BAG OF TRICKS.  I DO NOT HAVE A PROBLEM WITH EITHER PUBLICATION.  I know that human nature makes us all want to think the worst of people at times, but I really don't work that way. I just don't get how ANY course can be held to such and such a caliber by one group and a totally different level by another.  Between both groups, these are the people that are supposed to get it and appreciate it.  I was hoping someone could help me understand that.  I was expecting a stock answer from someone that one publication is more interested in the grass than the dirt or one placed more values on the views then the golf corridor.  Just trying to figure it out how this works for, and here it comes, the next one I get to be involved with.  It’s just that simple, no secret agenda, no sneaky plan, no nothin’, just building the best course possible on a given piece of land.

Now for answers by poster:

Brad,

I do not, will not, nor ever mean to disrespect or embarrass anyone.  If I have caused you any of those pains, please accept my apology as it was never my intent.  I would even go so far as to say that anyone who questions your process or whines about ought to get a nice hot cup of shut the hell up.  In fact, I admire what you do to a pretty high level and if I didn’t I’d tell you.  Just ask the guys that have met me.  This is a good question though and appositive for your publication.  What makes you unique to your competitors in the golf publication industry?  This is a story you should be telling every chance you get, but I have no business telling you how to run your business.  Nick was at our media day back in 2002 and a very nice guy.  I thought that was enough contact and that that was how your program worked.  I may have been wrong.

John,

I do believe Angels Crossing is one of the best 100 courses built since 1960 and I’ve been told as much by members of this board and others.  I don’t think the architect’s last name ought to have anything to do with it either.  Don’t forget my architect has one of the best pedigrees in the business, should he be held in less regard because he doesn’t spend a bunch of time self promoting?  As for buzz, I think we don’t charge enough to meet the level of snob appeal needed to get any respect.  I would bet a number of courses in the Midwest fall under the same problem.  Our markets just won’t pay what guys do out on the coasts, it doesn’t mean our product is of lesser value though.

I’m not disappointed with a six either, I just don’t know what the hell it means.  What criteria did we do well in, where are we weak.  I got nine more holes to build here and I’d hate to make the same mistakes again if possible.  We chose Bruce Matthews III because he was the best fit for what we wanted to do and I would recommend him to anyone in a heartbeat.  He is special in that he is adaptive in his style and takes a true minimal and respectful approach to design.  As for naming holes after classics, we really did do just what we said.  We tried to share our influences with the public and educate through our design and product.  Where possible we even tried to make great concepts better.  I almost feel like we made the mistake of being like the magician who told how the trick is done or where the ideas came from rather than trying to tell the world we’re geniuses.  We are just honest people trying to provide an honest design for an honest price.

Again, I am not upset or looking to serve myself.  I’m just trying to understand the differences for my own minds sake.  I have never been anything other than honest out here and really think I ought to get the benefit of the doubt.  This is the one place where you can actually ask this type question and get real answer not just a bunch of speculation.

Cheers!

JT

[Jim Thompson]

I just went back and read my correspondence with Brad and can certainly understand why he would consider me to be a schmuck right now.  Brad, if you would like, I would be happy to have this thread taken down and rephrase or contextualize the question at hand.  I’m just too honest to talk about things in innuendo and NOT clever enough to do it without people figuring it out.  Again, mea culpa.  Send me an IM and let me know if I need to contact Ran or Tommy or whatever.

[Brad Klein]

Jim, I don't think you are a shmuck and I'm not pained or embarrassed and Angels Crossing certainly isn't the issue. All I meant is that I spend a lot of time with behind-the-scenes issues providing info and details, and that stuff should remain background.

There's nothing secretive about our process, however. Anyone -- any owner, any reader -- who wants to find out more is sent one of our rater's handbooks via PdF file. And as soon as I can arrange it I'll just have it posted on our Golfweek.com Website for even easier access.

I can't help wondering if you've also contacted Golf Digest to see their response to these issues. I hope they were as helpful to you as I tried to be.

[Mike_Cirba]

David Kidd
Lester George
Robert Trent Jones Sr.
Graham Marsh
Mike DeVries
Steve Smyers
Rick Smith
Red Lawrence
Dan Proctor & Dan Axland
Hurdzan & Fry
Weiskopf & Morrish
Jim Engh
George Fazio
Ken Dye
Mike Strantz
Bobby Weed
Gil Hanse
Keith Foster
David Esler
Bob Cupp & John Fought
Rees Jones
Baxter Spann
Stephen Kay
Arnold Palmer & Ed Seay
Jack Nicklaus
Phil Mickelson & Gary Stephenson
Tom Watson
Gary Player & Tom Walker
Dick Bailey
Robert Von Hagge & Bruce Devlin
Todd Eckenrode & Gary Baird


Q. What do all of these men have in common?

[Adam Clayman]

Normally, it would be the Cliff Claven pat answer. "Never been in my kitchen" but in this case it isn't true.

 

[David Wigler]

Jim,

I have been lurking on this thread, so i guess I will way in.  The two publications frequently end up with very different lists.  No where was this more apparent than in the last rankings where GW had Kingsley in the top 50 in the US and GD (Mainly because of Huckaby  ) had Kinglsey outside the top 20 IN MICHIGAN.

GD has a lot more raters and I think it is fair (I hope they would agree) that as a broad generalization, they are better golfers but not as well trained on architecture.

One of the benefits of a big panel is that it makes it much easier for a course to reach a point where it numbers are statistically relevant.  One of the drawbacks is that it is harder to evaluate raters.

One of the benefits (Some have certainly argued drawbacks) of a smaller panel is that it is much easier to see what raters are doing.  One of the drawbacks is that, since there are fewer raters, courses that are off the beaten path (And you must admit that Angels Crossing qualifies as such) have a hard time attracting enough raters to reach a point of statistical relevance.

Here is a simple numerical example of what I mean.  Lets say Desmond Howard decides to build a golf course and I owe him big time for what he did to Notre Dame.  3 other raters come out and give it a 5 (Good golf course, somewhere between 200 - 500 in the country).  I give Heisman Acres a 10.  Now its average ranking is 6.25 and on any given year it might end up making the top 100.  Conversely if 30 golfers give it a 5, my ten becomes statistically irrelevant in artificially moving it up or down.

To Brad's credit, he is very good about telling GW raters which courses need more raters so they can reach that point of statistical relevance and I believe this system to be inherently fair.  The side affect is that you might see several raters who care about statistical relevance and want to help out do an utterly stupid trip that involves 800 miles of driving without every leaving Michigan, so that True North, Angels Crossing and The Grande can get two more reviews  .

JakaB and I have debated this for years and I know he will never agree with me but I cannot fathom any of the raters I have met giving a course a deduction because it was built by Architect x instead of Architect y.  I definitely cannot imagine someone saying "I must give this course an x because I do not want Rustic Canyon to drop out."  I can see someone saying "I gave Rustic Canyon a 6.5, so course x must be a 6 because I believe RC was clearly a little better."  In fact, that is exactly what a rater is supposed to do.

Lastly, if architects do come into play it would only be in the context that certain styles fit raters eyes and because of that, they would tend to rate styles the believe to be superior higher.  

Anyone who has read my 1,700ish post (And I really hope that is no one since 90%+ are useless drivel) would know that crisp line and bold features fit my eye.  I love the design style of Raynor.  When I see those similarities in a modern course (See Black Creek, Kingsley or Angels Crossing) it would be second nature that I would be predisposed to think more highly of it.  I agree that you have nothing to apologize for in the work Bruce and Yourself did and it is unique and praiseworthy.  

IMO your rating will come with time and traffic.  I wonder how many raters (Or out of state golfers for that matter) realize that you are less than 30 minutes from top 100's Lost Dunes, and Pointe O' Woods.  

I hope this long answer makes sense and gives you some clarity.  Now go kick USC's butt.

Dave

[Mike_Cirba]

JakaB and I have debated this for years and I know he will never agree with me but I cannot fathom any of the raters I have met giving a course a deduction because it was built by Architect x instead of Architect y.  I definitely cannot imagine someone saying "I must give this course an x because I do not want Rustic Canyon to drop out."  I can see someone saying "I gave Rustic Canyon a 6.5, so course x must be a 6 because I believe RC was clearly a little better."  In fact, that is exactly what a rater is supposed to do.

The answer to my question above listing all of the disparate and many little known architects and what they have in common is that they all have at least one golf course on the Golfweek Top 100 Modern listing.   I did not include a few others who had MANY courses on the listing, because I want to prove a point.

John Kavanaugh continues his Golfweek Rating and Rater bashing and his big Red Herring is that the Golfweek ratings are biased towards certain architects who are favored on this board.   It is obvious to me that John has a personal axe to grind, because along with the list of wide ranging architects and styles listed above, you have the following evidence;

Tom Doak - In John's words, we're all Doak Butt Boys.   How many courses does Mr. Doak have in the Top 100?  - 2

Gil Hanse - Another favorite here...how many?   - 2

C & C - Hey wait, maybe Kavanaugh's onto something...this talented due have 7 courses listed in the Top 100.

but....let's look further.

Jack Nicklaus - hardly a universal favorite here, and at times widely criticized.   Well, Jack has 8 courses in the Top 100.

Pete Dye - The father of the modern golf renaissance, even with his sometimes very penal style has 12.

Tom Fazio - The architect that John Kavanaugh complains we don't recognize, who we're all biased against, leads the ratings with an overall 17 courses listed.

John, you're full of it.

[George Pazin]

Well said, Mike, and nicely illustrated with ACTUAL FACTS.

You get a gold star for the day.

[Mike_Cirba]

Well said, Mike, and nicely illustrated with ACTUAL FACTS.

You get a gold star for the day.

Thanks George.

No rankings are perfect, but I'm just sick and tired of John Kavanaugh throwing around completely baseless charges, slanted innuendo, and insulting lies.

I can understand someone who says they don't believe in course ratings for whatever reason, but for John, this is obviously personal and consistently directed at Golfweek specifically for reasons I can only guess at.

[RJ_Daley]

Mike, on the other thread Tiger says that not every post needs or ought to be a well documented research article.  

So, why bring all these pesky facts into our little chat about these ratings bias's?

Well done.

You too, Wiggles!

[Wyatt Halliday]

"John Kavanaugh continues his Golfweek Rating and Rater bashing and his big Red Herring is that the Golfweek ratings are biased towards certain architects who are favored on this board."


Can someone clue me in on this. I am wondering if this is a classic case of the chicken/egg theory. Are the Golfweek rankings a favorite of this board because of the architects listed, or does this board have serious influence to just the Golfweek rankings in general?

[Jim Thompson]

Brad,

At the conclusion of the GolfDigest rating period Ron sent me a sheet that showed our course ratings vs. the mean ratings for the top ten in our group so that I tell where we matched up by category.  I requested the reviews of a couple raters for my own benefit and have recieved no response.  Without going to much into the comparison of groups of raters vs each other, I can say that the GolfWeekers we've had out have been very considerate and low key, except for one obnoxious Michigan fan .  The GD raters were more of a mixed bag, it was obviuos that a couple were here just to take advantage of their status and to some levels demanded some extra TLC, if you know what I mean.  That said, I also thought that a couple of GD raters did first class jobs and really were here for the golf rating process and I respect them very much.  They are the folks whose reviews I have requested, John Percival's in particular.  I'll finish this up later.

JT

[John Kavanaugh]

Well said, Mike, and nicely illustrated with ACTUAL FACTS.

You get a gold star for the day.

Thanks George.

No rankings are perfect, but I'm just sick and tired of John Kavanaugh throwing around completely baseless charges, slanted innuendo, and insulting lies.

I can understand someone who says they don't believe in course ratings for whatever reason, but for John, this is obviously personal and consistently directed at Golfweek specifically for reasons I can only guess at.

Mike,

I don't understand your attack against me...Where on this thread did I bash Golfweek.  Jim even agreed with my point that he would have been better off by not setting high expectations by naming his holes after classical features.  I said the Golf Digest guys don't intellectualize things like Golfweek guys...I thought that was a compliment.

Why not go to Jim's course or at least look at his write up and tell him why it is a 6.0....Just not quite a top 100 by your standards.  It obviously isn't a top 100 by Golf Digest standards either...That doesn't make it a bad course.

I love that you think hiring a famous architect has zero bearing on the rating of a course....btw..do you tie a sweater around your neck often.

[Mike_Cirba]

John,

Don't play all innocent on me now.   I know exactly what you were inferring, which is that if Golfweek moved something like Angels Crossing up into the Top 100, we'd have to make room for it by replacing something by Doak or Hanse.

That's crap, John, and you know it, and now, so does everyone else here.

I very much would like to go to Jim's course and hope to get there.   From pictures, it looks well done and intriguing, and certainly worth a visit.  

However, I wouldn't rate it unless I actually played it, and I'm not going to provide any feedback to Jim in terms of a rating number unless I actually played it for myself.  All I would suggest to Jim is that Brad provides us with a list of courses that are actively seeking raters, and also a list of courses that need additional panelists to provide greater scientific statistical validity (re: more panelist ratings), and insists we visit them.  

And John...if you think hiring a famous architect gives a greater chance of a Top 100 course, you'd be wrong there.   This year, two of the courses I played were by probably the most famous golfer of all time and probably the most prolific current architect.   Although both were very good courses, neither is THAT good.   On the other hand, I played a course by an architect who has a handful of original designs, a name that wouldn't get him arrested in a store full of criminals, and his course was sterling and I gave it a score that put it in Top 100 running because it was that good, original, creative, challenging, and fun.

[John Kavanaugh]

On the other hand, I played a course by an architect who has a handful of original designs, a name that wouldn't get him arrested in a store full of criminals, and his course was sterling and I gave it a score that put it in Top 100 running because it was that good, original, creative, challenging, and fun.

Good for you, tell me honestly...what are the odds of this course actually making the top 100.  It clearly was a sympathy vote in some pathetic attempt to make a statement...like the guy who votes for a write in candidate for President.  How long did it take you to let the architect know you were in his camp.  I'm making a change mannnnn..

Where are your threads about the course...where is what we might call the Rustic push..

[Mike_Young]

Rating seems to be a necessity for many but in reality.....
Many very good golf courses don't give it a second thought and worry only about the 500 or so customers that give them 35000 or so rounds per year.  These guys would have to spend money to market in order to cater to rating and the green fee would go up.....And in the end they know when some one is RUI and they will not pay for it.