Course Raters- status quo or upset the apple cart

[John Kavanaugh]

Mrs Redanman is a rater because he is far too cheap to pay for her to play. Even Dr. Klein hates to see a wife wait in the parking lot. It doesn't hurt that she is a red head.

[Tom_Doak]

Tom,

I don't know how much exposure you've had to the corporate or academic research worlds but the perfectly valid and important tactic of using "focus groups" or similar qualitative research panels can suffer from the same problem. If you're not real careful, as the person setting up the panels you'll be tempted to reuse (and reuse and reuse) the people from whom you've elicited "good input" on previous projects.

Of course what's really happening is that certain panelists are getting better and better at finding what the researcher wants to hear and the researcher is getting more and more used to hearing it. The endpoint can easily be a minimal effort path for gaining very high quality feedback that appears to occasionally "challenge" the status quo but in the most important elements actually reinforces it.

Occupational hazard in that kind of work. No different really than the principle which says "If you don't want to go under the knife, then don't go asking a surgeon for his opinion". Or golf course designs, renovations, restorations, etc. etc. We're all only human and it takes a superhuman effort not to Get What We Want even if we don't want to.

Brent:

I was never really "in charge" of the top 100 for GOLF Magazine [I answered to their editor], but I did field most of the letters we would get from wannabe panelists, and had some input in recommending people for the panel.  When someone would write in to volunteer their services, I would send them a ballot and ask them to check which courses they had played -- but NOT to grade them.  I didn't want to know how they'd vote; I wanted to find out which of the guys were really interested in going to see out of the way courses [which all of the magazines need help with], versus which of them were belt-notchers trying to check off the courses that were already in the list.

Everyone who participates in these things wants to have an influence on the list.  I'd just rather listen to the guys who have found an out-of-the -way course they think should be included, instead of the guys who will threaten to cast off a famous course if it doesn't appeal to them.

[Adam Clayman]

 If anyone believes the GW panel is made up of similarly minded opinions on gca, you have never been to a rater retreat. This notion that JC is spewing forth, about one standard deviation and you're out, is bigger B.S. than what Kavanaugh thinks of GW.

If someone gives Pebble Beach a 4, that's an outlier. Same with someone giving Sandpines a 9.5. Clear outliers that without justification, either shows a lack of understanding on the subject, and/or, ulterior motives to either raise or lower a course for someones personal gain.
 
Throwing out a high and a low is a common statistical procedure, isn't it? We use to do that for our tournament handicaps, using 6 scores, but really only 4 counted. 

[John Kavanaugh]

Adam,

I agree that Golfweek covers the gamut of possible opinions. I am sure you agree with me that when Golfweek began using their raters as a form of profit for the magazine the quality of the ratings suffered. The more raters they let in the better their cash flow.  The only qualification that matters now is if a raters dues check clears. It is shameful.

What upsets me is when I see raters quit paying dues to golf clubs because they pay dues to a magazine. Golfweek is stealing what golf needs now more than ever.  Members.

[Mike Sweeney]

What upsets me is when I see raters quit paying dues to golf clubs because they pay dues to a magazine. Golfweek is stealing what golf needs now more than ever.  Members.

4200 private golf courses X 300 members (estimated average membership) = 1,260,000 membership slots in America.

When Brad gets the Golfweek membership to .01% of the private club slots in America give us a ring back. 

[John Kavanaugh]

What upsets me is when I see raters quit paying dues to golf clubs because they pay dues to a magazine. Golfweek is stealing what golf needs now more than ever.  Members.

4200 private golf courses X 300 members (estimated average membership) = 1,260,000 membership slots in America.

When Brad gets the Golfweek membership to .01% of the private club slots in America give us a ring back. 

Mike,

If Golfweek would strictly prohibit raters from using their status to either access or be comped at any course we would see at least 100 new memberships purchased throughout the country. That is 1,000,000 dollars into a dying industry. That is a dozen good people not losing their jobs. That matters.  Cultures have been built on 12 good people following one decent man with a dream.

[Adam Clayman]

John, I wouldn't begrudge anyone who can find a way to make money, legally. Is GW doing something illegal? Or are they just stopping to pick up the dimes, instead of tripping over a quarter?  Who are these angels that get to decide for someone else what is right? You? One of the finest minds in golf. How many people/jobs does GW and their publisher support?

Your dislike for GW is becoming too predictable. It has gone on for so long, I'm beginning to think you are really upset, because you were never asked to be on the panel.

[John Kavanaugh]

John, I wouldn't begrudge anyone who can find a way to make money, legally. Is GW doing something illegal? Or are they just stopping to pick up the dimes, instead of tripping over a quarter?  Who are these angels that get to decide for someone else what is right? You? One of the finest minds in golf. How many people/jobs does GW and their publisher support?

Your dislike for GW is becoming too predictable. It has gone on for so long, I'm beginning to think you are really upset, because you were never asked to be on the panel.

Adam,

I have never or would ever deny that every time I write a dues check that I am jealous of those who play the same courses for free. I made a choice to express how I truly feel about many things in this game knowing it would shut me out of the golden ticket. I have to live with that choice. I am especially angry this week as I book a trip to Vegas knowing that if I had just played along I could have saved me and my friends thousands of dollars in green fees.  It is all legal and all good, don't worry about me, I'll be fine as I pay the piper. Just don't tell my friends about what could have been if I had only played the game.

[John Kirk]

"People, people we are the same
No we're not the same
Cause we don't know the game"

Chuck D. - Public Enemy  "Fight The Power"

The cushy golf life is an exclusive world.  John K. knows how to play the game - what good is it when your best buddies can't play along?

[Joe_Tucholski]

Mr. Jones mentions in passing the key assumption underlying this whole ratings business. It is assumed that there is some underlying reality which the raters are trained to discover with only a limited amount of individual-rater variability and perhaps a certain amount of individual bias. All of which should "average out" if you set it up correctly, once you get enough ratings of each course. Sort of a Platonic view of golf course quality you might say.

Nothing wrong with that assumption if that's how you view the situation. But there could be another equally defensible assumption that each rater brings to bear a slightly different (or perhaps in some cases qualitatively different) "ideal" of what a golf course should be. In that case your training mostly serves to either dissuade the quirkier raters from honesty about their assessment or to week them out altogether from your rater pool. Discarding "outliers" is sort of last ditch backstop in case an occasional rater with a very different "ideal" happens to submit ratings reflecting that version of reality instead of the agreed-upon one.

I like the way Brent brought up the two views.

I think the status quo with regards the large magazine ratings systems is the way it should be.  I guess I follow the assumption that the raters are provided an objective methodology for most of the criteria and therefore if the rating is outside two standard deviations the rating should be thrown away (GD does have a top 50 courses for women and that covers the frequent outliers cited by Tom).  There are three GD rating criteria that I could see legitimate variability - memorability, aesthetics and ambience.  The magazine is responsible for the product that it produces and therefore I assume the magazine wants to utilize objective, specific measures as they define to obtain those goals.  If they just wanted what is your favorite course from their panelist they wouldn't provide guidance and they would just ask for a single score.  If a reader of the magazine is unhappy with the results of the lists they should utilize a different magazine ratings (or some blog/conglomerate of blogs rankings) that more closely aligns to their personal views or they should contact the magazines and ask them to change their rating criteria (didn't GD just change the conditioning criteria to focus on fast, firm and rolling).  

I remember reading here that a panelist shouldn't actually play the course like they normally would but instead should try to hit specific shots and primarily walk the course to observe not to play.

[Adam Clayman]

 I am especially angry this week as I book a trip to Vegas knowing that if I had just played along I could have saved me and my friends thousands of dollars in green fees. 

More BS.

If you were allowed into the inner sanctum, you would respect the fact that your buddies don't eat for free.

[JC Jones]

If anyone believes the GW panel is made up of similarly minded opinions on gca, you have never been to a rater retreat. This notion that JC is spewing forth, about one standard deviation and you're out, is bigger B.S. than what Kavanaugh thinks of GW.

If someone gives Pebble Beach a 4, that's an outlier. Same with someone giving Sandpines a 9.5. Clear outliers that without justification, either shows a lack of understanding on the subject, and/or, ulterior motives to either raise or lower a course for someones personal gain.
 
Throwing out a high and a low is a common statistical procedure, isn't it? We use to do that for our tournament handicaps, using 6 scores, but really only 4 counted. 

How many deviations from the group would you say is a "clear outlier"?

When someone produces a "clear outlier," are they given the opportunity to justify it as you suggest above?

Why can't Pebble Beach be a 4?  Are ratings of courses objective?  Do you agree with Tom Doak about the status quo hypocrisy? 

Why do you think this generation's raters have it right and previous generations didn't?  Is Harbour Town a Top 100 course or is a slower fall from grace more palatable than having it get a bunch of 5s one year and falling out?

[Mike Sweeney]

If Golfweek would strictly prohibit raters from using their status to either access or be comped at any course we would see at least 100 new memberships purchased throughout the country. That is 1,000,000 dollars into a dying industry. That is a dozen good people not losing their jobs. That matters.  Cultures have been built on 12 good people following one decent man with a dream.

And then another print magazine goes down and those people lose their jobs in that dying industry. Brad Klein and his "followers" have created a new industry and Jesus was a Jew !! 

http://www.golfweek.com/videos/2011/apr/20/1019/

http://golfweekmediakit.com/assets/Golfweek_Events_Golfweek_Events_Golfweek_Events.pdf

The private club model that you cherish is simply changing. Yale used to have caddies, now they don't. Course is still great and the university seems to be holding its own.

There is a reason that all these clubs seek out ratings. They want more members and more unescorted play. Brad Klein is bringing them at least one of those, and probably both.

I now have access to 6+ private courses around NYC through The Cornell Club. No increase in dues, no changes in structure, the clubs simply want additional play at their unescorted rates. Golfweek, Cornell, Outpost, Boxgroove (weird name), Nyackers, Stationers.......the list goes on and on these days of ways to play "private" golf.

PS. I am not a rater but I do love golf, and I keep my membership at Yale despite the fact that I can access the course through The Cornell Club, so don't lecture me!!

[JC Jones]

Mike,

Did you go to Cornell?

[Billsteele]

I believe Mike Sweeney received his MBA from Cornell...despite spending more time on the golf course than in class. What a poor reflection on the Ivy League's graduate programs.

[JC Jones]

The illusive Bill Steele!!

I agree, though I'd expand it, Mike is a poor reflection on a lot of things 

[Mike Sweeney]

Mike,

Did you go to Cornell?

It was coin flip between Cornell and MSU. I picked RTJ over Art Hills !

http://www.golfmsu.msu.edu/about

[Tim Martin]

If Golfweek would strictly prohibit raters from using their status to either access or be comped at any course we would see at least 100 new memberships purchased throughout the country. That is 1,000,000 dollars into a dying industry. That is a dozen good people not losing their jobs. That matters.  Cultures have been built on 12 good people following one decent man with a dream.

And then another print magazine goes down and those people lose their jobs in that dying industry. Brad Klein and his "followers" have created a new industry and Jesus was a Jew !! 

http://www.golfweek.com/videos/2011/apr/20/1019/

http://golfweekmediakit.com/assets/Golfweek_Events_Golfweek_Events_Golfweek_Events.pdf

The private club model that you cherish is simply changing. Yale used to have caddies, now they don't. Course is still great and the university seems to be holding its own.

There is a reason that all these clubs seek out ratings. They want more members and more unescorted play. Brad Klein is bringing them at least one of those, and probably both.

I now have access to 6+ private courses around NYC through The Cornell Club. No increase in dues, no changes in structure, the clubs simply want additional play at their unescorted rates. Golfweek, Cornell, Outpost, Boxgroove (weird name), Nyackers, Stationers.......the list goes on and on these days of ways to play "private" golf.

PS. I am not a rater but I do love golf, and I keep my membership at Yale despite the fact that I can access the course through The Cornell Club, so don't lecture me!!

Mike-Lots of clubs had caddies back in the day. What does that have to do with whether the course is still great or not?  I don`t know how many rounds a year you play at Yale but if it is at least ten we both know it`s a better deal to join at the entry level rate than pay the unaccompanied rate through the Cornell Club. Doesn`t seem like much of a burden.

[Mike Sweeney]

 Doesn`t seem like much of a burden.

Tim,

I have been to Yale three times this year so far and maybe 8 times last year. I have a 15 round membership. Do the math and mind your own business.

[Tim Martin]

 Doesn`t seem like much of a burden.

Tim,

I have been to Yale three times this year so far and maybe 8 times last year. I have a 15 round membership. Do the math and mind your own business.

Mike-Thanks for keeping Yale afloat. The rest of the membership certainly appreciates it.

[JC Jones]

Mike,

Did you go to Cornell?

It was coin flip between Cornell and MSU. I picked RTJ over Art Hills !

http://www.golfmsu.msu.edu/about

While you might be a poor reflection on the Ivy League, you are clearly not dumb enough to go to a public Big Ten school.

[Michael Wharton-Palmer]

Correction....Golfweek raters do NOT pay an annual fee...

[Dan Herrmann]

OK - one question for raters.

Has there ever been a course you would have rated differently (higher or lower) if it weren't for the "peer pressure" factor?

[Mike Hendren]

To borrow from Bill Shakespeare, these rater threads inevitably are nothing more than tales told by idiots, full of sound and fury, signifying nothing. This thread reminds me so much of the whizzing contest we routinely see in Washington.

Finally, with apologies to Bob Seger, just about everybody on this website is "getting their share." 

Bogey

[Adam Clayman]

Dan, The ballots are secret. There is no peer presure or groupthink.

If anything, Brad goes out of his way NOT to influence anyone's opinion. When he talks, it's not about how someone should think, but rather, to get them to think. If he says anything negative about the courses architecture, it's more of a practical matter on the nuts and bolts of construction.  Not an opinion of what he likes or dislikes.

There's is too much disinformation spreading. Ben Sims thread reflects that.