[Brad Klein]

Tom,

You are certainly correct in that we don't add up each of the individual criteria and pretend to be scientific about that -- but we do have thresholds (15 minimum votes for top-100), and criteria we ask each rater to evaluate, and I spend an awful lot of time managing spread sheets for each of several different categories of golf courses, which is great when you need to spit out the top-100 residential or the top-100 resort courses, or the best 50 new for the last two years. Given my thin scientific background, spreadsheets are pretty high tech. But my my point in the end is, like yours, that this is nothing but objective aggregation of subjective standards.

[Mark_Fine]

I suspect this has already been stated but the advantage of a course having a lot of votes/reviews is that one great review or one very poor review will get balanced out.  Also, the more votes from your panel, the more accurate the assesment reflects the overall views of that panel.  

[Brad Klein]

Mark, that's the statistical advantage of X+1  votes; it cancels out the arbitrary one. My experience, reflected in what some mathematical types tell me is the case generally, is that once you get to 30 votes, an individual vote has no real consequence. I might add that there's virtually no movement thereafter, as well.

[Wayne Freeman]

I've played golf with raters from all 3 mags, and have found that everyone is very serious about the task at hand.  Furthermore, no one I've ever played with expected to be comped. How you're treated shouldn't be any real factor in the rating either, although most courses accommodatiing raters usually put on their Sunday best.  I didn't rate it, but I was treated poorest at Cypress Point- my all time #1,  when we had to put our shoes on in the parking lot and go right to the tee box. They couldn't take anything away from the joy I experienced that day.

[Tim Bert]

Why don't the publications simply mandate that the raters can't accept comps?  This would get rid of any perceived (or real) bias based on whether a rater paid or not, and it may also filter the list of raters down to those that are genuinely interested in rating the architecture as opposed to those just trying to play a bunch of nice courses for cheap or free.

Go ahead, tell me why I'm naive and it would never work!

[Tim Bert]

How you're treated shouldn't be any real factor in the rating either, although most courses accommodatiing raters usually put on their Sunday best.  I didn't rate it, but I was treated poorest at Cypress Point- my all time #1,  when we had to put our shoes on in the parking lot and go right to the tee box. They couldn't take anything away from the joy I experienced that day.

Put your shoes on in the parking lot?  Oh, the uncivil treatment!!!  I don't know too many people that wouldn't gladly trade such public humiliation for a crack at CPC.

[Wayne Freeman]

Tim-  that's not the whole of it.  The attitude was terrible in general.  We weren't able to even warm up or hit a few practice putts.  And if the caddies didn't like your bag, they would dump out your clubs and put them in one of their small carry bags.

[Tom_Doak]

Wayne:

Dr. Klein started emptying out Jack Nicklaus' bag at Sebonack when he caddied for him on opening day, to make the bag lighter.  They do it all the time at St. Andrews, too.  So you're getting no sympathy here.

[Tim Bert]

I'm not feeling sympathetic yet either.  Maybe if they slashed your tires, stripped you, tarred and feathered you, and talked during your backswing the entire round... maybe.

[ForkaB]

Sean

Sounds like Group think and dodgy mathematics/statisitics to me (unless you can do double-blind type stuff, which you cannot--imagine a control group of reasonably competent and informed golfers standing on the 1st tee at St. Andrews and not knowing where they were?).

Why don't we all just lie back and enjoy the fact that all ratings/rankings are just bullshit, but bullshit we know and love?

[Brad Klein]

Doak, I resent that misinformation. Jack had three-dozen golf balls in there, I just took out two dozen and left him with 12. Seemed like more than enough to me. I didn't try to dump all the rain gear, though, or the modest Duane Reade drug store supply of items that were also in there.

Sean, there's no "maxing" out at 30, but that's the number at which statistical stability is reached. Don't blame me or anyone else for that one. It's just (literally) in the nature of things.

[Jonathan Cummings]

Klein emptying JN's bag - LOL, what a mental picture that makes!

Another widely accepted statistical practice (assuming you have the luxury of lots of votes) is to throw out the highest and lowest one or two votes.  This causes standard deviations to be lower and confidence levels higher for the generated averages.

Another sound practice is to monitor the overall average of rater X against the average of the overall panel.  If rater X is consistantly higher/lower than the rest of the panel then his votes are introducing bias into the calculated averages.  You can then either quantify rater X's bias and adjust all his scores by that amount or educate him on his voting pattern ("hey, you're higher then everybody else on everything...").

JC

[Brad Klein]

16,000 golf courses, maybe 1,500 of them "vote-worthy," and you think 150 would be a good settled number. Good luck. To do that you'd need a minimum of 3,500 raters.

[Brad Klein]

How many raters?

Golf Magazine 100+ (not enough for statistical validity or effective coverage)

Golf Digest 800 +/- (too many to handle)

Golfweek 400 (just right, but takes an awful lot of baby sitting by an overworked staff of one plus some admin help in the home office)  

[John Kavanaugh]

Sean,

Stop before you convince someone to hire more raters...This is a dying industry that even at current numbers can not support giving any more product away.  1300 raters * 50 courses * $150 greenfee = $9,750,000

[Jim Franklin]

Wayne -

That is funny about CPC. The first time I played there, one of my buddies was changing his shoes on the first tee and had one loafer on and one golf shoe on when the caddie looked over and said, "We got a mighty fine locker room over there you can use to change shoes." We were like, holy crap, we are going to get booted before we even play the course. So he ends up putting his loafer back on and changes his hoes in the mighty fine locker room.

I must say I have been treated great every time I have been out there (6). I can't wait to see it again.

JB -

You crack me up, but give it a rest. The rankings are VERY popular with the masses and the magazines enjoy them as well. If you run across rude/bad/inappropriate/undesireable raters, let me know and I will try to get them removed from the panel. I know GD and GW don't want their publications embarrassed by anyone. Send me a private IM if you wish.

Jim  

[John Kavanaugh]

Jim,

I do believe the $10,000,000 spent on raters is better for the game than the $20,000,000 Nike pays Tiger.  My post above was not intended to be a rater bash or a call to remove any existing panelists.  I don't think people realize how many more people either as friends or raters for local mags or papers or societies receive comped golf above and beyond the 1300 people associated with the big three.  I love being a second tier recipient and would hate to see the process changed.  I love raters more than any non rater I know...

Did you see this dangerous and disturbing article in USA Today.  I think access to Ghin needs to be stopped.

http://www.usatoday.com/money/companies/management/2006-11-01-ceos-golf-memberships_x.htm

[Jim Franklin]

John -

I agree that paying Tiger $20 mill only increases the cost for the rest of us. I wish I knew what to do about it, but am too busy with three kids, work, travel etc... to even think of a solution (probably isn't one anyway). I have not read the article, but will when I get out of my Board meeting this afternoon.

[JESII]

Hey Tommy N,

I would say the earth is in serious tilt mode when on consecutive days we have Mike Cirba apologizing for his criticism of the Merion bunker project and John Kavanaugh bending over conciliatory to the ranker society.

What the hell?

[John Kavanaugh]

JES,

I don't know if you have seen what's in Franklin's bag..but him saying paying Tiger 20 mil is bad for the game trumps it all.

[Adam Clayman]

But John, You are incorrect that raters cost 10,000,000.00 dollars.

Comped golf is not a cost. More than likely, it isn't even a lost opportunity for revenue.

[JESII]

JES,

I don't know if you have seen what's in Franklin's bag..but him saying paying Tiger 20 mil is bad for the game trumps it all.

And Tom freakin' Paul just conceeded that soft greens may be OK so long as granny is happy on the firm and fast thread.  

Double what the hell??

[Jim Franklin]

And another thing, look in my closet and see how many shirts I own. I certainly try to support my local proshops so there really isn't that much lost revenue.

[John Kavanaugh]

Comped golf is not a cost. More than likely, it isn't even a lost opportunity for revenue.

I agree with the above statement 100%.  I know that that not being a rater causes me to play courses where I pay dues instead of going out and creating lost revenue at new and different courses I have not seen.  

[Tim Bert]

Brad,

You came the closest to answering my original question with your early comments in the thread.

What I'm really trying to figure out is if the ratings are a straight average of the sample that's seen the course, with no additional weight given if a higher percentage of the total rater population has seen the course.

Said another way, if the minimum number of raters has seen one course and 200 raters has seen another and the average score works out the same - are they ranked the same or is there more credibility and therefore a higher score assigned to the course with more ratings?

[Tags:]

Tags:

[Tags:]

Tags:

Call to undefined function theme_linktree()