(2) Variance. I would not rate panelists against other panelists based on whether they are within a certain percentage of other ratings. This creates conformity. Panelists try to fit their ratings within a pre-established view of what is acceptable so as not to be rated poorly. This is the reason for such little change in the Golf Digest ratings this year. For example, Golf Digest has Rich Harvest Links rated very highly. However, many people on this site whom I really respect don't think highly of it. Golf Digest should be trying to get this rating "right", rather than entrench it. I believe that the person in charge of the panel should just determine if the panelist is qualified or not based on his ratings. That person may not always be right, but you have to place your trust in someone.
I think this is good and I agree with, though your last part may be unworkable from a practical standpoint. I think you're absolutely correct with regard to variance, and you see this especially with regard to conditioning, for example. I play most of my golf at a somewhat brownish green, far from lush, firm and fast fescue course that I think is probably consistently one of the best conditioned courses i have ever played. But it's not Augusta, and it doesn't look like Augusta, and it's not supposed to look like Augusta. Digest says they'r'e trying to push this type of conditioning as the ideal. It takes less water, and the firmness brings the natural contours of the land into play better than a lush, target style setup. If I rate those conditions as ideal, I get dinged in the final grades because i disagree with the majority of people who think every course should look like Augusta. This will be a problem as long as raters are ranked against other raters, or really against any criteria, and creating specific goals for each criterion is merely a way to grade raters, and not to rate golf courses. If the powers that be say "we have a number in mind for this aspect of a particular course, and we're evaluating how close your rating comes to that mark" they're grading me, not the golf course.
The big problem with trying to come up with a trusted list of golf course ratings is that there are a lot of golf courses, and you need a lot of people in the rater pool to see them all. This necessarily limits the amount of deliberation and consensus. Hell, we argue over which four teams deserve to be in the college football playoff, and those decisions are made by a very small committee. And more importantly, individual scales vary. So we're all on a 0-10 scale, but some of us actually use that, while other are in practice really only rating in the 4-8 band. If rater A gives the worst course he's seen a four, and B gives it a zero, they both agree that it's terrible, but they're not using the same scale, and when you're dealing with a thousand different de facto scales, it becomes very difficult to evaluate raters without normalizing each and every one, especially when there's little opportunity given to raters to justify individual rankings.
Case in point, I gave a very low rating to a course I played in 2015. It was arguably one of the worst courses I've ever played, and when evaluated in terms of value (not one of the criteria, granted) it's undeniably the worst. The mean total rating for that course was, among other raters, the worst of all the courses I rated in the last two years, so people somewhat agree with me. But it was flagged because I'm apparently using the entirety of the 10 point scale, whereas no one else seems to rate anything lower than a 4 or higher than an 8. So this course gets about a five, a relatively terrible score, while I gave it a 2.5, which for me was also a relatively terrible score. So basically I agreed with everyone, just not in the correct way. But you can't really sit down with a thousand raters and ask them to justify each individual score. My goal is to be able to justify each rating in each category if called to the principal's office. That's all I can do. I don't know of any panelists who have been kicked off the panel, but in my opinion it should only be done for things like fraud, or clear violations of policy. A "good" rater is someone who takes the role seriously, can defend their opinions, and is committed to evaluating some minimum number of courses per year. Those decisions should be made before someone is handed the keys. If someone is "bad" rater because they don't write down the numbers that the publishers want to see, then the evaluation of the rater is kind of a sham. This may get me booted from the panel, but oh well.