Greg:
I think the quality of a panel is a valid and interesting topic. Here are some thoughts:
1. Magazines should first ask any applicant to the panel for a list of every golf course that he has played to date. Simply put, I have played with panelists with little experience on great courses and their opinion is usually flawed. Everyone has been there at a time. Just think back to "that course" that you once thought was as good as golf got. It is usually embarrassing.
2. Magazines should next ask what you have done to enhance your knowledge of great golf courses. If one hasn't at least read the books by Thomas, Mackenzie, Doak, Tillinghast, Colt, Ross and others, they shouldn't be rating a course. Without being educated on a topic, how can you be asked to apply the necessay standards. You would be surprised how many panelists can't discuss basic architectural principles.
3. While it is not necessary for a panelist to be a great player (in fact I don't think most pros know much about great architecture), there has to be a base skill level.... and Mike Keiser, Tom Doak and Brad Klein all qualify. You need the ability to see different shots and the value of those shots based on the design of holes. Often the better players can see these types of options and thus rate the variety of shots needed for a course. Average players that have this ability are usually based on their love for the topic of golf architecture. However, I do agree that a 20+ handicap shouldn't be involved unless age has caught up with him.
4. I would never comp a rater if I was a course. If you are Shadow Creek or Pebble Beach and you want to offer a discounted rate so it is not so expensive, that is fine, as a rater shouldn't have to go broke rating courses. However, courses shouldn't buy ratings, just as magazines shouldn't profit off ratings. There has to be some middle ground on this issue. If members don't play for free, neither should a rater.
Don't know if this is of any value. Just some thoughts.