When we were putting together Beyond The Contour's Top 100 + Top 100 Public in Canada (there's been Canadian rankings for 40-odd years, but never a Top 100 public & any ranking is generally pretty vapid), we really liked GOLF Magazine's ranking structure of "buckets," ie, top 3, top 5, top 25, top 50—whatever GOLF used, I can't remember off the top of my head. We did it slightly different, asking people to list courses they had seen and how they'd place them in the top 10, and then in buckets from 11-20, 21-30, 31-40, 41-50, 51-75, 76-100, and then the close calls from 101-125.
Perhaps we did it to differentiate ourselves or just a necessity of such a large land-mass, spread out country with our small panel, but we had every panellist fill out a spreadsheet where they checked off the golf courses they had played/seen from a master list of contenders (based on previous lists and other rankings like SCORE + Top 100 & some regional choices). The more someone saw, the more weight their ranking held. Example:
If someone—say, Panellist A—saw 150 of the 200 courses listed, their weight would be 150. If Panellist B saw 75 courses of the 200, their weight was obviously 75.
Panellist A ranks Cabot Links as the #1 golf course in Canada and Jasper Park Lodge #2. Panellist B ranks Jasper Park at 1, and Cabot Links at 2.
Panellist ACabot Links: 1 x 150 = 150
Jasper Park: 2 x 150 = 300
Panellist B Jasper Park: 1 x 75 = 75
Cabot Links: 2 x 75 = 150
From there, the totals are added up:
Cabot Links: 300
Jasper Park: 375
And divided by the total divisor, ie, the total between the two panellists (150 + 75 = 225):
Cabot Links: 1.33
Jasper Park: 1.66
So, Cabot Links average is lower, and thus, the better golf course. We did that with 30 panellists, 200 golf courses, and it gave us an interesting list.
While not exactly what you're talking about Mayday, we didn't use criteria either, just asked people to slot them into the respective tiers and check off what they had seen. What we found was the top ~10 most travelled people, by the end of the exercise, could move a course up one or two in the top 50 (because generally, in Canada, more people have seen the top 50 given the sharp drop off in quality), and up to 10 spots in the bottom 50. The bottom ~20 panellists likely couldn't shift anything in the top 50 by the end of the exercise, but could move something up or down 2-4 spots, depending on their weight. I would've liked to seen some reasoning on why people did what they did, but the product turned out better than I was hoping for so I didn't really begin to start poking around.
I'm happy to host anything anyone wants to put together, but I think Ran wants a break from rankings for a bit so we might have to hold off while he detoxes from corporate media