Jay:
To me, quirk means unconventional, or something you're unlikely to find at a typical golf course (meaning, an average muni or daily fee course). So, what does that mean? I'd argue some or alot of the following:
-- Blind tee shots or approach shots into greens (meaning, the player can't see the result of his shot). One or two per round is not enough to make a course quirky; a half-dozen is close, and 9 or 10 per round probably makes it quirky. Machrie, on the island of Islay in Scotland, is probably the definition of this (see James Finegan's book on golf courses in Scotland).
-- Unconventional routing or hole arrangements. Elie is well-known for having 16 par 4s; while the holes themselves may not necessarily be quirky, the totality of that many par 4s represents a quirky approach to arranging holes. Lawsonia, with its stretch of holes from 9-14 that goes (in terms of par) 5-3-5-3-5-3, is awfully quirky for an American course (although I wouldn't necessarily describe any of the individual holes as quirky; really good -- yes, but not quirky).
-- Half-par holes are sometimes viewed as quirky, and a course with a lot of them probably could be characterized as quirky. See this thread on Madison WI's Blackhawk CC, about as quirky of a traditional parkland course as you'll find:
http://www.golfclubatlas.com/forum/index.php/topic,41775.0.html The Balcomie course at Crail is another course I'd characterize as quirky under this category.
-- The use of strange land-forms or man-made artifacts/features in the design of the hole probably defines quirky for many; see the Pit at North Berwick, with its approach shot over an old stone wall, or the old barricade/fortress incorporated at Painswick. Cullen, on the northeastern coast of Scotland west of Fraserburgh and east of Nairn, routes its course around enormous old sea stacks, and they come into play on several holes.
http://www.cullengolfclub.co.uk/index.htm -- Crossing holes probably meets the definition of quirk; far more examples in the UK, such as Stonehaven, which has one hole that crosses
two fairways during its journey. Often found in routings confined to a small amount of space, and thus the layout maximizes the land by the use of crossing fairways. Sean Arble's picture threads of the UK's lesser-known courses often highlight crossing holes.
-- I think shared fairways represent a kind of quirk -- Wild Horse in Nebraska has a modern example on the front nine, and they are often found in the UK.
Things that to me don't really represent quirk: dramatic internal green contours (everyone's doing them these days); false fronts; front-to-back tilted greens; back-to-back holes with the same par (excluding par 4s; see the 5-5 finish at Baltusrol Lower or PacDunes 3-3 strertch).