I agree with Cary & Jeff that the "over the top" comes in the way of overly severe penalties and lost balls. Such as when water catches even marginal mishits or native areas are left way too close to landing areas. The occasional “hit the shot or lose your ball” challenge is fine, but when it is a central theme at a course, it slides to the “OTT” side of the spectrum.
I was surprised to hear Bethpage Black included in this discussion. Frankly, I think Bethpage is a long ways from being overly difficult. Yes, it is a challenge to play well, but not “difficult” in terms of being OTT. The greens are not absurd, OB is rarely an issue, the course is not carved from woods, and there are very few forced carries over “severe” hazards (i.e. water / unplayable lies / waist high grasses). Yes, there is plenty of sand to navigate, but I don’t think sand makes a course OTT because you don’t have to reload. You simply have a more challenging subsequent shot. If you just play “ready golf,” the “challenge” should not force a group to bog down (like lost balls or other severe hazards). As Tom Doak noted, the 6 hour rounds are a function of the “people,” not the course. I suspect that many of the people taking 6 hours at Bethpage would still take 5.5 hours at my local Municipal Course.
In fact, my experience with Bethpage and Oakmont is that they are very similar. It’s almost impossible to lose a ball at either course, so you’re not needing to “reload” or add penalty strokes to your card. Neither of these courses would be “OTT” in my book.
And as we discussed in previous threads, I wouldn’t consider a course like Tobacco Road to be OTT (certainly closer than Oakmont or Bethpage). But if you replaced most of sand with water hazards, that would be a definite candidate.