I can't argue with the choices that have been listed so far, especially Oakmont#1, Dornoch #14, Carnoustie #18, and TOC #17. However, I have played each of those holes only 2-5 times.
Thinking of holes I have played a lot, none are as tough on me as #5 at Pinehurst #2. I usually play to make a 5, and even that is pretty tough. I know that most here have played it. For starters, it is very long and it requires 4 very good shots, or at least one outstanding shot.
For the average guy, it takes two excellent shots just to reach the green in two. Even then it is very difficult to keep the ball on the green. Any shot landing or running up on the left side of the green will probably turn hard left and wind up in the left front bunker. Or maybe it runs long and over the back left down a steep slope. The only safe miss is long right from where you have a decent chance of getting up and in. The short hitter who leaves a wedge for the third shot will find that the green is even hard to hit with a wedge. BTW, the green is very difficult to read and 3-putts are very common.
Most long hitters have similar problems. During the past two Opens, most pros found the green hard to hit from a hanging, side-hill lie and there were very few birdies made. The previous hole is a par 5. If you put a gun to my head and told me I had to make a 4 and could choose which hole to play, I would choose the par 5 where I have probably made more 4's.
Jim Lewis