The Jeffersonville selection is an interesting one, and like many of these conversations, perhaps says more about the player than the hole. There's a line to take from the appropriate tee that cuts off significant distance. It's never been the most difficult hole there to me (8, 11, 13, 17 are all better candidates) though a golfer can make it difficult.
And that's sort of the crux, to me, of this conversation. Stroke allocation is as much about where the bogey golfer needs TWO strokes against the scratch, if not more, than where the 1-handicap needs a stroke against the scratch. The first situation likely involves a penalty stroke while the second situation likely involves a difficult putting green.
Difficulty tends to boil down to two things: Possibility of a HUGE number v. difficulty in making par or better.
Let's call that the "Tobacco Road Assumption."
Now that I've stalled for more than enough words I'll nominate my candidate:
I mean, the First hole is the No. 2 handicapped hole, so I take a bit of an issue that this is more the golfer than anything else. As for the line from the tee, I guess you mean cutting off to the left, over all the trees. That is any where from a 285 - 325 yard tee shot, uphill. A tee shot to the bunkers on the right side is about 250 uphill, leaving you with about leaving you about 170 - 180 into the green. They shaved and made a lot of the hillside before and to the right of the green, so it's possible to utilize the ground a bit more on the approach.
Also, the Seventeenth is a lot more subject to taking a line off the tee to the right in making the approach much more comfortable. And you don't need nearly as much length off the tee as you at the First.
I see the First as fairly difficult in making par while I see your point there may be other holes here where the possibility of a huge number is greater, where OB and/or water is a lot more in play. Yet that possibility is still there if any shot moves off to the left down the hillside too much.
But I don't think I'm coming out of left field saying the First is the toughest!