I've been lucky enough to play Congressional fairly often -- certainly far more often than any other course that the pros play, either regularly or in the majors. A few observations:
1. I think it's a great course and a terrific test of golf. But what holds it back from being in the same conversation with the upper, upper echelon is that there isn't anything (except perhaps for 18) that stands out -- nothing particularly unique, no quirk, no Tillie greens/bunkers, etc. It's still incredibly demanding and a great place to play, but Matt's right that there are few holes that are truly, truly special. [Indeed, there are arguably as many or more special holes on the gold course.]
2. The course is on a really nice piece of property, but a lot of the vistas are lost or obscured because of trees. If they did some tree clearing, there are spots on the course that would have incredible views of 5+ holes. I realize the trees are part of the course's defenses, but I think they could be trimmed without significantly affecting play.
3. There are several holes that have interesting greenside areas of tightly mowed grass [e.g., where Tiger putted from on #16 yesterday], but with a few exceptions, these areas are almost always short of the green. The greens already have some interesting contours, and I think it would make several greensites really cool if they added more tightly mowed areas, particularly on the sides and in back of greens.
4. It's obvious, and Andy already said it, but these guys are preposterously good. Watching what they're doing to 9 and 16 is just incredible -- those are not pushover par 5's! -- as is their ability to hit long irons to the right spots on greens like #12.