I have played Harbour Town far more often than any other world renowned golf course, probably close to 20 times. I have always thought it is something of an acquired taste, the first time or two I didn't think that much of it, but after repeated playings I started to understand the subtlety of the design far better.
The standard knocks have always been the pace of play, the conditioning, the price to play and the omnipresent housing element. However conditioning has improved quite a bit in recent years, and personally, I don't even notice the housing, so much attention needs to be paid to getting your ball into the proper position on the fairways. To me the housing just fades into the background, barely noticeable.