The oldest public course that I've played would have to be Van Cortlandt Park (1895) in the Bronx. The clubhouse and locker room are worth the visit, and playing 18 is just icing on the cake. I drove there from Philly to play my 600th different golf course. I like to have something particularly memorable for milestone courses. I played in a New York foursome, which the pro explained to me was me, a Manhattan bus driver, an investment banker, and a guy from Dublin Ireland.
The oldest private course in the U.S. for me would have to be Merchantville in South Jessey, which is of uncertain vintage. Though the card and the merchandise proclaim "1892", it's probably more like 1895 for golf, as opposed to football, rugby or tennis.
The oldest course (non-U.S.) is indisputably the Old Course at St. Andrews.
The oldest course logo in the States might be Philadelphia Cricket Club at 1754 but that was for cricket and tennis, around what was to become the St. Martin's course in Chestnut Hill. Cricket is the only place in the country that I know of that can claim three operating courses that were established in three different centuries (19th, 20th, and 21st). Playing all three is an interesting way to trace the evolution of course design.
A classic feature that I've seen and I'll mention might be "chocolate drops", like the ones running down the left side of the big par 5 that abuts the river at Tilly's Shawnee-on-Delaware. Those drops also appear at Pocono Manor, where the East course manages to defend every green with drops (back right & left, front right & left) on a course that involves exactly one sand bunker over 18 holes.