Par is a standard of excellence, similar to scoring an A on a test or 1600 on the SATs. When it is said that we play the course, it normally means that we compete with old man par.
Take TEP's example. Who derives more pleasure, a golfer who beats his opponent with a birdie, or one who does the same with a quadruplebogey? In the former, both players may feel great satisfaction in playing the hole well, while winning a hole with a double-par probably does little to warm either contestant.
Golf without par is like having a course without a routing or boundaries. Tom Doak's "Sheep Ranch" may come close to this, but is it really golf as we know it?
My home club used to host an annual "Hard Hat Open" which was basically cross-country golf with no par targets or defined corridors. The mission was to go from designated tees to designated greens throughout the course without killing each other. It was normally played as a scramble and the team with the fewest number of strokes won. I think that it took some 4 hours to play the 9 hole event. Par here was entirely meaningless, and while the event was mildly popular for a time, it was discontinued for a lack of participation.