From the website of the Gosport and Stokes Bay GC - formerly the United Services GC (with thanks to my friend Mike Morrison)
The original nine-hole course of 1885 was laid out on Haslar Common and the glacis of Fort Moncton. It was on this course that the term “bogey” originated. In 1890 Mr H. Rotherham of the Coventry Golf Club inaugurated the basic idea of playing against “ground score”, which was the number of strokes a golfer playing steady golf would be expected to take on each hole. Other clubs soon followed this example.
In 1892 the Secretary of the Great Yarmouth Golf Club came to Haslar and played at the United Services Golf Club. He introduced the BogeyMan idea to the Honourable Secretary and they worked out a ground score for the course. The actual term Bogeyman originated from a popular tune of the era.
When the two secretaries were about to tee-off, one stated that as the club was an officers’ club, and every member had a rank, then the new invisible member, who never played a bad shot, must surely be a Colonel. Consequently this was how the name Colonel Bogey came into existence.