Tom,
Golf, for me, is both sport and game. The need for physical activity would define it as sport but because there are rules it is also a game. I like how JohnV explained it but it also can be seen from another perspective that I feel is just as valid.
Golf as a game has many rules. When we are playing golf in a competitive fashion we must follow these rules or else be penalized in some fashion.
Golf as sport doesn't have any rules. When we are out on the course for the sheer enjoyment of hitting shots (preferably good ones), putting, chipping, etc., we are playing the sport of golf.
Who knows what Hemingway had in mind but bullfighting has plenty of physical activity, a prerequisite for consideration as a sport, and no rules (staying away from the horns is not a rule, just good, common sense), therefore it sure isn't a game.