Peter,
I think Taft's efforts for golf backfired in part because he had a PR problem right out of the gate. When Teddy Roosevelt chose, somewhat reluctantly, to emulate Washington's example and step aside after two terms, TR did a heck of a lot to promote Taft within the Republican Party and to the press and public leading up to Taft's election. Taft never really wanted the Presidency; he wanted to be Chief Justice of the Supreme Court. TR pushed him, along with Taft's wife -- she wanted to be a First Lady -- and much of the party leadership, into the nomination. When Taft won, the first thing he did was go play golf, which did not sit well with Roosevelt and the press. When he continued to play golf instead of tackling issues with the "square deal" energy that the public came to associate with TR, the reputation of golf suffered even more, along with Taft's popularity. The Republican Party at that time was the party of the railroad & oil barons, bankers, Old Money -- in short, wealth. Wilson won the election for the Democrats four years later, beating Taft and TR, who left the Republican Party and ran as a Progressive Party candidate.