It really isn't a bad drive from Halifax. There's some slow downs (traffic lights etc.) in the middle, but otherwise, it's a very nice drive. The last leg, once on Cape Breton Island, is my favorite part.
The most amazing drive is from Inverness to Highlands Links along the coast road. Better than Big Sur, because you can see it.
The region exudes a laid back ethos. Caring about drive times, would miss the whole point of destination golf. Chill'in with Nature.