My dad was a golfer and so I basically grew up with it, knowing he was playing, watching golf on TV with him, etc. Local courses all had restrictions on the minimum age of players, so I think I first started getting out around age 7 ... I was a big, tall kid too so even places that wouldn't have allowed someone that young didn't usually ask.
I don't really ever recall not being obsessed with the game. I played in the backyard, in the open lot behind our house, and on cheesy computer games. I quickly became obsessed with courses, too; finding out whatever I could about them, reading books, drawing.
There's a lot I love about the game. Most of it is the way it's so impossible to master. Even the best players in the world are eternally unsatisfied with their games. (It takes a certain temperament to see this as a good thing, I suppose.) The challenge is ever varying. You can play a course you've never seen before or one you've played a hundred times before, but it's still never the same. That was probably the initial appeal and it's still a lot of the appeal. As I've gotten older, I have also come to think of golf as a sort of therapy time--a place where I can go and my mind can be quiet and the stresses of life are put aside for a few hours. If I'm playing well that's great; if not, I am now much better at appreciating the new challenge of every shot, and the simple pleasure of being outside, walking, etc.