Any caddy can ruin a round or enhance a round, but I don't believe they're able to diminish a great golf course unless you let them.
I have had fantastic experiences with unknown caddies at both known and unknown courses. I think the key is in knowing and discussing your preferences at the beginning of the round. I don't always remember to do this, but am always happy when I do. For instance, from a caddy who doesn't know my game, I NEVER want to hear which specific club to use. I generally like to ask about distance and how long the shot will play, but i almost always want to ask.
This is especially true of putting. I've had the great experience of forming a relationship with a couple of caddies at clubs to which i've belonged, who come to know my game and that I tend to die the ball into the hole or ram it into the back on this or that kind of putt. In these rare cases, having a second opinion on a read is really valuable. But likewise, when I've forgotten to tell the unfamiliar caddy that I like to read my own putts, it's really annoying to have them (with the best of intentions) try to help read a putt for me that I've already decided on and am ready to putt.
Also know that I am generally happy to carry my own luggage, so to speak, on the course. But am also happy to support a caddy program where and when appropriate. If the choice is take a buggy or take a caddy, and those are the only two choices, it's a no-brainer to take a caddy.
In any case, the thread topic is a good question. I think the answer is, only if you let them.