Isn't it also likely that a putt started slightly off line will get help from a little imperfection, and go in when it shouldn't have?
You could prove mathematically that it doesn't work out that way. Randomness is still overall a negative. Introducing randomness decreases the ability to make putts, largely because you're not talking about one bump here or there, but many bumps.
Even if you're talking about only
two bumps, a bump earlier in the putt will have a much greater effect on where the ball finishes than a bump later in the putt. So a putt that hits an early bump that deflects it 5° right when the ball is still 30' from the hole (on a 35' putt) will not be countered by a bump that sends the ball left 5° 3' from the hole.
Why do PGA Tour players complain about the greens at Pebble or about them at Chambers Bay? Because they're losing control, and they don't putt as well there when they're bumpier.