Kalen,
Everyone who hits it to the wrong side of the green has the same putt over the bad ground. Think of this way (note this is not entirely true), the player can restore the putting green to the condition it was in at the start of the day. That way everyone gets the same result. If that area is so bad on the putting green, the Committee can make it ground under repair. But, even then, the player who has it on his line of putt could get relief, but the one off the green wouldn't.
The putting green is different than the general area, is different from bunkers. You get different permissions, different breaks etc. There is (almost) no distinction between the fairway and the rough in the rules.