If the hole is much smaller, it would be impossible to hole shots from off the green. Even ignoring everything else, that one factor alone makes it ridiculous to even consider.
Anyway, you obviously aren't paying attention because within GCA at least no one really cares what scores are being shot. Its the fact that holes that were designed to be a driver/long iron are now driver/PW due to the distance the ball travels being totally out of control that is the concern, its hazards and doglegs that are now meaningless when it is possible for more and more of the field to simply fly over everything. Making the hole smaller might mean fewer birdies after a PW to 12 feet, but it isn't going to make it any less boring to watch the whole field hitting wedges at the pin whether the hole is 3" in diameter or 8".