Lloyd,
Well, it's not really a rule, but I'm not going to quibble about it, and of course, who here wouldn't let a single or a deuce or any group of faster players through if conditions permit. The point I was trying to make was one you touched upon later in your post, you know, the putz who runs up on the group ahead and makes his presence known, especially when there is no place to go.
Put any two-hours-for-18-holes twosome on a busy course and they are going to negatively impact the pace of play for every other group out there. Think about it, how much waiting time does a group experience when letting another group pass? Let's just call it one or two minutes, for arguments sake. They'll have to work themselves through at least 17, and probably more, other groups. Add it up and they'll probably have caused the course to slow down by at least 20, but probably more, minutes for everyone else out there.
Personally, I search out enjoyable courses and play them when I know they will be less crowded(easier to do in our neck of the woods) because I enjoy a brisk pace.
No doubt, most people could probably pick up the pace, but the pace that Garland suggests he plays at is a nuisance to everyone else on a busy course.