As mentioned, I think it depends on the course, and given the number of shotgun starts in modern golf, you just have to accept it. I think it also depends on how much you want to play a course. While PB plays better starting at no. 1, I can imagine not really caring if I had never played Pebble.
The right answer is "its a grey area." Admit its hard to start on the no. 1 handicap hole (unless playing Bob O Link near Chicago, which has (had?) the first hole at No. 1.
As to the design aspects, yes, we consider rhythm and all that, but in reality, we find the best holes first, maybe tweak those if we can in considering rhythm. If we have found 18 good holes, it shouldn't be terrible to play out of order. And, overall rhythm is hard to consider for most. The biggest factor is probably putting the harder nine as the back nine, and perhaps even more in particular, the harder opening hole as 10 rather than 1.
At my recently opened Tempest, that was the issue. The pro shop has a great view of hole 10, which is pretty hard, whereas on the far side of the clubhouse, we designed holes 1-4 as that traditional gentle handshake. Also, holes 11 and 12 are quite hard, making for a really slow start. If it wasn't for that, I imagine some future pro would consider turning the nines around (as they so often do) just for visual control from his desk/counter.