Speaking from a UK perspective, I suspect square greens are largely a hangover from the old days from the first golfing boom. Not sure why greens were square (either built square, or cut square where they just went with the lay of the land) but you can still see them on some old designs that have had minimum upgrading through the years, particularly with greens built into the side of hills. One good example is Spey Bay on the Moray coast. You can also see them in early photos in the contemporary golf mags, like the example of Prestwick that Ian posted.
For a period they were also built level (1890’s early 1900’s) before critical press comment against “bowling green” like flat greens lead to greens being built with undulations although often retaining the square shape. MacKenzie, Colt and others later picked up on this reaction to flat greens in their own designs, taking it one step further with extremely undulating greens like at Sitwell Park, before again critical comment lead them to toning down their own green designs.
I suspect it was at that point that greens started to become more irregularly shaped. But going back to why greens were square in the first place, and I suspect it was to do with the introduction of mowing machines in the 1830’s (?). When you think of other games like cricket and bowls and the rectangular nature of the wicket and bowling green, and of how golf greens were cut, you do wonder if it was purely a product of cutting grass in lines as opposed to a sweeping motion with a scythe.
Related to that, I also wonder how many of the very early constructors and greenkeepers of the first boom had worked on bowling greens and what influence that had. Anyway it’s easy to see how square greens ended up in US with the influx of Scots and English professional/greenkeepers in the early days.
Niall