Anthony
From what I know these matches were played within the rules by the players, but as for the spectators......
Melvyn, please feel free to correct me, but my understanding is that in these challenge matches, the players were using OPM (Other People's Money) and that there may well have been betting going on by the sponsors, fans and even participants!
In 1887, Archie Simpson laid out a challenge to all players of the day for a home and away match with a purse of 100 pounds. Willie Park Jr. (1887 Open Champion) took him up on this and Archie beat him by 9 holes (out of 36) at Willie's home course (Musselburgh) and then another 2 holes at Carnoustie. When Archie finally got a day job (as pro at Royal Aberdeen in 1891) his annual salary was 4 quid.
Often I think that one of the most class-centered/misconceived rules of the R&A/USGA is the ban on betting. How else are we of the lower classes expected to have any fun?
Good stuff, Melvyn
Rich