Donald Ross wrote that you could, so I think you could. Not sure that he specified whether he meant target size, surrounding hazards, or contours. In any case, probably one of three would be enough to make it slightly harder, would be really tough if two or all three aspects were used, and counterproductive, i.e., smaller greens usually need to be 100% cuppable. Even the TPC Island green has surrounding hazards and green contours but has enough size to accommodate the shot, probably statistically oversized for the 100+ yards in.
For shorter par 3's, the ball on the tee may also mean more backspin for all players. The practical side of design suggests a bigger green to distribute ball marks and for at least the front of the green, where most shots land, a flatter green, because ball marks cause less damage. A bigger green, together with a probable desire for small targets to be tougher suggests a multi-target green, separated by steps, ridge, or valley is one typical design solution. And, each of those targets might have a different slope toward, away, or to the side of the line of play to make it play differently with every pin move.
The other thing to consider is that for many average players, the par 3 hole is sort of a respite, where they feel they have a better chance of hitting the green. On public courses, do we really want to take that away from them?