Model railroading? No. But, judging by the OP, railfanning is included - Even for a harmless Canadian like myself, it's hard to avoid trouble. I remember vividly being threatened with arrest by a female RCMP officer while taking photos from a road, through a fence, of the Winnipeg VIA station. I was 12 at the time, and (you guessed it) it was less than a year after September 11th.
A few years later, I'm over in Missoula, Montana, enjoying some MRL action, when a BNSF cop pulls up and tells me that I'm on railroad property and that if he saw me within a block of the tracks again he'd cite me for trespassing. Now, that might only seem a bit firm until you take into account that I am most definitely on a public road, not even at a crossing! Needless to say, I gave him a defiant glare and agreed to get out of his town, as it were, and went straight on over to the far side of the creek where a far more legally grey shot was waiting for me. (Warehouse private property, not railroad.)
'Course, didn't see him again.
Most other situations I've handled with the grace of a gentleman...or at least with enough grace that I haven't been told to leave since.