I haven't had any of these encounters yet. I constantly get people who want to watch me fly and think it is super cool. I also try to fly in a manner where I don't intrude on people's privacy (and it sounds like you weren't either).
There's a lot of miserable folks out there who want nothing...