I'm going to have to be that "Guy" and say that 'taking the high road and walking away only reinforces their bad behavior and they will continue to do it because we made it successful for them". Just because they're ignorant of the rules ie: yes, you do own the airspace over your property but your property ownership comes with an easement to the FAA for any and all aerospace vehicles. It can be explained in a way to de-escalate and to educate as long as we don't come across as "The Know it all Expert". To me it would be the same as walking up to someone driving on the road in your neighborhood and saying "hey you can't drive here" and then expecting them to just leave.
I learned the hard way that letting the "bully" get their way and intimidate you is the best way to let their bad behavior continue. I always prefer to educate as has been said on the Why I'm here and why I'm allowed to be here. I offer to show them the LAANC FAA approval for my flight so they know I've taken the steps for my flight to be "Authorized" by the FAA.
Of course every situation is different, but until we stand our ground and stand up for our rights to be piloting an unmanned aerial vehicle where the FAA authorized the flight, it will continue to happen. Be reasonable, offer to explain and educate without brow beating the person who either may not know better and will be receptive or are just looking to get their way.
I have a thick skin so if you want to flame me, have at it!
Matthew Caswell, SMSgt, USAF (Retired)
FAA Certified Drone Pilot