Well, that's the point of having all the permits and certifications up to date, let him call the police once and end of story.
Once a crazy woman came with his car to my front door and started yelling at me, but I didn't even had time to land before she went away. Later I discovered she lives in a house at about 100 meters or so, but never came back... and it's been 150hours of flight since then XDD
When you fly drones expect this to happen, just stay calm and let them call the cops (or call them yourself), if you have all the permits and certifications and are flying more or less legally you'll be fine.
That's why I love Long Range flights, you never get assaulted like that.
PS: The trick with if you fly from your home is to climb to top altitude (120m) right after take off, then move 200m out of the way and just fly normally in a 200-500 meters range from your house. When you are coming back do the same, never come back at low altitude, increase the altitude to 120m and then just go down right over your propriety.
The people that will come to you is the people you have in a radius of 200 meters or so, past that they'll just don't know where the drone came from, so 200 to 500 meters is the "safe" VLOS radius, specially for little drones like the
Mini 2.