I do follow the
Drone code when I fly.
If I'm flying early morning (which I mostly tend to) the danger to anyone, by me flying over a property, would be minimal because most people would be in their houses.
I would still be flying at least 50m away from them.
If you look at the
PDF it clearly shows that 50m 'away' is 50m vertically and 50m horizontally, so in essence it would be a globe around a property or person. In built up areas the distance is 150m (which basically means you can't fly in built up areas!)
If you did cause damage to a person or property and you were before a judge, you need to be able to convince the judge/jury that you had done a proper risk assessment and deemed it safe to fly. Only if you believe you could do that should you be flying anyway! YOU are always responsible and accountable for your drone.
As per your original post - if someone is aggressive towards you when you fly (even though you may be flying legally and safely), apologise and land your drone and move on somewhere else. It stops any further aggression and the problem is solved. People DO think they control the airspace above their property and person, much like they think they own the pavement outside their property!
Something else I have considered is to carry a few copies of the Drone code and give them a copy if they complain but are not aggressive!