There are some takeaways from this anecdote.
First, when he first escalated the altercation the police should have gotten the call. You do not know how quickly such and encounter can escalate. If I were the OP I would use the police report and have a restraining order filed against the neighbor. You don't know what kind of crazy he is and a restraining order will make it so that his presence within a certain distance from you will get him taken away without having to explain any further incidents.
Second, something that needs to be stated for law abiding citizens: While you don't know where or when, probably when you least expect it THE FIGHT WILL ALWAYS COME TO YOU! You don't pick the time, place of amount of violence.
Third, know the law in your area. Sometimes the law boxes you into a corner. You were in your yard with nowhere to go if the neighbor got violent. That's where stand your ground laws and castle doctrine come into play. In some states stand your ground and the castle doctrine don't exist within the law. Regardless, your first line of legal defense regardless of where you are is "did you try to walk away/escape the situation"?
If I am confronted while flying and approached, if the approachee is pleasant I might politely ask if I might finish what I'm doing and land my drone safely before we have a discussion. If beligerant, after an attempt at a kind reply, I might say "I am an authorized FAA pilot and you are disrupting a legal flight. You need to wait until I have safely landed before I will have any discussion with you".
Unrelated to the above: While this happened to the OP in his back yard, it could happen anywhere. In today's crazy angry world, situational awareness is key. Know what's around you, especially before you fly as your attention will soon be immersed in your drone and it'
s controller. If you're not flying, try to stay out of your cell phones until you're in a safe space. You are particularly vulnerable when you're in a "transitional" space (from your car to a building, building to building, waiting for a stop light, etc. Thugs no longer limit their attacks to poorer neighborhoods and are constantly scanning for their next victim. Dismounting soap box.