Much/most of the world has strong biases against drones.
When shooting someone else's property, even a friends', get permission first. Since you didn't do that, he didn't know who or why, so naturally, he assumed the worst (see above comment about common bias against drones).
Often, when I meet someone with strong bias against drones and they're giving me their spiel on that regard, I find that I am getting ALL OF THE ANTI-DRONE BIAS this person has built up over time, shot at me in a concentrated stream of anger.
And there's nothing you can do about that in the moment. Maybe later, but right there and then, nothing. Certainly, you getting angry back isn't going to make it better.
Chris
Edit: so if we picture this scenario, you knock on his door or give him a jingle, and you ask "Hey, I'm a budding photographer. Can I take a nice picture of your house? I mean, for you? If I you like it enough, you can print it, free, as a friend. It would be from up in the air (I am doing some aerial photography these days) and I just thought you would like a shot of your beautiful property from that cool "in the air" perspective. It would give you the lay of the land. I think it would be great."
You might get a small rise of bias here, but nothing near what you did by simply buzzing over to his property.
PS: he was in the wrong shooting at your drone. In the future, after you're on good terms, maybe you can let him know about liabilities. Yes, even if someone is trespassing (not that you were), he would probably still have to pay for any drone he shot down.