While you may think that, the fact is the FAA will NOT see it that way. Do you want to take the chance? Oh, that guy on the bicycle coming past me scared me, think I'll whack him with a broom. See if that defense works in court.
Drones are considered aircraft and rules fall under 18 U.S.C. § 32 (
18 U.S. Code § 32 - Destruction of aircraft or aircraft facilities). Thus trying to knock the drone down is a FEDERAL crime. On top of that, you could be charged with willful destruction of private property as well. So my advice stands...do NOT attack the drown unless you can prove you are in physical danger (i.e. it's diving directly at you), then self-defense might be a matter, but self-defense would require proof that you were in reasonable danger, just not liking the drone there is not self-defense.
I would again, keep calling the police, file harassment complaints with them and the HOA. And consider suing him in civil court. What he is doing IS harassment and should be dealt with as such, but do NOT do something that could end up with YOU being arrested by retaliating.