You can still fly and even hover over the property! we have pretty strict laws about this here in Cali, thanks to TMZ and the other idiots....
If you were directly harassing the neighbor, then your neighbor would have a case. They cannot claim that simply flying over the property harasses them. (unless they have some sort of medical thing or they have proof you do it unreasonably often and cause a hazard) maybe....
If you are FILMING and by doing so you make the neighbors property or their activities the main subject of your video, then your neighbor can sue under the right of publicity act......ONLY if you use the video for monetary gain.
You cannot film your neighbors activities while they are on their own property........That means PROLONGED FILMING...If you are simply filming as you fly over you are NOT breaking the law...even if you pause for a few moments and film something you are still good, as long as what you are filming is not the main subject of the video. e.g....If you are taking video of your family walking to the mailbox and you happen to get his home and activities in the background..So be it... He has no case and you have broken no law.
if you must fly over a portion of his property to film the walk well..The FAA says that's fine and your neighbor has nothing to say. Your neighbor could post a sign I suppose, but unless a judge of some kind approves it, its just a yard decoration.