You’re not the first, this is pretty common. I have the police come to my house over this a couple times a year. If you’re not breaking any laws you have nothing to worry about. Have them prove you are invading their privacy.. and ask the police for evidence that it’s your drone and freely offer up your flight logs if there is any concerns.
Agree, The police are welcome to stop whether your drone or anyones; you, nor Police don't need to prove it's yours when no law broken. I've had them stop a few times and just let me know a neighbor had called them; they normally tell the caller if they aren't below roof height they technically aren't breaking any law and not trespassing. Generally, roof top level or highest personally owned object (tree) on property is viewed as personal property. Although, being aware is great too, I appreciate when the police stop by to let me know there was a complaint. I'll normally attempt to avoid any ill-feelings and make an attempt not to Fly over low but I'm not overly concerned if flying over at 300' AGL.
I often test fly or experiment in my yard everything from a Mavic up to the large M600Pro Hex and S1000 Octal. I welcome the kids, and have Tello & Spark for them try flying. This too helps with neighbor acceptance, even if not their kids when others drive by an see it's all friendly it helps with the social environment. The Supreme Court ruled years back there is "no reasonable expectation of privacy" in your backyard or property, and can be applied to aerial view. Aerial view is very different from invasion, spying or peeping tom, etc.
Several communities have attempted to pass local ordinances on "NFZ" and have often been challenged & overruled; requiring the FAA to establish NFZ, not a community or local law enforcement. Most law enforcement acknowledges the FAA as sole authority on airspace and at most will "request" you observe your social neighborhood but at the same time will normally inform the complaint there is no law being broken... as long as you observe & avoid laws governing various peeping tom or trespassing regs.
I often see a new drone Owner stand on their rear deck and fly up the property line between back yards at below or above roof level. That isn't respecting the neighborhood, and encouraging complaints and frustrations. I normally attempt to fly the center of the streets, and avoid any back yard pass overs (Unless performing a realty project).
I wouldn't be overly concerned with unreasonable neighbors; you'll have those walking a dog, mowing your lawn different, working on your car, etc. As long as you're clearly not in their personal airspace and not hovering for long periods... that tends to agitate easily. I'm always receptive to wave, chat or discuss... and I'll offer to help them get into drones too if they wanted or take an aerial of their property: 360 Pano or Landscape scene. Actually, I've found the complainers are either the totally irrational type or the type that have a concealed passion of interest. I've got 1 in my neighborhood that drives by and shares his own flavor of "Bird", I just nod, smile & wave!