Sometimes people feel that a drone in close proximity is somehow interested in them. Basically, stay away from people with your UAV unless they know and approve of your close proximity. One thing you should know and may share with those who are concerned about privacy is that satellite observations are much more intrusive than your drone. Firstly, you're only looking at the ground for a moment, but not so a satellite. Secondly, your images aren't parsed into data and shared to, who knows where?
Lastly, note that the airspace above the USA is almost exclusively the purvue of the FAA. Except for local privacy laws (which are vague and nondescript in most places) you can fly wherever the FAA says its legal.
btw, we don't have laws in the state of Hawaii, protecting people's privacy by way of UAV observations at public places.