By simply observing existing FAA guidelines as you should be whenever flying you will almost inherently not be invading other people's personal space. If you are, then there's a good chance you're flying someplace you really shouldn't be. That's one of the things covered by the regs and automatically included in the flying guidelines.
One of the great things about the Mavic is it IS so light and small and unassuming that it is not too imposing to non-drone spectators or passerby. So far, the people I have run into while flying around their communities have been nothing but enthusiastic and inquisitive about it after seeing it going by. After flying it around other people's areas and usually striking up a conversation about drones - which quickly takes away valuable air and light time - I will sometimes just hover it for them then send it spiraling quickly skyward, which a lot of people seem to like. I even let a polite old gent who followed me out in a field once take the sticks for a couple of minutes while in Tripod mode when my Mavic was way up there and safe away from any harm, just to see the smile on his face that he "was doing it!"
Again, and it's probably not a popular thing to say or even politically correct nowadays to express this, but I think a lot of how other people take you as an "interloping" drone pilot in their neighborhood has to do with how you handle and conduct yourself, how you are composed, how you appear and yes, even how you are dressed, down to the language you choose to use when expressing yourself around them. I mean, you ALREADY appear unusual to them carrying your drone, backpack, and controllers in your hands - not to mention perhaps even
goggles! - while traipsing through their environment, so doing your part of fitting into their surroundings never hurts either. Fact is, most people today ARE tech savvy to some degree and expect to see the unusual on occasion. As long as you show most folks their due respect and consideration, the only people that will still give you trouble are generally the ones who are always giving someone a hard time about something anyhow.