They think the same way, if your standing in plain site or not. In fact, I believe the pilot is perceived as less of a concern, if their is no drone hovering nearby them. But when someone sees the drone and thus knows your the one controlling it, game over.
When your alone with your remote and they can't see or hear a drone, they don't know what your doing and will likely ignore you. They may think your just playing a video game or something. But if they see you and see the drone, their imaginations go wild. And it's usually not in a positive direction.
It's not like we "hide", the way you think. If I fly on the beach, I might just stand on the sidewalk near the beach or the sidewalk across the street from the beach, but not on the beach standing their with my remote for all to see. Other times, I just pick a parking lot near the building, park, nature park, beach or other scenery I want to shoot, but not in the area itself. We don't hide, we just move to a less conspicuous place. A place where the drone will be seen, but not where we will readily be seen flying it. If someone sees us, they likely won't have a clue what where doing, because they probably won't be able to see or hear the drone flying in the VLOS radius where flying in. The only time we are seen with the drone is when it's landing near us or taking off. The rest of the time we appear unrelated to the drone and thus, aren't targeted as the "nefarious freaks" the news cycle and society has caused most of the public to perceive us to be.