I would call this very narrow, self-centered thinking.
What you either fail to account for, or dismiss as irrelevant is this: In the view of many, there are many places where it makes sense to prohibit you from flying, but you can.
Once you grant other's differing view the same legitimacy as your own, compromise can take place.
For example, a public park is there for everyone. Consider a park that has a baseball field and a soccer field. If some people are playing softball on the soccer field because the baseball field was already in use, it is reasonable to me for a crowd wanting to play soccer to ask the softball folks to yield.
Or if I was flying my drone there. It's a soccer field, not an airfield. I should yield.
If there is a place in a park prized by the community for peace and quiet, drones might be banned. No problem. That's what compromise is about. Even though it might also be a killer place to fly, the broader community may have a different idea and through the democratic process we come to some solution without people bleeding.
Outright prohibition is easy, and the first thing most authorities reach for when there's conflict. But it's not the only solution. Specified times to allow drone flights – every second weekend of the month, for example, can be implemented. However, as the tiny minority in the community, it's us that will have to suggest such compromises, and fight for them.