This is very similar and interchangeable with drone anxiety, and there's another thread of that name on here recently you might like to read. (A search will find it I'm sure.)
Nothing wrong with your approach at all, in fact it's that sort of thoughtfulness towards others that is good for drone pilots to have.
The number of times I've gone down to some scenic spot, at the beach etc, to fly a sunset, and end up not flying due to the crowds there also enjoying the space.
Rules aside for OOP here (strict !), I simply don't wish to impede on others peace & quiet. At all.
I generally fly remote areas, where it's harder for the general population to get to anyway, but I think I will continue to fly when it feels 'right' when around others.