The two issues are 1) the ownership and rules where you take off/land, and 2) the rules governing flight. The farther you can keep from people, the more likely you won't have problems. Some people are doing dangerous or intense activities and they don't want the distraction of the drone. If someone asks me to leave or to stay farther away I will. The new zoom feature on my MA2 makes it easier to film from a distance.
I'm not sure if you're asking questions or not but I'll answer them just in case.
1) Owners (& Security Guards) are definitely allowed to ask you to leave their property and forbid flying from it. And they don't even need an excuse.
2) Rules governing flight are dependent by why you're flying. It sounds like you''re flying hobby, which are technically called Recreational Flyers by the FAA. You can find those here: Recreational Flyers & Modeler Community-Based Organizations
Rules for flying anything that falls outside the recreational flights will be under 14CFR Part 107 rules. You can find those here: Certificated Remote Pilots including Commercial Operators
Both of those FAA pages give you the summary of each type of rules. I also have an article that explains the difference in language easier to read than FAA website. Hobby v. 107: Unscrambled
Hope that helps.