Yes of course, but three miles seems extreme. If you live a couple miles from a stadium you certainly should be able to fly around your house. Its not like an airport where something can leave the stadium and cause a problem.No same goes for sports stadiums. The biggest stadium in my home city has a geo fence meaning you cannot fly anywhere near it.
Makes sense - you don’t want a flying blender landing on one of 60,000 people potentially.
Common sense should come into play somewhere.