You are in the US so there are two issues. One is the make/model of your UAV. If it is a DJI product, the software may control your UAV and not let you fly within the geofence area without an authorization. If it is not a DJI product, then you have a legal restriction under the FAA as the second issue. The FAA requires you to follow the rules under your local AMA or other organization as a Hobbiest, or have a 107 certification if you are flying commercially.
The hobby flyers that can't take off at 5 miles from the airport most likely have a DJI or other model with geofencing built into the firmware. The 107 pilot that flies 4 miles from the airport under 400 feet most likely went through the proper steps and has authorization to fly there and that authorization provides an unlock to the geofencing.
I see your tag says your FAA Licensed (should be Certified, not licensed) so you should have studied this information to take your 107 test. If you are within the 5 mile marker, depending on the airspace classification and whether your hobby or 107, you will have to take the appropriate steps to unlock your UAV, and or get the appropriate authorization.
Good luck and fly safe whichever route you take.