Here is the deal with 400', General Aviation can fly as low as 500 feet in some areas, Helicopters lower than that.
The idea is never ( with some exceptions) to be where the airplanes are.
I can tell you that the chances of my seeing a sUAV as I am buzzing along at 120 Knts, or surviving it hitting my prop or windshield is not something I want to try.
Once you get up in the air, the winds do funny things, You may be near calm on the ground, and once you get up a few hundred feet you may get a wind that the sUAV may not be able to overpower to get back to you, or it may get blown off course with loss of perception as to where it is.
There are times where you are legally allowed to be higher than 400' AGL,and there are good reasons to be up there, and you may well be well out of the operational area of most aircraft as we need to stay a FAA mandated distance horizontally and vertically from other objects ( radio towers for example ) but then we get into that pesky VLOS thing. I have pretty good strobes that I snap onto my Mavic, and even with them ( 3 mile legal supposedly) it is hard to keep track of the Mavis when flying without a visual observer.