The reason recreational drones are limited to 400 feet AGL, is that manned aircraft are normally limited to 500' and above when overflying people or structures. The 100' separation between those activities is a buffer. In non-controlled air space with no people or structures, manned aircraft can fly right down on the deck (as I often do with my Husky).
What's worse, a mid-air between two drones operating under 400 feet, or a drone above 400' hitting a manned aircraft?