You can argue ad infinitum about the fraction of manned flights that require operations below 500 ft AGL, but the fact remains that such flights occur. In a city environment, disregarding approaches to airports, it's mostly going to be LE, medical, heliport transfers and news helicopters. So how would your proposal work if there are already manned flights in that airspace? You can't exclude them. Suggesting that they be required to get waivers is, firstly, redundant, because they are already most likely operating in controlled airspace under ATC control and, secondly, pointless, because that achieves nothing since recreational pilots are not going to be getting guidance from ATC about authorized manned traffic? Suggesting that they bear even some responsibility for avoiding small UAVs is unreasonable, given that drones are small, difficult to see, and not broadcasting position data. It's also rather ironic in view of how insistent most of the drone community seems to be that pilots of manned aircraft cannot possibly even spot a drone when flying.
Commercial sUAS delivery systems almost certainly will be required to avoid manned traffic rather than the other way around. That will be relatively simple to achieve once all traffic is using ADS-B or equivalent.