It's still only a guideline. For example, speed limit (white sign in the States) is a law, if you drive faster, a cop can give you a ticket. There are also yellow speed signs, e.g. before sharp turns or congested areas - these are recommended speeds, in other words, guidelines. You can drive faster and cops won't give you a ticket (as long as you're not faster than speed limit).
400ft guideline is like that - "please, as much as possible, fly below 400ft, it helps avoid accidents" (just like driving slower in a sharp turn does).
There's lots of "white" space in the States and elsewhere in the world where there's absolutely no, zero, nada air trafic, and flying as high as you want is absolutely no, zero, nada problem to anyone. I know because I fly a lot in the desert in the US West.
All known pidiot accidents involving manned aircraft were caused by flying in congested airspace, not by high altitude. I've lived in New York City for a long time and know well that helicopters fly below 400ft all the time and can appear at any place at any time, so a drone below 400ft can, and has, cause an accident.