Part of the reason for the VLOS requirement is for the same reasons we have NFZ and RID. People would be flying everywhere and without control.
The risk is slight, but not zero. Back in 2017, a guy flew his Phantom 4 BVLOS and it struck an Army helicopter, causing damage to the main rotorblade...