From average joes.Spoofing is the easiest thing to avoid. Encrypt the control signal.
From the NSA, not so much.
Not suggesting the NSA is out there monitoring drone signals. Rather, that the tech to get through most encryption already exists and is not kept secret from other parts of our National Security apparatus; the kind of encryption necessary to secure these signals from the military is too heavy to use in this application. The computing resources necessary are not easily and inexpensively integrated in to the RC and the aircraft.
I've done some work in this area in my career, about 10 years ago.