It’s great that you’re asking this question, it will make you so much better a wildlife photographer!
Basically, if you or your drone alters the natural behavior of a protected animal in any way, it’s too close. In the US under the US marine mammal protection act, the US Fish and Wildlife service recommends no closer than 1000 feet from marine mammals for drones, and the migratory bird species protection act (which is an international agreement and law in many signatory countries) protects raptors and all other migratory birds from harassment (take), which includes making them fly from nests, stop eating, etc.
As a hobby wildlife photographer (also while working as a wildlife biologist in my previous career), I follow this code of ethics whenever I go out to photograph wildlife:
www.naturephotographers.net