You are allowed to fly over, but you must obey all applicable laws.
Bill 156, for example, makes it an offence to interfere with a farm animal*, and of course the usual rights to privacy and peaceful enjoyment of property apply too.
I've got some lovely photos I've taken near Claremont, but there's a couple of farms I'm avoiding because they have animals, and I know how skittish they can be. I'd love a picture with the cows or sheep as more than dots, but not at the cost of spooking them. I'm not worried about the law, I just don't want to interfere with someone's livelihood — being a small farmer is hard work, and they don't need a city-slicker making it harder. Some animals are placid, some are high-strung, so I want the farmer's permission before flying close. At 100 m I'm not worried, but I'm not flying 5-10 m above a herd without asking first.
*The purpose of the bill is to stop animal rights campaigners from going undercover to gather information on mistreated animals, but as written it also includes things like giving an animal in a stalled transport truck water ('interacting' is 'interfering') so a drone spooking animals could also be construed as interfering. And as my friend the lawyer points out, ending up in court is expensive even if you are found not guilty.