I'm really new in it, only a few hours all together so I didn't have the chance to meet and discuss any bystander's feelings but these are the most obvious reasons for hate:
1. Fear of the unknown as it was mentioned earlier.
2. Possible spying activities.
The main difference between quads and security / surveillance systems that the latter ones are controlled and managed according to the official policies. At least that's what most people would like to think. On the other hand most these activities are well hidden.
Even if you notice a surveillance act on your environment, who would you trust more? Some nerd looking geek who has no life for sure or someone from the Great Authority?
Cell phone and other cameras are other topic - you obviously see who uses them and have the chance to ask questions.
With quads you have good chances that you don't see the operator - he/she has the chance to hide -, buzzes like large bunch of angry bees (obviously you only recognize quads when you're hearing them), capable of seeing through the walls and a dark background community uses them to take over the world.
What other reasons would be possible? If they think they're normal why these quad operators are not spending their freetime in pubs / malls like ordinary people?