The problem is extremes -- neighbors who expect that only their desires regarding the neighborhood, and the desired use of the airspace by the drone owner don't matter; or, attitudes like yours, that completely dismiss the reasonable expectation of golfers -- on a golf course.
Who cares? I do. You should too. That's how we live peacefully in a society together. Since the purpose of a Golf Course is facilitating playing golf, the owners/developers invest a huge amount of resources into creating and maintaining the course; patrons spend considerable time and money on the pasttime, dismissing golfer's concerns about what's going on over their course is simply antisocial and juvenile.
So for you to come along and expect that it should be available for your drone flying pleasure, the golfers be damned -- well, you are the problem. You are our problem.
You are the problem because it is people like you, with attitudes like yours, that are going to provoke society to constrain MY flying options, and MY ability to enjoy this hobby, even though I'M more than willing to consider other's needs and work with them, rather than spit in their face.
Yes, I do care what golfers WANT to hear while they're playing golf on a golf course. I think that, in a friendly, peaceful, cooperative culture they have a RIGHT to some peace and quiet in that setting and activity.