I have been out of the low level game for quite a few years. When I was flying we flew low levels by using visual guidance based on a map on our kneeboard, along with a compass, a clock, and an airspeed indicator. We did our best to stay on route and on time, and we were pretty good at it. But people in training would often get off route.
I was once tasked to design a VR low level route. It was a pain in the a**. In war time, if we had a target deep in enemy territory, the only way to survive the ingress and egress was to go in very low and very fast. So we would pick routes we could easily navigate. For turn points, we would use bridges, very easily recognized bends in a river, water towers, easily spotted terrain features, etc. But when you design a route in the US, you can't often do that, because the straight line from the bridge to the water tower happens to cross 2 miles abeam a little airfield. And guess what. We couldn't invade their space any more then than I can with my MP now. So sometimes when we plotted out charts for the turn points on the map we found ourselves turning over a seemingly visually insignificant point in the middle of nowhere. Imagine you just flew 50 miles at treetop level and high speed to "nowhere", then you turn based on time to a new heading and fly another 50 miles hoping to find your next turn point where you expect it. We had no input to give us winds. It would be quite easy to get off course.
However, today, I'm suspect all military aircraft have advanced navigation systems which make this much easier. But there are also temporary operations where an entire area is open to low level ops. In those, we were generally limited to 300' in order to give room for the slow movers below. If there were no Helos, were could go to 200', and sometimes in the desert, to the deck.
And sometimes guys simply flat hat. But if a guy flies over my house one time at 50' and 500 kts, I'm going to give him a break. I don't know whether he had good reason or not. But knowing what he does for a living for me and my family, he gets a pass. If, he does it repeatedly, I might call the nearest military airfield and ask some questions, because he may well be a bad actor. But being spring loaded to the "I'm going to call the FAA and complain position", ,,,,, Well, if that's who you are, then that's who you are. And it won't keep them from risking their lives to defend yours.