As you mentioned the CAA later in your post, I’m guessing that you live in the UK. The regulations say that:
“Low flying
In general, except when necessary for take-off or landing, an aircraft should be 1,000 ft over a built-up area or otherwise 500ft from people, vehicles, vessels and structures. Note that the legislation does not mention animals.”
So if the aircraft isn’t over a built-up area or within 500ft (150m - note this is a distance, not a height) of people, vehicles, vessels or structures, it can theoretically be below 400ft and remain within the regulations.
Quite correct, I'm from Wales and we often flew out on a day trip in a flight of 3-5 to somewhere in north Wales on the coast. Although we might be at a reasonable altitude over the valleys, we would some times fly over a mountain and the terrain below us would suddenly rise up of course. We did not bother to climb to maintain 1,500ft above the ground, we just flew a straight course.
When flying over such terrain you also have no idea if someone has built a house on the other side of a ridge just below the top. Therefore, a pilot may unintentionally fly right over that house on a trip somewhere. Seconds later that pilot would be well above the ground underneath them.
When I lived in Colorado, one dead calm late Autumn afternoon, I left Steam Boat Springs and headed home to Granby. I climbed to altitude to miss all the mountain peaks on my course and just settled in for the flight. Way off in the distance was once of those perfect textbook drawn mountains that went up perfectly to a nice sharp peak. It was dead ahead and kept watching it for quite a while as it did not seem to get closer. Then as I approached, it finally began to look like it was getting closer. Then it was speeding up a little as I plodded on. I was deciding whether I would need to add some throttle or just change course slightly to skirt around it.
I kept watching it and all of a sudden it came close very quickly, but I could easily determine I would be clearing it by about 50ft or so. As I mentioned it was one of those perfectly sharp pointed mountains that a child might draw, so it would be no trouble to just divert left or right around the peak. I stayed on course, and it just passed right underneath me and in an instant, I was 500 then 1,000ft above the ground again. It was a beautiful sight and feeling as that peak came right up to me. There was fresh snow on the peak and I could see the way the shrubbery on top had been wind blown to one side.
So yes, pilots do pass low over mountain or hill ridges from time to time, however, depending on wind conditions, passing low over a hillside ridge or worse a mountain ridge, things can quickly become dangerous, due to what the winds are doing over those ridges. Generally in the mountains you want to give yourself a minimum of 1,500ft over the highest point of that ridge to be safe from adverse effect from and wind present. Therefore, it is not very often that a pilot will choose to fly very low over a hill or mountain ridge.
And if anyone is flying their drone in such an area, if should be a piece of cake to just drop down to one side or the other of that ridge top to get out of the way. Therefore, there is no need to blow these scenarios all out of proportion. And again, I am not speaking of fighter pilot training or nape of the earth military helicopter training.