From personal experience, I live near the Snake River and there are places where the canyon is a sheer cliff 450' or more down to the river below, but a few thousand feet wide. When I fly off those cliffs, I generally stay within a couple hundred feet of the canyon walls as I descend to a couple hundred feet above the river (-200 on the altimeter). At the same time, there are places where I launch from the river bank and fly up the cliffs or surrounding hills. Again, I stay within a couple hundred feet horizontal, and do my best to avoid being 400' AGL.
I don't think the rules are to keep you from flying in unusual places, but some common sense application is needed. Keep in mind "ground level" is pretty well defined. At the top of a cliff the ground is, well, the ground level. At the bottom of a cliff, again, the ground is the ground. Staying less than 400' above seems pretty easy except on the slope. And I doubt any FAA person is going to ding me if I incidentally exceed 400' as I descend from one ground reference to the other IF I am staying close to the slope - like within 400'.
I don't think the rules are to keep you from flying in unusual places, but some common sense application is needed. Keep in mind "ground level" is pretty well defined. At the top of a cliff the ground is, well, the ground level. At the bottom of a cliff, again, the ground is the ground. Staying less than 400' above seems pretty easy except on the slope. And I doubt any FAA person is going to ding me if I incidentally exceed 400' as I descend from one ground reference to the other IF I am staying close to the slope - like within 400'.