If you fall outside of 336, you are now under the FAA's authority to regulate and that is what they've done with Part 107.
The part you don't seem to understand is that "flying for fun" doesn't exist. You don't self-declare what type of flying you're flying. What type of flying you're doing and whether you fall under FAA regulation is determined solely by whether you're within the bounds of 336 or not.
It now seems to be coming into focus. The FAA is given the authority to enforce protection of the NAS, but also given limitations to their rule making ability in regards to model aircraft.
I agree that outside 336 puts you in FAA authority. And breaking rules or stipulations (i.e. endangering NAS) also puts you in FAA authority. The part you're not understanding from me is that according to the FAA, you actually do declare what type of flying you do.
And since they are the arm that governs our hobby (and our livelihood if we do this for business), I think this is critical information to know.
So if you say you fly within stipulations of 336, the FAA has no jurisdiction. Maybe true. Doesn't really matter because the FAA already recognizes you as a hobby flier. And everybody is happy anyway. But if you fly outside of 336, then you say you become a part 107 flier, correct? So what would that mean? If you break a rule or stipulation, you are subject to FAA punishment. This would happen regardless of your designation (or non-designated 336 exempt) if you actually created an event that endangered NAS. Most basically, to be a part 107 flier, you have to apply for and pass a test to become one. The FAA goes out of its way to make a distinction between the two types of fliers and the rules by which they fly.
The distinction is further exemplified by looking at the chart closely. If it was a rule to fly under 400', it would have been stated in the chart, just as it was stated in the chart for commercial fliers. Same for night flying, etc. There is a definite reason for making the distinction.
On the subject of the AMA:
As far as AMA goes, there is this guidance publication (Note that despite the recommendation to fly below 400' on p.1, every flight code the AMA indicates is actual requirement is cited in the Safety Code at the end of the publication. There are multiple references in the document that explain the 400' altitude limit and when it applies, just as the official Safety Code).
See particularly p.6, and also p.8 and 9 which applies to stabilized aircraft:
Can be confusing, be certainly not as black and white plain simple English as you might think.