I quickly encountered an effect in my footage that is, as i learned by now, known as I-frame flickering. which, as far as i understand, occurs because the I-frames that occur every 8 frames get more encoder bandwidth, therefore have more fine detail than the 7 P-frames inbetween. So especially in dark areas or areas with a lot of fine detail (like gras or vegetation), the limited bandwith basically works like "noise reduction" in those areas. it decides to "drop" this fine detail to save some data/bandwidth.
So you get 1 "detailed" frame with all the fine detail, followed by 7 "noise reduced" frames with less fine detail. And our eyes perceive this as flickering.
With some help from reddit i found this youtube video that explains whats happening and how you can fix it: