I had this issue a few days ago while flying at night.
I thought it could be because I was flying at night, but with a strong GPS lock I thought it should still be stable regardless of ambient lighting, so later that night I did a compass calibration and IMU calibration (my first one ever) and it has been fine since. That should cure it for you.