I had this issue a few days ago while flying at night. 
I thought it could be because I was flying at night, but with a strong GPS lock I thought it should still be stable regardless of ambient lighting, so later that night I did a compass calibration and IMU calibration (my first one ever) and it has been fine since.  That should cure it for you.