Meta4, I tried looking for your past excellent post regarding how the FC, IMU & compass get initiated and interact properly upon startup after the required number of GPS satellites are obtained....I couldn't find it.
I think if you reposted it...
That's completely normal.
You'd only need to recalibrate the compass if you add or remove accessories to/from the drone.
(That's accessories that could have an effect on the magnetic fields of the drone.)
A GPS has no way to sense direction...
This sounds very much like a yaw error and it should be easy to confirm if you post your flight data.
There are a couple of options ...
1. Go to DJI Flight Log Viewer | Phantom Help
Follow the instructions there to upload your flight record...