Correct - it cannot hold position if it doesn't know the direction it is facing because it doesn't know the relationship between the earth frame of reference and the aircraft frame of reference. GPS data will tell it that it is drifting (in the earth frame of reference) but without the compass data it doesn't know in which direction to apply thrust (in the aircraft frame of reference) to correct for the drift.
The problem described by the OP is more likely where it thinks it knows which direction it is facing, but is wrong. In that situation its attempts to hold position result in applying thrust in the wrong direction to oppose drift. Depending on the error in the yaw value it is using, that can result in toilet bowl motion (for errors around 90°) or linear flight in the wrong direction (for errors nearer 180°), or any combination of the two for intermediate values.
