Barometers are not precise enough to give altitude tracking down to the precision the Mavic is able to obtain on their own. The Mavic uses all of its sensors in combination to get super precise measurements of it's position, including the barometer, GPS, accelerometers, thermometers and (when close to the ground) VPS.
When you take out part of that puzzle (GPS) you greatly reduce the accuracy of all positioning data, including height. Depending on the relative location of the satellites available, this will determine which dimensions are lacking precision.
The IMU estimates of position and velocity don't, primarily, use GPS or barometric data. In P-GPS mode, the IMU initializes position with GPS data and heading with compass data. It records both home point GPS altitude and barometric pressure (and, by extension, barometric altitude assuming a standard atmosphere) but it doesn't need those directly to estimate flight altitude.
Once moving, the primary update to location, velocity and attitude comes from the 200 Hz inertial data from the 3-axis accelerometers and rate gyros via quaternion algorithms, with the barometer (200 Hz), compass (50 Hz) and GPS (5 Hz) data providing low-gain adjustments to counteract bias and drift in the inertial data (aka sensor fusion).