It’s hard to say without more information.
At low height the internal barometer would not be accurate enough to maintain the height at which you stopped and would be using the downward facing infrared sensors to detect height by bouncing the light off the ground’s surface. How accurately it does this can depend a lot on what the surface is made from and how reflective it is.
The height information from the sensor is fed to the flight controller, which is then used to maintain height by controlling motor speed, while combined with GPS data and motion detection (3-axis) by the Inertial Measurement Unit. The IMU by itself cannot detect height above ground and GPS altitude data is nowhere near accurate enough to maintain static height.
So the cause could be varied. There is no need in this case to calibrate the IMU or Compass.
Does the height drop repeatedly?
Does it do the same at higher levels such as at 10, 20, 30, 40 meters?
What type of surface is the aircraft above when this occurs?
Is the issue the same regardless of the surface type?