If your up a couple hundred ft , your not stopping the drift but if your just barely off the ground you might not be locked in to the GPS yet. We need a few more details to get you up an running .
Sounds like you are wanting to fight against physics.
Just as your car can't stop on a dime when you are cruising at 60 mph, the Mavic 4 pro will take some distance to come to a stop.
The faster it's flying, the greater distance it will need, and it's a faster drone than previous models.