I don't tend to use the small indicator much but yesterday I had a play about, and came to the same conclusion: calibrate phone compass with it latched in to the controller.
With everything connected, drone hovering, good GPS lock on phone and drone the indicator was about 95deg out to the actual drone position. With everything still "live" I tried waving the phone/controller in a few figure-of-8 patterns and it suddenly improved a lot, with the indicator displaying correctly to within about 5-10 deg. I kept the drone hovering, with no stick input throughout the whole procedure, so the only thing that changed was the phone compass calibration