It’s only drifting right from take off. Not being able to calibrate your controller while in flight makes sense, but, swapping controllers in my opinion would eliminate my idea. There is what we call “drift” with faulty controllers…the spring on the port /starboard toggle could be that tiny...