Maybe worth investing in a dji remote phone holder that supports your phone away from where it is normally placed in the controller the prices vary but get a decent one
I have the same phone and I understand what your saying.
First I would make sure your screen is clean and dry. Next use slow a deliberate strokes and pull the phone slightly out of the left clamp before you try to swipe. Then push it back in once the app is open.