help driftting

Hey guys im new to the mavic and on my 2nd flight noticed the bird drift to the right when i let off on the stick. My question is if my table was off slightly from level would that throw off the imu and make the mavic drift? Or could it be my controller? And also wanted to add i was at a soccer...