This is exactly what mine has done. It's been a really great aircraft, but the yaw issue has gotten progressively worse for mine, much as you describe yours, except I can no longer fly mine at all, because the yaw issue is more continuous now, and kicks in during flight whether simply hovering, or even during forward flight, it will gradually yaw left. Pretty sure it's always been left. It can be countered with right yaw, to yield somewhat forward flight.
I have some insight for you though, and it's that I stopped updating my firmware after the first time, because the aircraft flew absolutley perfectly, and had plenty of brilliant features to keep me busy for a long time. Yet, the yaw issue appeared one day, and has gotten progressively worse over a year and a half or so. This leads me me to believe there is a hardware issue, possibly a break down of gyro data. Maybe the gyro itself is failing, or maybe noise is being introduced somewhere along the path.
My next step is to get my Air replaced through Care Refresh.
(If any of the ever helpful forum members has something to try, feel free to suggest it, as it may help others, now, or at another time, but Ive completed all possible actions that Im aware of, that could fix the issue, and the history of the problem overwhelmingly suggests progressive hardware failure, imho.)