I've had 3 flights with my Mini 2 SE so far, is anxiety when flying normal for new pilots. I'm obviously still getting used to the controls and hoping it will pass with time.
Anxiety when flying is normal, most new drone pilots are more or less nervous.
Read the manual twice, fly in an open area and practice some flight manouvers like circle and "number 8",
Soon you will feel more and more confident!
When I started I was always worried about crashing or losing my Drone. but then I learned the only way to get better is to forget about the Drone sometimes and just FLY. Your anxiety will pass and you may find yourself like I did, wanting more excitement. and getting more Drones. That's when you know your ready for the goggles!
A bit of anxiety is a sign that you're paying attention and concerned about flying well and safely. If it's overly severe, it's a sign that you're doing something you probably shouldn't be doing. It will gradually decline as you fly more, until you get too comfortable and eventually make a significant mistake. I had a flight instructor who warned of that happening and said that if that mistake didn't kill you, it would make you a better pilor.
Don't push yourself. Take your time and build some hours on the controls in a wide-open area without obstructions. As @waynorth suggested, practicing maneuvers is very helpful. Have a look at the Pilot Institute's free online videos on your specific drone model and the ones on basic drone flying skills.