The fear is absolutely real.
It's good to practice, practice, practice what to do in the case of lost/bad signal, or if something else happens you didn't account for. RTH isn't your only option.
First day, I decided to take off (4 feet) and land.
First time, it worked great. Took off, landed. Yay!
Second time, it worked great. Took off, landed. Double yay!
Third time, it took off and shot up 20 feet into the air, coming back onto the roof of my house, and hovering. I was freaking out! My heart was literally pounding in my chest. Then, it started to rain a little. I was like: "What would Captain Kirk in Wrath of Khan do?"
I closed my eyes, took a deep breath, imagined where the drone was, thought in 3D (versus 2D), visualized its current orientation, took over manual control, slid it "forward" (away from the house roof), and once I could see it visually again, I landed it manually, holding the stick down steadily until the rotors stopped spinning.
I was done for the day.
That was literally me trying to do an automatic take-off and landing. The third time, I have NO IDEA what happened. Same take-off spot, same GPS signals, same procedure. Random things happen.
Needless to say, my backyard Nest camera captured the whole thing, and I look "hilarious" when I'm freaking out. Practice is the only thing that will give you confidence over time.