Hi!
Yes you can swap back and forth & there’s no limit to the number of times you can swap.
Just remember that you will need an internet connection to actually bind the remote to your aircraft and DJI account to ensure you’re covered if you have a flyaway.
This is why you can fly 5 times with an unbound remote, you may not have internet access, after that you won’t be covered for a flyaway, until you either get internet access and bind the controller or you switch back to the currently bound controller.
Hope that makes sense!