I never re-format my SD card because that makes it much tougher to recover anything on it, in case you make a mistake and need to get something back. ("Undelete" utilities work better if the card has not been re-formatted).
The only time re-formatting is done is if you need to change the file system from FAT32 to eFAT or NTSF.
After copying the SD card's content to an external drive or other card, I simply delete the files I no longer need.