The Micro USB port is more flimsy and more prone to damage and dirt than the USB port on the bottom of the controller. I only use the micro USB for charging the battery.
As @Qoncussion said, you can use contact cleaner and maybe a Qtip to get into the smaller port.
I use a can of compressed air to clean my remote controller ports. The cans that include the plastic tube extension are best for getting into those small areas. You can find them in your local office supply store or here on Amazon.