Strange. I have 4 batteries and my RC is fine (50 or 60%) after 4 full flights. I'm using iPad Mini 4 since Sept. Before that I used a S5, Nexus 5x and Nexus 9 and never had this problem either.
I've used two Mavic Pro RCs, one from Oct 2016 and the other from Jan 2017 - both seemed to be about the same in terms of power. Certainly worse than the P3/P4 controllers but fine for at least a half dozen flights.
Are you flying in cold temps? I am not - it's always nice in California!
I have the EXACT same setup and my controller does not even last through two full batteries... I am in Colorado, we have similar weather to SoCal, and other than the elevation, I have no clue what it is. I have always thought the battery life on the controller was horrible, especially when the fly more kit came with more batteries than I could use on a charge of the controller... I wonder if I have a faulty unit?