You don't always want to cut glare. Or sometimes you want to cut some, but not all. Or sometimes, you just don't HAVE any glare, so the PL is not necessary.
I'll never forget one of my favorite shots from a European trip where I had on the PL and had it on full (circular, turned exactly 90 degrees from the sun for full effect), but since I was pointed north (with the sun, nor glare in the image at all), the ocean in the foreground was black. When I got home, no amount of post-processing give me back any blues or greens to make the water look natural. It was not a good time to use a PL.
Also, it cuts light (more than the ND part of the filter). If you are not shooting in a condition where you NEED glare reduction, you might be cutting out too much light (slowing down the shutter speed to the point of getting unwanted motion blur).
I suppose if you are always shooting on a full sunny day, and you always have your craft oriented towards the sun (and therefore have glare), then you might just leave the PL on.
First and foremost, you are using ND filters, which do not affect the quality of the light at all -- it just reduces the amount of light. And the ONLY reason to do that is to control shutter speed (to get it to the appropriate speed to match your frame rate, if you are going for that classic cinematic look).
Using a polarizer is for a completely different reason. Assuming that you will always have both needs at the same time is not going to pan out.
You can actually get PolarPro that is just PL (no ND). It doesn't come in any of the set packages. I bought it for $30 on Amazon by itself, thinking I might need glare reduction at a time where I don't need ND light reduction (lower shutter speed). So far, that has never happened.
Chris