This is not controlled by your shutter speed, but by your frame rate. Shutter speed only controls how crisp a single frame appears, frame rate controls the motion between frames. To ensure a wheel continues to keep turning (in a forward motion), the frame rate must be
faster than the wheel rotation rate.
But wait! There's more... Many car wheel are actually displayed in a repeating pattern. This means that the frame rate has to be faster than the repeating pattern (or a multiple of this), not the entire wheel rotation. Or written another way, the time between frames must be less than
half the time it takes the wheel to rotate so far as the pattern begins to repeat (usually when the following spoke is in the same position the last spoke was in the pervious frame). The reason that it is half the time, is because if you get close a full pattern rotation then the wheel begins to look like it's rotating backwards.
But have no fear, this can all be worked out beforehand you just need to know a few things first:
Diameter of the tire (not the wheel, but the whole tire)
How many times the pattern repeats on the wheel (usually how many spokes there are)
Speed of the car
Frame rate you will be shooting at (we will assume 30fps)
~~~~
In the example below I will assume that the tire diameter is 17" and the pattern repeats 6 times.
Let work out the tire circumference:
pi x 17 = 53.4 inches
Now let's work out the repeating pattern distancing:
53.4 / 6 = 8.9 inches
Divide this by two so that the wheel moves forward in motion:
8.9 / 2 = 4.45 inches
So now we know that we have to take a new frame before the car has travelled 4.45 inches. So since we're shooting at 30 fps:
4.45 x 30 = 89 inches per second = 5.1mph (8.18km/h)
And there you have it! For a 17" diameter tire with 6 spokes, you must stay below 5mph to have the wheel rotating forward.
But that's not very fun is it? 5mph is far too slow!
~~~~~~
Let's go faster!!! We don't care if the spokes rotate more than once between a frame, so long as the next frame
appears to show a following spoke, it can actually be any spoke!
So here we can get into multiples of the same speed. So here you can just reduce the number of repeating patterns on the wheel all the way down to the top speed the car can be going of ~45mph (72.5kmh), where the entire wheel almost rotates before the next frame. But we can go faster still, why limit ourselves to one full rotation, we can go hundreds of rotations between frames!
tl:dr
Really it's all about timing your frame with the position of the wheel. See the above formulae to work out at what speed your tires will align well with your frame rate and enjoy the maths & science!