Wind speed and wind turbulence aren't the same thing.
A steady 20mph wind is easier to maintain position than a 10mph turbulent wind.
Turbulence requires more adjustments per second.
I think you’re both right. It’s been my experience that there’s GENERALLY more wind at higher altitudes, but obstacles on the ground (like buildings etc.) can cause turbulence. Have a great day and fly safe.