VLC is pretty terribly optimized, on Windows try PotPlayer, it's better in pretty much every way and works fine with those heavy formats.
Works fine in Resolve Studio on Windows too.
Well, in this case it looks like it's your problem 😁
I have an M4 Mini to play with and yeah, same thing on that. Seems it's a format that doesn't fit within what Apple supports.
Time to get a PC I guess...