EDIT: I've tested it multiple times again in the new pre-release and it seems to be fixed, or caused by something else I didn't think about prior to reporting this bug. Please ignore this bug report.

When standing near the wall of a cave, you'll notice while moving your head around that there's a sharp decline in rendering performance, leading to highly unstable framerates. I've compared this to the performance I get while looking at caves in spectator mode, and it is indeed quite similar. Although I can't be 100% sure this is the exact cause of performance dropping in caves, I'm pretty sure this is what's going on.

Steps to reproduce: Go into a cave, and stand next to a wall. Then, look around while facing the wall, and notice how performance/FPS drops.