Looking directly at villagers causes choppy frames and very low framerate. As soon as you take the crosshair off the villager/ turn the other direction

PC Specs: 

8GB Ram
Graphics card: Nvidia Titan V
Intel gen6 core
World size: 2.6 mb

^{(Had to provide a google drive link as the file size was larger than 10.00 mb, in the comments)}