When I was shooting villagers with spectral arrows so I could stalk them freely, sometimes the arrows would just bounce off, or disappear but the villagers would still make the hurt sound, while not getting the glowing effect.