Villagers as well as baby Villagers behave very strangely and sometimes either ignore players completely or looks in certain directions where there is nothing. This was tested in a world with no entities except for the Villagers and a few armor stands (which they did not look at). Some riding armor stands, others having generic movement speed set to 0. Villagers tracking also stops when motion is applied.