Endermen will only become hostile when looked at for over a second or so. This makes this feature practically useless, as if you accidentally look at one, you have time to move the cursor off.