Golems attack player for no reason, no need to hit the golem or villager, it will attack anyway. If difficulty is changed to peaceful, the golem will still attack you but do no damage.