All villagers and every other mob that are kept safely in fenced villages are gone when I come back after a short absence. Tagged villagers included. After the same happened in at least 5 villages, I went exploring, found a new village and started working in it, same just happened: I left the area and upon return the village was empty.