For some reason one of the parent villagers will leave right after having a baby villager. They will leave the building they are in and will run away. While they are doing this you might go far away from them and when you come back the villager will just despawn. I've had this happen to me 3 times now. So right now it's impossible to have a bunch of trading buddies at your base. Please fix this soon Mojang, thanks.