My villager trading hall has them trapped in minecarts which are held back by iron trapdoors. I came over to the hall once and 4 of my 13 villagers were just out and about roaming around. Their minecart was still in place though. I managed to get them back in their spots but at some point later in the day, it happened again.