Employed villagers don't follow what they are supposed to do. For instance, they wander when they are supposed to work, and that makes it highly likely that they despawn randomly.