Couldn't replicate this outwidth the server, but has continued to happen in the server.

This is how they are laid out:

Roughly 15 of the 22 have escaped, some going back into the wrong minecarts or teleporting into the waterstream above.

I've only noticed this happen when leaving and rejoining the server (no other players in server at time).

In case it has something to do with the farm, the design is by Silentwisperer:

Villager Trading Hall Tutorial