Villager won't change profession even after I broke his worktable.

I brought a villager to my base, and he accidentally used one of the blocks I placed in my house as a worktable. I decided to ignore it and bring it out of my base, approximately 50 blocks away. After I got him out of my base, I went back into my base and broke the worktable that he had accidentally used as his profession. He did not change professions. I never traded with him. Is there any way to fix this?