I went to visit the new Woodland Mansion, to see if this has problems too. It seems like grass can get inside the mansion, but not replace the walls. I've also seen andesite and stone at the lower level which were most surely from a cave that would be there if the Mansion didn't exist.