I have a huge Nether Castle I built in the nether. Many rooms have paintings and banners. When I walk in certain rooms after a while, I'll notice a painting that I placed is no longer there. Once I re-place it, another one randomly disappears. I do not recall there being a painting capacity limit.