So my friend was messing around with Resistance IV mobs today (he was curious if it was actually a 100% resist as opposed to 99.999999% resist. Resistance IV and up, for the record, makes mob, invincible, so to speak.) He tested it by using the new anvil like damage tags for a FallingSand spawner to do about 2 billion damage in one shot.

When he dropped it on the resistance IV mob, it stayed at 20 health and nothing special happened. Just for giggles, he dropped it on an unbuffed mob, no extra additions. It sort of broke the damage formula, and the mob ended up with 30,000 health somehow. So, 20 - about 2 billion = about 30,000 apparently. I'm assuming this caused an integer rollover. I'm also assuming this happens because Mojang didn't foresee anyone causing that much damage to a mob.