I tried to combine diamond sword having Smite V and Unbreaking III with Sharpness V enchanted book.
Note: Smite V and Sharpness V are not allowed on the same sword.

The anvil behaves like combining the items:
1) The book is consumed.
2) The player level is consumed
3) The new sword have higher enchantment cost even if the description has not changed

Expected behavior: The combination being invalid, disable the anvil output. This is the behavior for other invalid combinations like applying Power to a sword.

Additionally there could be an error hint "Invalid combination".
We have already error hint "Too Expensive"