Removing smelted items directly from the hopper or a chest fed by the hopper give zero XP.

All uncollected XP appears to be given when smelter is broken with a pickaxe.

Issue has been replicated with all ores and food items tested.
Issue has been replicated in multiple worlds.
Issue has been replicated with a blast furnace smelting into a hopper.