Spoiler to Add-on.
When I put minecraft:on_interact component to block and made it triggers event that remove interact item with clear command, but it doesn't work.
But the item is wiggled in hotbar.
(clear @p bread 0 1)
And it only works when run command on command block or chat.

I tried decrement_stack but, same. (doesn't work)