In nether portals, the item in your hand or offhand is sometimes blocked by the purple block around you combined with the portal's nausea.  Also, when I was standing in the portal and tried to open my inventory to put the totem in my offhand, it didn't open.