When holding either a tool, a none-interactive item or having an empty hand, the hand spam "hits" the fence without doing anything to it. This prevents you from using the left-click function of the item in your off-hand.