This might be intended, but if you have a weapon or tool with attribute modifiers, your attack speed and damage are reset to the default 4 attacks/sec and 1 attack damage. For example /give @p diamond_sword{AttributeModifiers:[]} will give you a diamond sword that does 1/2 heart of damage and let you attack 4 times a second, like your bare fist, with the exception of a sweep attack.