I did some tests for another bug/thread where a person had no Haste effect being applied, well i found out that Efficiency 3 with regular Haste has no noticable increase in speed from just Efficiency 3, the same goes for Efficiency 4 with Haste 2. Might just be me though. I would say give it a test. I also wanna add in that going from Efficiency 2 with Regular Haste to Efficiency 3 seems like its the same speed, also this happens with Efficiency 3 with Haste 2 to Efficiency 4

From Thread MC-135207