When joining a world or server, memory gets allocated but never released. When rejoining the same world/server, new memory is used anyways. Sometimes the space used seems to jump back down a little but it does not cancel out the new allocation. The RAM needed by the game after a couple of restarts seems to exceed the number specified by "f3". This also seems to happen when rejoining the same server over and over again.
I was able to "reproduce" it many times (as in: over the past months I sometimes had to restart minecraft to keep it from using swap space )

Possibly related: MC-86925
Most likely related: MC-1628