Note: I'm not a native English speaker.

----------------------------------------

What I expected to happen was...:
javaw.exe should not exceed the memory limit imposed with the -Xmx flag.

What actually happened was...:
javaw.exe uses MUCH more memory than specified with the -Xmx flag, significantly exceeding the imposed memory limit by several gibibytes.

Steps to Reproduce:
Unfortunately, I can't think of a step-by-step list to reproduce the issue, but I believe the information supplied in this bug report should be enough.
I will try and do my best to reply with more information in a timely fashion when requested (school sucks).

----------------------------------------

When playing Minecraft in multiplayer, javaw.exe will use more and more RAM progressively (note that I didn't monitor the RAM usage over time).
After a long session, I notice javaw.exe uses WAY more memory than specified using the -Xmx flag.
This causes excessive thrashing, slowdown, and forces me to close background processes and long browsing sessions in Firefox (and I'm a heavy user of Private Browsing).

Examples:

1. I launch Minecraft with -Xmx2G; Minecraft uses ~5.4GiB RAM after a long session;
2. I launch Minecraft with -Xmx2G; Minecraft uses ~4.5GiB RAM after a long session;
3. EDIT: I launch Minecraft with -Xmx2G; Minecraft uses ~3.8GiB RAM after roaming and flying a bit in minecartrapidtransit.net for a few minutes; attached is a screenshot showing the memory usage as reported by the OS, and the crash report (manually triggered with F3-C) (2017-12-23_22.21.05) (note: right-clicking javaw.exe in Task Manager and choosing "Open file location" opens the following folder: C:\Program Files (x86)\Minecraft\runtime\jre-x64\1.8.0_25\bin which is the folder of the bundled Minecraft JRE).

As far as I can tell (speculations only):

• The bug was observed in multiplayer only (it might or might not occur in singleplayer too);
• The bug was observed when playing in minecartrapidtransit.net (it might or might not occur when playing in other servers too);
• The bug was observed in recent versions of Minecraft, probably since Minecraft 1.11 (it might or might not occur in older versions too); I can't tell for sure, since there was a long period in which I haven't played Minecraft.

Solutions I tried myself and didn't work:

• Launching Minecraft with various JVM arguments, including enabling large pages support (-XX:+UseLargePages) and various other tricks;
• Using different resource packs.

----------------------------------------

I use the official, unmodified Minecraft client and launcher.

Attached is the crash report (holding F3-C for 10 seconds method) of one session: javaw.exe was using ~5GiB RAM (as much as I remember).