Gamemode switcher can forget previous gamemode. It seems to be kinda random, however I encountered this bug few times and I'm using this feature a lot actually.

How to define if swicher forgot previous gamemode?
Quick F3+F4 can sometimes drop you into survival mode even if your previous gamemode was creative OR SPECTATOR, so I believe it can happen with adventure mode too.

How to reproduce?
I can't really tell. It's just random. Spamming F3+F4 would force that finally, but it's just very likely to encounter this bug at last when messing around on creative and spectator modes