Game controllers and minecarts do not cooperate with each other. When sitting in them, all controls revert to default, non-customized settings. Then when you jump out, putting your crosshairs anywhere on it teleports you back into it.