I have fished for over 10 full Minecraft days (not nights) on a vanilla MC servers (Java) and have not gotten a single Nautilus shell.  I am fishing in warm ocean from the shore.

I have gotten over 10 of each of the other 6 "treasure" items.  I am using a Mending, Luck of the Sea III, Lure III, and Unbreaking III rod.

I will continue try, but over this period of time, something is definitely not right.