If the target creature is an illegal target as Floodpits Drowner's last ability tries to resolve, it won't resolve and none of its effects will happen. Floodpits Drowner won't be shuffled into its owner's library.