The ability that defines Nightmare’s power and toughness works everywhere, not just on the battlefield.

Nightmare’s ability counts all lands you control with the subtype Swamp, not just ones named Swamp.

Nightmare’s power and toughness changes as the number of Swamps you control changes.

If you control 0 swamps, then the Nightmare has 0 toughness and will be put into its owner’s graveyard as a state-based action right before the next player gains priority.