Combat damage that would be dealt to Gloom Surgeon will be prevented even if you can’t exile that many cards from the top of your library.

If damage that can’t be prevented (such as the damage Malignus deals) is dealt to Gloom Surgeon, you’ll still exile cards from your library.