a baby bust like with japan and russia is fine as long as the population remains native the nation will eventually recover. Just minus the type of genes that are ok with not breeding but once colonized both the native population and nation will die permanently.