Sherman blocked freed slaves from following his army. It wasn't that they were "based", it's that they were in command of armies doing army shit and not pretending to be humanitarians.