The emperors didn't come from a single lineage. Initially they were Caesarean Julii, but as time went on it was more of a case that the army would just pick someone and make them the emperor. Many of them weren't even Roman.