The adherents of Lowe and Horsman, seceders in 1866 from the Reform Party. John Bright said of these members that they retired to the cave of Adullam, and tried to gather round them all the discontented. The allusion is to David in his flight from Saul, who “escaped to the cave Adullam; and every one that was in distress, and every one that was in debt, and every one that was discontented, gathered themselves unto him” (1 Samuel xxii 1, 2).
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Adullamites from Infoplease: