The old 3rd regiment of foot soldiers. The men's coats were lined and faced with buff, they also wore buff waistcoats, buff breeches, and buff stockings. These are the “Old Buffs,” raised in 1689.
At one time called the Buff Howards, from Howard their colonel (1737-1749). The “Young Buffs” are the old 31st Foot raised in 1702; now called the “Huntingdonshire Regiment,” whose present uniform is scarlet with buff facings.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Buffs from Infoplease: