Buff is a contraction of buffle or buffalo; and buff skin is the skin of the buffalo prepared. “To stand in buff” is to stand without clothing in one's bare skin. “To strip to the buff” is to strip to the skin. The French for “buff” is buffle, which also means a buffalo.
“And for the good old cause stood buff, `Gainst many a bitter kick and cuff.”
Butler: Hudibras's Epitaph.
“I must even stand buff and outface him.” —Fielding.
BUFF in “Blind-man's buff,” the well-known game, is an allusion to the three buffs or pats which the “blind-man” gets when he has caught a player. (Norman-French, buffe, a blow; Welsh, paff, verb, paffio, to thump; our buffet is a little slap.)
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Buff from Infoplease: