The offscouring of society, or rather, “refuse and sweepings.” Rief is Anglo-Saxon, and means a rag; Raff is also Anglo-Saxon, and means sweepings. (Danish, rips-raps.) The French have the expression “Avoir rifle et rafle,” meaning to have everything; whence radoux (one who has everything), and the phrase “Il n'a laissé ni rif ni raf” (he has left nothing behind him).
“I have neither ryff nor ruff [rag to cover me nor roof over my
Ilka man agayne his gud he gaffe That he had tane with ryfe and raffe.
Quoted by Halliwell in his Archaic Dictionary.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Riff-raff from Infoplease: