A tatter, hence a remnant, hence a vagabond or ragamuffin.
“Lash hence these overweening rags of France.”
Money by me? Heart and good-will you might, But surely, master, not a rag of money.
(The). The Army and Navy Club. “The rag,” of course, is the flag.
“ `By the way come and dine to-night at the Rag, said the major.” —Truth, Queer Story, April 1, 1886.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Rag from Infoplease: